Asynchronous PHP Gotchas

As part of a PixCede re-write (read: correcting damage from a rabbit I chased to far down a hole -- more on that in another post), I decided to modularize the scripts for handling new messages. Previously, procmail was sending the email to a single PHP script that handled extracting attachments, storing the image, and sending the confirmation message. As the script started to get unwieldy, I decided to break it into separate scripts:

• newMailDaemon.php - main script that handles writing the attachment to disk and asynchronously calling additional tasks. This is purposely kept minimal, so that there is less chance of something going wrong (e.g. compilation error). Worst case scenario, the email is written to disk; if the other tasks fail, the unit of work can be replayed from the message dump


• processMail.php - handles extracting the image, creating the shortname, inserting it into the database, and sending the email confirmation. This will be split up eventually

After newMailDaemon.php writes the mail file to disk, it calls exec(..) to asynchronously kick off processMail.php. I was about to pull my hair out until I figured out a couple of things:

• exec(..) does not get called as though through a shell. As a result of that, you need to use absolute paths. That means that `php ./processMail.php` becomes `/usr/bin/php /var/pixcede/xxxx/processMail.php`


• As a corollary to the previous point, absolute paths need to be specified in exec'ed scripts as well; e.g. the database file specified to sqlite3


• Permissions matter! I was using a logger to follow the actions once an email was resolved, and I couldn't figure out why nothing from processMail.php was getting logged. I was logging the command that got exec'ed, and running it manually -- and it always worked. procmail calling newMailDaemon.php calling processMail.php runs at the permissions of the user account that procmail is acting on behalf of, so the script needs to permissions to run from that user. Whatever is running the main script needs permission to exec additional scripts

This design is far from perfect. The different actions need separated more; ideally, having newMailDaemon load and parse an external workflow file wouldbe nice -- then I could lock down newMailDaemon and not have to change it to update the workflow (less chance of it failing). To handle and track the different steps, I would like to keep a table of actions to process, have entries put into that, and have another script act on that table until all tasks are set to "complete"; that sounds more scalable and manageable than exec'ing scripts for each action that needs to be done. Having newMailDaemon schedule tasks and processTasks execute those tasks sounds cleaner.

At any rate, PixCede works again now!

Part 2: Drupal 6.2 and DB2 Express-C 9.5 on Ubuntu 7.10

This is part 2 in my series on getting Drupal 6.2 and DB2 Express-C 9.5 to play nicely with each other. In the first part, I installed DB2 Express-C 9.5. In this part, I'll look at getting Apache/PHP5 setup and DB2 working from within PHP.

Step 1: Configuring DB2. There were some additional configuration steps needed for DB2, which are detailed on the developerWorks forum. There were some superfluous steps, but the worst you're going to do is overwrite something with the same thing. In particular, all the user accounts were set up already. If you're not familiar with DB2, you can issue commands from the shell, in the form of:

db2 "create database testdb"

but just make sure you're running these commands as the user db2inst1, or you source db2inst1's db2profile. If you can create `testdb`, then DB2 is working correctly.

Step 2: Install Apache httpd 2.2 and PHP 5.2. The Ubuntu Guide has information on setting up Apache and PHP, but the quick and dirty steps are:

sudo apt-get install apache2
sudo apt-get install php5 libapache2-mod-php5
sudo /etc/init.d/apache2 restart

Step 3: Install DB2 support for PHP. There is a PECL module for IBM DB2, which can be installed with PHP's PEAR (PHP Extension and Application Repository, like apt-get for Ubuntu).

sudo apt-get install php-pear
sudo pecl install ibm_db2

When I first ran this, I got the error: "sh: phpize: not found". After doing a quick search, I realized that I needed the PHP development files to be able to compile the DB2 driver:

sudo apt-get install php5-dev

After the module is compiled, it will ask you where your installation of DB2 is. In my experience, it wouldn't believe me when I told it where DB2 was installed to. Luckily, it ended up not mattering. When you get to this part of the PECL install, just hit Ctrl-C.

Next, I needed to configured PHP to use the IBM DB2 driver. In `/etc/php5/apache2/php.ini`, go to the extension section and add:

extension_dir="/opt/ibm/db2/V9.5/dsdriver/php32"
extension="extension=ibm_db2_5.2.1.so"

Note: correct the path if you have your module installed somewhere else; do a `locate ibm_db2_5.2.1.so`)

After making the changes, I reloaded Apache:

sudo /etc/init.d/apache2 reload

Verify that the module is loaded by making a quick PHP page in /var/www/ with the content:

<?php phpinfo(); ?>

and make sure there's a DB2 driver loaded. I started playing around with DB2 in PHP, but it was getting late and I decided to put that off for another day. IBM has some information on developing PHP5 for DB2, but it's barebones at best. I'll post a sample script next time.

Ok, so at this point, I have Apache, PHP, and DB2 Express-C 9.5 all installed and playing well with each other. In the next article in this series, I'll look at getting Drupal 6.2 to use DB2 on the back end.

Managing JPG and CR2 files

My latest toy is a Canon Digital Rebel XTi, which has the option of shooting in JPG and RAW, or both. I don't have a lot of use for RAW (CR2 on the XTi) images right now, but I might someday. And with space as cheap as it is, I set the camera to shoot in both. They both end up in the same directory, but only the JPGs show up in GQView on Xubuntu. So, I go through and delete the JPG version of what I don't want -- CR2s don't show up in the file browser. I mean, it would be easy to go through by hand and delete CR2s if an associated JPG does not exist. But here in the land of over-engineering, we create a script to do it!

I think this would be a lot easier in bash, but I don't know bash scripting. So... quick and dirty? Yea, PHP is my language of choice.

#!/usr/bin/php
function sysout($message) {
    fwrite(STDOUT, $message);
}
$cr2Extension = ".CR2";
$jpgExtension = ".JPG";

if ($argc < 1) {
    sysout("No directory specified, exiting.\n");
    exit(-1);
}
$directory = $argv[1];
if (!is_dir($directory)) {
    sysout("`$directory` is not a directory, exiting.\n");
    exit(-1);
}

sysout("Working directory: $directory\n");
sysout("Removing CR2 files if JPG does not exist...\n");
$files = scandir($directory);

$count = 0;
foreach ($files as $file) {
    if (strripos($file, $cr2Extension) > -1) {
        sysout("Found CR2: $file\n");
        $jpg = $directory . substr($file, 0, strripos($file, ".")) . $jpgExtension;
        if (!is_file($jpg)) {
            sysout("Deleting: $directory$file\n");
            unlink($directory . $file);
            $count++;
        }
    }
}
sysout("Process ended successfully. $count files deleted.\n");
exit(0);
?>

If anyone wants to enlighten me to the 5 line bash or Python solution, by all means. This is easily adapted to Nikon style cameras by changing $cr2Extension to something else.