Asynchronous PHP Gotchas

As part of a PixCede re-write (read: correcting damage from a rabbit I chased to far down a hole -- more on that in another post), I decided to modularize the scripts for handling new messages. Previously, procmail was sending the email to a single PHP script that handled extracting attachments, storing the image, and sending the confirmation message. As the script started to get unwieldy, I decided to break it into separate scripts:

• newMailDaemon.php - main script that handles writing the attachment to disk and asynchronously calling additional tasks. This is purposely kept minimal, so that there is less chance of something going wrong (e.g. compilation error). Worst case scenario, the email is written to disk; if the other tasks fail, the unit of work can be replayed from the message dump


• processMail.php - handles extracting the image, creating the shortname, inserting it into the database, and sending the email confirmation. This will be split up eventually

After newMailDaemon.php writes the mail file to disk, it calls exec(..) to asynchronously kick off processMail.php. I was about to pull my hair out until I figured out a couple of things:

• exec(..) does not get called as though through a shell. As a result of that, you need to use absolute paths. That means that `php ./processMail.php` becomes `/usr/bin/php /var/pixcede/xxxx/processMail.php`


• As a corollary to the previous point, absolute paths need to be specified in exec'ed scripts as well; e.g. the database file specified to sqlite3


• Permissions matter! I was using a logger to follow the actions once an email was resolved, and I couldn't figure out why nothing from processMail.php was getting logged. I was logging the command that got exec'ed, and running it manually -- and it always worked. procmail calling newMailDaemon.php calling processMail.php runs at the permissions of the user account that procmail is acting on behalf of, so the script needs to permissions to run from that user. Whatever is running the main script needs permission to exec additional scripts

This design is far from perfect. The different actions need separated more; ideally, having newMailDaemon load and parse an external workflow file wouldbe nice -- then I could lock down newMailDaemon and not have to change it to update the workflow (less chance of it failing). To handle and track the different steps, I would like to keep a table of actions to process, have entries put into that, and have another script act on that table until all tasks are set to "complete"; that sounds more scalable and manageable than exec'ing scripts for each action that needs to be done. Having newMailDaemon schedule tasks and processTasks execute those tasks sounds cleaner.

At any rate, PixCede works again now!

PixCede gets shorter file identifiers

My initial filenaming convention for PixCede was pure rubbish.

It all started with the best of intentions. After reading about the development of Pastebin and why the database got scrapped, I was inspired to make PixCede rely on the filesystem and not the database. I did, however, need to keep the timestamp so I could sort the images by the order in which they arrived. So my initial naming convention was:

2008-07-04T07:19:03+00:00_f6cfee1f4e477963a3c24d8f9b769722.jpg

That is, PHP's date('c') followed by an MD5 of the file. My logic was, date('c') would keep the time property and, if by chance any two images hit the system in the same second, they would certainly have different contents, and the MD5 would differentiate them (unless of course, the same image hit at the same time, but.. I don't see why it would need to be there twice). Using date('c') was just a bad idea from the start; if I had spent 2 seconds more thinking about it, I would have just used time(), which returns the number of seconds since the Unix epoch. Using an MD5 hash is a dumb idea too, because it's a pretty expensive operation.

So for version two, I used time() concatenated to uniqid(), a function that creates a UID based on the current time in microseconds (it's what the PHP manual pages recommend to use for Session IDs). Without any parameters, uniqid() returns a 13 character string. That brings me to:

1219098558133aaffc6178602.jpg

Substantially better, but.. as the great doctor says, if something's worth doing, it's worth doing right. Ideally, I want PixCede to send a small enough unique identifier back to the user that he can type it into his browser, after receiving an SMS back. Next on the chopping block: the 13 character unique identifier.

In a dream world, I might get 1000 pictures per second with PixCede. Realistically, I think I only need to worry about two at once (and that's a stretch), but 1000 seems like a nice round number. If I use PHP's base_convert(), I can create a random number from 0 to 1,679,616 (36^4), and convert it to base 36 (10 digits + 26 letters) and only use 4 precious characters. This brings it down to:

1219098558xe21.jpg

Which is a lot better! It preserves the ordering in the first 10 characters, and uses 4 characters on the end to make it unique. But why not go balls to the wall? I might as well convert both the timestamp and the random number to base 36; a timestamp in base36 will sort just as well as one in base10. The final code I am using:

$id = (time() * pow (10, 7)) + rand(0, pow(36, 4));
$uid = base_convert($id, 10, 36);


3bnd9c4wf66.jpg

11 character total, a saving of 47 over my original, poorly encoded 58 characters! This, I feel is an acceptable UID to have to type in by hand. If you want to improve on it, base62 is just small function away (there's a user contributed one on the PHP base_convert() page).

EDIT:
I decided to go ahead and switch it over to base62 encoding. The comment on base_convert() actually didn't work for what I needed; the images were mostly sorted, but off a little bit. It turns out you need to switch the upper and lower case letter sets in the dec2any() function ("0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"), and now everything works proper, with a final, 9 character encoding of:

tfebHWV5s.jpg

PixCede: Thoughts on how to proceed

I haven't done any "real" work on PixCede since late last week, but I've done a lot of thinking about it -- which is probably a lot better than implementing without thinking. I've also been using it myself more than I thought I would be, which is really exciting. I've told a couple of my friends and collected some feedback, but Zach is the only one who started really playing with it.

I've thought a lot about whether or not to allow emails from non-phones through. The main concern is the potential for abuse -- who's to stop someone from writing a script to constantly spam the front page with whatever? Zach had a good suggestion of getting a short SMS number (like GOOGL) and only allowing pictures to come in through that. Then, I realized that getting a picture from one of my computers to the other meant emailing it to myself, which for some reason just seems like a huge pain in the ass. I used GMail to email PixCede a picture of myself on the Half Dome, and just retrieved it through the web site on the other computer. Yea, still email on one end, but on the receiving end, I felt incredibly less inconvenienced. Therefore, I think I'm going to implement a simple file upload feature to PixCede so that you can post a picture from one computer quickly, and retrieve it from another computer quickly -- without the hassle of logging into email on both computers.

I know what you're thinking: great idea! TinyPic had that idea YEARS ago! Short answer, yes, long answer... not quite. For one, I really am going to try to model this site after PasteBin more so than TinyPic and other similar services. I really like the dead simple usage of PasteBin, and I also like the content expiration too. I feel like that would discourage people from using PixCede as an image hosting solution -- which it is not. PixCede is a simple way to transfer pictures from one device, be that a cell phone, a workstation, or something else I don't know about, to another. So in that respect, I hope to differentiate myself from TinyPic and Flickr -- this is more of a PasteBin fork.

So, speaking of PixCede, I realized that the owner made all of his code open source. I love not re-inventing the wheel, so I'm going to start reading his blog and poking into the source code. I've briefly scanned his blog, and I was happy to see that he switched from a database back end to just a plain file system -- a decision I arrived at with PixCede a week or so ago. For something this dead simple, a relational database is really overkill. It just seems like the common decision to "my web app needs to persist information" is to just cram it into a database without really thinking about it. A well designed file structure will work just as well in this case.

PixCede: First working version!

It now seems that PixCede is "working enough" to justify putting up that blog I've been avoiding. Thanks to Simple PHP Blog, I was able to get one up really quick without having to install database software.

After fighting with the mailparse extension for PHP, I decided to switch to the PEAR mimeDecode module. From what I understand, it will run a little slower than the extension, but it has the main advantage of working. The move to mimeDecode follows the initial proof of concept that used procmail to pipe new messages to the command line utility munpack, which was being called from within a PHP script. Currently, procmail pipes to a PHP script that does the MIME decoding in the same script. The only other thing I'm doing is creating a thumbnail for the main page, renaming the image, and storing it to disk.

Like I said, functionality is basic right now. I don't have a database running; images are just renamed to {timestamp}_{md5 of content}.jpg. The naming convention takes care of order of submission, and throwing the hash at the end makes sure that unless two people submit the exact same image at the exact same time, there won't be any collisions. But, then again, if the image is the same, and the submission time is the same, would you really need it stored twice anyway?

I still have a huge todo list:
* Allow images to be directly accessible
* Automatically update the main page when new pictures are submitted
* Video uploads
* Confirmation SMS with direct URL
* Browse by time periods

I'm not sure what the "final" product is going to end up looking like; it will just be a continual evolution. While this is completely open and anyone can use it, I'm assuming it's still going to be people I tell by word of mouth for now. If you have suggestions or comments, please let me know!

Introducing PixCede

PixCede is an idea I came up with on the bus into work one day. I've been doing some development on it in my free time, and finally put up a working "proof of concept" last week. There's a development blog on the site, but I've decided to mirror PixCede posts on this blog.

(ripped from my about page)
What is PixCede?
Take a picture with your cell phone camera, send it to submit@pixcede.com, and check pixcede.com to see your picture. Additionally, you can directly view your image using the submission code sent back to your phone.

Ok, that's what it does, but what is it?
PixCede is defined in purely functional terms because I don't care what you use it for. In fact, I'm very curious to see what kind of pictures show up on here. I'll worry about how it works, you worry about what to use it for.

That sounds dumb. Can't you think of anything to use it for?
Sure, I'll tell you exactly what I'm going to use it for: I take pictures with my cell phone camera, and then they just stay there. I'm just going to use this as a sort of Pastebin for my pictures, between cell phones and computers. I'd imagine a lot of the pictures I submit will be either things I think look cool, cool cars, or funny things. But seriously, you should use it for whatever you want. I'm just providing the tool.

Sounds like a cheap way to host pictures!
PixCede is not a traditional image hosting site. I am paying for bandwidth out of my own pocket. If you need a dedicated image hosting site, use S3, Flickr, Picasa, or any number of other services designed for that

So what are you getting out of it?
Entertainment. I, hopefully like you, am interested in seeing what kind of pictures get submitted! A couple of months ago I saw a site about a guy who took a disposable camera and tied it to a park bench with a note that said something to the effect of "take a picture of whatever you want, I'll develop the film once it's all been used." This is my virtual camera-attached-to-a-park-bench.

Am I giving you my pictures? Will you make money off of selling them?
First off, I really don't know who you are -- by design. But I certainly don't think it's right for me to own them either. By submitting a picture to PixCede, you are putting it into the public domain.

Are you making any money off of PixCede?
Not right now. If it really takes off and I have to start putting up AdSense to cover excessive bandwidth costs, I'll let the community know beforehand. For right now though, I'm taking a barebones Craigslist approach.

What does 'PixCede' mean?
The 'Pix' part is pretty obvious: pictures. The suffix '-cede' means to transfer. Altogether now... 'picture transfer.'