Using Google Voice to Extend T-Mobile MyFaves

I switched to T-Mobile last fall when I switched to the G1, and have found their MyFaves feature really useful. For those of you who don't know about it, MyFaves are five phone numbers that you link to your account and get unlimited talk time to. This is GREAT. It turns out I talk to my two most frequently called numbers more than any of my other numbers combined. However, the 866 conference call number that I dial into for work is not allowed to be added as a MyFave. That's annoying. And it was more annoying after doing that a handful of times thinking the MyFave add went through (when it didn't) and getting the bill.

Enter Google Voice (formerly known as GrandCentral before being acquired). GrandCentral started out as a service that gave you a free phone number that you could link to your different physical phones. When your GrandCentral number rings, it rings all of your linked devices and you can pick up the one that's most convenient. For example, if you're at work and you don't want to pay for airtime minutes, you can pick up on your work phone. Neat. I signed up for this service when they launched and forgot about it until recently, with all the hub-bub about Google Voice.

Google Voice adds a lot of cool things that I will probably start using eventually. But let me get to the meat of it. Add your Google Voice number as a MyFave through T-Mobile. Then, from the web interface, you can supply it with a phone number and have it connect the call. When your cell phone rings, it's from your Google Voice number -- and since that's a MyFave, you get unlimited free talk time. Using Google Voice, you can turn your five MyFaves into unlimited MyFaves and never again have to pay extra minutes.

The Little S3 Backup Script That Could

After having an external enclosure fail (grrr LaCie!) and realizing that all of my backup data was spread over about 6 hard drives, I decided to check out Amazon's Simple Storage Service (S3) to back up all of my important files to one central location. A friend recommended Jungle Disk because it was "free", but since it wasn't, I decided to play with it on my own. Given my recent love for Bash scripting, I decided to start that way.

My first task was to make a back up of all my pictures, which was a set of about 100 folders following the format "YYYY-mm-dd name of event". Since S3 just gives you a table interface, I decided to archive each folder in a tar and then upload the tar. It seemed unlikely that I'd want only one picture from a set, so this also made sense. The caveats were that I needed to change spaces to underscores in the .tar filename and that I only wanted to keep the .tar around long enough to upload it. I found Google's s3-bash to be the most straightforward utility for copying files to S3 from my shell.

#!/bin/sh

ls -d /media/gadget/pictures/* | while read -r FILE; do
   # transpose spaces to underscores
   tarfile=`echo $FILE | tr ' ' '_' `.tar
   tar cvf $tarfile "$FILE"
   /opt/s3-bash/s3-put -k #MY_KEY# -s /opt/s3-bash/.secret \ 
            -T $tarfile /derwiki-media/`basename $tarfile`
   rm $tarfile
done

Twitter Bowl 2009: A Ten Day Experiment in Gonzo (part 1)

Twitter Bowl is a Twitter mash-up that Adam Bossy (@abossy) and I (@derwiki) created in ten days for the Super Bowl. I'd like to share with you our experience and the lessons that we learned.

Day 1: Initial Conception and Working Prototype
This was the easy part. Coming up with the idea was inspired by a co-workers presentation about mash-ups, and a baseball mash-up he made. I had used TweetGrid for the AFC championship to see what the buzz was -- who the Twitterverse thought would be going to the Super Bowl. It was surprisingly entertaining to follow, but I figured the general public wouldn't want to play around with TweetGrid. Luckily, there's a TweetGrid widget, so I embedded two of those on a page with different keywords for different teams. From the time I opened Vim until I finished the first iteration, it honestly took two hours of work. I already had a Slicehost account, so I just threw it up on there, registered the domain, and set up the DNS records. It took all of 5 minutes to add Google analytics to the site and start tracking.

Day 2: Post First Iteration to Ask Hacker News
We didn't get as much traffic as I had hoped, and most of the criticisms weren't constructive: "your site design sucks", "do something useful", "solve a real problem", etc. By this point, I had gotten enough positive feedback from friends that I said fuck the naysayers cuz they don't mean a thang. They were right, but they didn't tell me how to improve it. There were a few good things that came out of this post, so it was completely worth it. Izak30 pointed out that there might be trademark issues using the term "Super Bowl" outside of the widget on the site, so we nixed that. He also wrote a blog post about our site and how we were soliciting feedback so early in the development cycle. I'd like to thank Evan Heiser (@sass9210) for the idea on making the new design. He pointed out out that the ESPN widget didn't really fit the rest of the site, but since it was unchangeable, we should make the rest of the site match it's styling. Brilliant!

Day 3: Snowboarding.
We talked about Twitter Bowl to people, but weren't able to really work on it. No regrets here :)

Day 4: Revamping the UI and Adding Inline Tweeting
With Bossy solidly on board, he plowed through some design mock-ups in Photoshop. We actually posted our ideas to Ask Proggit, but didn't get anyone to respond. We ended up just asking a lot of our friends and choosing the most popular design. As Bossy started chopping up the graphics into HTML/CSS/images, I started working on the inline Tweeting function. This was another recommendation from Evan, saying that it would make visitors feel more like active participants. There were some technical hurdles that I won't bore you with (cross site Ajax), but Script.aculo.us and a quick 'n dirty PHP script saved the day.

Day 5: Switching to GWT and integrating inline Tweeting
Amongst frustration with tables/CSS hell, Bossy started re-implementing the page using the Google Web Toolkit. In the meantime, we kept up our bastardized version that had basic inline Tweeting functionality -- but just didn't feel clean. I think it was this day that I also created the Twitter Bowl fan page on Facebook.

Day 6: Advertise Advertise Advertise!
I literally told anyone and everyone. I created a Twitter account, @tw1tterbowl, to announce to no one in particular about the site. My hopes were that people would search Twitter for things related to the Super Bowl and run across Twitter Bowl. We also posted to Reddit, and received a lackluster response. It was about this time that we realized web site + HN/Reddit/Digg wasn't going to be enough. At the recommendation of friend Ed Shelton, I started contacting every radio station in Pittsburgh who might want to talk about it and emailing people who ran Steelers blogs. No radio station that I know of said anything about it, but Matt Loede at Steelers Gab wrote an entire post about us.

Day 7: Implement Advertising
After still not hearing back from AdSense (wtf Google!) we signed up for AdBrite and put a banner on our site.

Day 8: Finished GWT Version
Bossy finished up the new site done with the Google Web Toolkit and we pushed it live.

Day 9: Snowboarding Again.
What can I say, I'm addicted!

Day 10: Super Bowl Sunday
We did a last minute blitz of advertising. We posted to Hacker News again, Digg, Twittered about it, and then sat back to watch the game. We kept a laptop open that was updating total/unique hits in real time. Just as anticipated, we saw a spike in both during the actual Super Bowl. That Sunday, we doubled the amount of traffic that our site had received in total. The next blog post will be a more detailed analysis of the traffic, but I will tell you that our total unique count is around 2000. That might seem paltry, but I don't know 2000 people, so I consider this a success.

That was a quick ten days! I was so happy to have something fun to program again, even if it wasn't technically challenging. What was challenging were the other aspects: incorporating feedback, advertising, and improving the user experience. And we learned a lot about that! Our major lessons:

Release as early as you can and start collecting feedback. This is the single most important thing I will take away from this experience. You, the developer, are creating the site/service for the user. There is absolutely nothing stopping you from using the community from the earliest stages. For Twitter Bowl, we had an idea, created a prototype, and launched. Everything we implemented after that, everything we tweaked, and how we advertised was a collaboration between myself, Bossy, and everyone else who cared enough to tell us.

Ask anyone for advice who you think might care, and don't get discouraged if they don't care. I asked everyone I knew for feedback almost every day (thanks guys!). My own personal Twitter account was non-stop about Twitter Bowl, and I asked everyone I knew for advice on how to improve the site -- especially the people who don't drink the Twitter Kool Aid (read: non-programmers/techies). But by the same token, I didn't get upset if people had nasty things or nothing to say. The good feedback/advice I got made all the crap worthwhile.

If you're going to post to Hacker News, Reddit, etc. asking for help, listen to the feedback! More importantly, take the time to personally thank everyone who responded with constructive criticism. They deserve it! But seriously, distance yourself from the project. Criticisms to the project are not criticisms of you. Every bit of feedback you receive is valuable and might help you realize a mistake you're making before it becomes a big mistake (this goes hand in hand with launching early and iterating quickly).

Don't re-invent the wheel. It's 10x quicker to consume existing solutions than to create your own. You end up wasting less time on details, and can focus more on the bigger picture. We used Google extensively: web toolkit for the page, analytics for daily updates, and AdSense to make 27 cents (seriously, that's it). TweetGrid provided a widget that provided the basic functionality; ESPN provided a widget that allowed us to legally put a Super Bowl logo on the page.

Don't advertise. I'm still really pissed at us for this one. We made twenty-seven lousy cents at the expense of putting up distracting and useless advertisements. If we had put up TipJoy and been tipped a quarter, we would have done just as well. Even with the modest goal of earning back the domain name registration ($10), we fell entirely too short and wasted too much time setting up and implementing advertising. I mean, it wasn't a lot of time -- but at a return of 25 cents, any amount of time was wasted time as far as I was concerned. On the other hand, this did confirm my suspicion that the advertising-based revenue model is total bullshit. If I ever do a start up that doesn't have a different revenue model... well, I just won't do that. This experiment cleared up any delusions that I might have had.

I'm going to conclude this post with a charge: if you have an idea, try it! This project was an invaluable learning experience, and cost very little. Maybe 10-20 hours of development time, and maybe 5-10 hours of advertising. That's IT. What's stopping you from putting your idea out there?

Learning How To Snowboard, In Engineer-Speak

This is going to be a slight departure from what I normally write about. This past winter, I've been learning how to snowboard and it's been fantastic! One of the new loves of my life, right up their with programming, fireworks, and Dark Side of the Moon. I was fortunate enough to have engineers teach me how to snowboard. That sounds silly, but some things are better expressed in engineer speak from one engineer to another; you can express ideas more succinctly and explain the underlying concept. This blog post is meant for engineers to quickly learn the fundamentals of how snowboarding works.

1. Standing up
Ok, the first thing you need to do is learn how to stand up. The important thing is that you position your board perpendicular to the slope of the hill! Remember, you move quickest along the long edge of your board, so you want the edge to be perpendicular to the slope vector of the hill to minimize downward motion. You want to stand up as fluidly as possible; kind of rock your upper torso back and forth, and then push your weight forward. As you stand up, remember that you're not putting the board flat down on the snow. If you do that, you'll start sliding and probably fall. Your back edge needs to dig into the snow, and your front edge should be raised slightly above the ground. All you're trying to do is balance here.

2. Sliding forward
Start from standing up and lean slightly forward, tilting your board down the slope you're going down. Do it slowly and you'll start inching forward. To stop, just lean back. The more you lean forward, the faster you will go, because your downward force on the hill will be distributed over a larger surface area and your board will have less ability to dig into the snow.

3. Leaf Falling
As you continue to slide forward, your board-to-slope will veer away from 90 degrees. You'll find yourself either moving more left or more right as your board starts turning slightly. Don't worry! You can easily correct this. If you start moving left, kick your right foot forward. The goal is to make the board closer to perpendicular to your motion vector. Once you correct, you will probably start sliding gradually towards the right. This is good! Kick your left foot forward and continue the process. You will look like a leaf wafting down from a tree, hence the name.

4. Stopping while Leaf Falling
This is all about getting your board perpendicular to the slope vector again. And if you can leaf fall, stopping is simple. When you go to change directions, don't kick your foot as far foward. Kick it far enough that you stop moving with respect to your motion vector, then immediately swoosh your feet so that the board is perpendicular to the slope vector of the mountain. This is a two step process: halt motion that is existing, and position the board to prevent more motion.

5. Faster!
As you become more comfortable with leaf falling, you can start aligning the edge of the board more with the direction vector; this will increase your velocity. When you think you're going to fast and want to stop, kick your trailing foot so that you're board is perpendicular to your motion vector. Get proficient at this before you move on.

6. Linking Turns
To graduate to linking turns is a natural progression -- if you're going fast enough. Carving an edge into the snow is much easier with a little bit of speed, so don't be too afraid of it. When you're sliding with your board flat on the ground, you have the least control -- your weight is being distributed over the maximum surface area. You only have control of the board when you're on your toe edge or your heel edge, because you're concentrating most or all of your force on that edge. So remember, carving == control. Going along with that, to go straight, you don't. As you go down the slope, you switch from toe edge to heel edge making "S" tracks in the snow. If you stard on your heel edge, you'll cut left across the mountain. Don't cut too sharply! When you've gone as far over as you want to, you shift weight from the heel edge to the toe edge. You'll briefly not have as much control when you're flat on your board, but as you continue leaning forward, you'll toe edge will engage and you'll curve to your right. Do the same thing in reverse when you get far to the right.

That's it! Like engineering, a solid understanding of the fundamentals and why certain things work will take you a long way. And just like engineering, you need practice, so hit the problem set (slopes)!

Using sed to remove deprecated function calls

I am super lazy. As a programmer, this is a good thing. For a project that I work on, I was recently given a code maintenance task of cleaning up assert statements. ASSERT_S_#_MED had been deprecated in favor of just ASSERT_S_# (where # is the number of informational parameters for logging purposes).

The caveats are that I don't know how many files have the deprecated call, and I didn't want to do a blind stripping of _MED because I wasn't sure if it was legitimately somewhere else in the code; so I needed to make sure a digit existed before it.

  1 #!/bin/sh
  2 LIST="`grep '_[1-9]_MED' *.C | awk '{print $1}' | sed 's/://g' |uniq`"
  3 echo $LIST
  4 for i in $LIST
  5 do
  6   cleartool co -nc $i
  7   sed -i 's/_\([1-9]\)_MED/_\1/g' $i
  8   cleartool ci -nc $i
  9 done

Line 2: Populate a list of all files with the deprecated call. The awk takes the first part of the grep, then the sed strips the colon, and then uniq removes duplicates.
Line 7: The sed matches _[1-9]_MED, captures the digit, and uses a backreference to put it in the replacement. The -i flag means it edits in place.
Line 6,8: Version control that I'm using for the project

Hopefully this will be useful to someone! If you have a better way of doing it, I encourage you to post in the comments. We're all learning, after all!

Inspiring a New Generation of Engineers

I used to have an old Motorola e810, a standard non-smart phone with a camera, circa 2005. Being the cheap bastard that I am, this phone doubled as my alarm clock, boasting multiple different alarms. I stumbled across an interesting quirk: you couldn't set more than one alarm for a time; that is, if you had an alarm for 6:00pm that said "Laundry," you couldn't have one for 6:00pm that said "Order Piza". Honestly, it wasn't a big deal. It didn't matter which alarm went off, as long as one did, and that was probably a design decision made when they created the phone. But it still bothered me.

The only thing I could think of that made sense was that each alarm was some sort of key/value pair where the time was the key, and duplicates weren't allowed. I can sort of see the logic for this. If it was a hash table, for instance, then at each minute you just perform a look up to see if there was an alarm set. This makes slightly more sense then iterating over all of the possible alarms at each new minute (although since most people only have one or two alarms, probably not a huge savings).

While it was a fun exercise to try and determine why the phone wasn't working as I expected, it was sort of mental masturbation. Without the source code to the phone, I have no way of knowing if my educated guess is even in the ballpark. We're in the black box stage of our information age: we don't have to know why things work, we just know that they work. Is the voodoo too complex? Maybe. Because we don't care to know? Another likely possibility. But why with electronics versus other complicated devices?

The Man Code states that you should have a certain amount of automobile expertise. Since the dawn of cars, it's been accepted, if not expected, for man to have prowess with the car. In the beginning they were simple enough that this was a feasible task: there's the carburetor, there's the engine, there's the fuel pump -- ok, I've got this. However, a 2005 Honda Civic is not a 1949 Chrysler. The Civic is exponentially more complex, with computers, fuel saving devices, emissions lowering devices, etc -- but yet you can still change the oil, replacing the starter is not out of the question, and even replacing the brake pads is a manageable task. Heck, if you don't want to pay someone to replace the rotors or the the alternator, you can do it yourself. If you're a real gear head, you can even tackle an engine swap.

Let's go back to the cell phone real quick. I consider myself the computer equivalent of a gear head, but I was unable to do anything about this problem. I couldn't even confirm why it behaved this way because the software for the phone was unavailable. Not that it was too complex for me to figure out, but that I didn't even get the chance to try. I'm sure that everyone has run into that one glitch, that one feature of a program that doesn't work quite right -- the one that makes you think to yourself, "this is so close to working, I ought to be able to tweak it." Well, if it was a car, you could. You could tweak the air to fuel ratio in your carburetor. It's not like the hood of you car was sealed shut by Honda when they sent it to you, preventing you from peeking inside to see how everything works.

In the past, complex devices like tractors, lawnmowers, furnaces, cameras, clocks, etc have been possible to tinker with, if you're willing to put forth the time to get past the learning curve. Cellular phones, Windows, the iPod, GPSes, and other electronic gadgetry aren't accessible in the same way. And don't give me that bullshit about decompiling it to assembly code and tweaking it there; a human readable form (source code) exists that software gear heads just can't get to because the company safe guards it as proprietary information.

This is why open source is important. If you're curious how MySQL or Firefox works, you simply pop the hood and look. How many inventions that we depend on have been created by creative tinkerers? It's fine that most people don't care how their web browser works -- but for that .001% of people who do care enough to want to see makes it tick, why in the world would we prevent them from that?

This is one of the reasons I am such an advocate of the Google iPhone (aka the G1 aka the Android). Linux, Mozilla, MySQL, and a slew of other software has long been freely available to tinker with, but those are limited to computers. One of the other most important devices that we all use on a daily basis, the phone, has just joined the ranks. This is going to open up the phone to unexpected innovations that we have seen with computer, automobiles, and everything else with which tinkers are free to play with. Society has everything to gain from this.

Ultimately, I'm hoping that the open source movement will inspire a new generation of engineers. Not everyone will be interested, but there's no reason to prevent the people who will be. People who learn how things work because they want the satisfaction of knowing how something works. People who want to tweak something so that it works better for them. People who see something good and want to make it great.

Why I Program

On the bus this morning, I read part of "The World is Flat" that deals with left-brain versus right-brain activities; to Friedman, jobs that can be replaced by technology and outsourcing (left-brain) and those that require an element of creativity (right-brain). Friedman made a statement:

"This weekend there will be accountants painting watercolors in their garages. There will be lawyers writing screen plays. But I guarantee you that you won't find any sculptors who on weekends will be doing other people's taxes for fun."

Friedman is trying to convey that some things are done out of passion, and some out of necessity. People do taxes out of necessity, people paint watercolors out of passion. It's all very black and white, and the jobs of necessity are the ones that are more easily sent overseas to India.

Can't there be be passion for traditional "jobs of necessity"? You can outsource your financial management to an established group or mutual fund -- but what about the excitement of finding good deal and making a sound investment? The satisfaction of having researched it and adding your own judgment to what you find? You need a table to eat dinner on, but can't creating that table be a creative outlet? When categorizing these jobs, I don't think there are clear cut categories; it's up to the individual to say what he is passionate about.

Whenever I go home or spend time with my parents, a lot of my free time is devoted to programming, and they can't figure it out. "You do this all day for your job [or for school]. Can't you put down the laptop?" While it's true that I sit in front of a computer for my day job, I'm working on problems of necessity. Database development isn't particularly exciting for me; what I've been working doesn't require a lot of creativity and seems mechanical. So, in my free time, I need to exercise my creative muscle, and I choose programming for that task.

I've often said that good code is beautiful (no, I'm not talking about the Perl Camel). A well designed component or architecture has elegance and sophistication; simplicity yet robustness. You step back, look at the code, say "Damn, that's gorgeous", and you know when you've created beauty instead of just hacking out a quick fix. This element of beauty is something that I've found lacking in a lot of code and with a lot of programmers. Too often programming is treated like a boring job of necessity than the passionate job of creativity that it can be.

This is why I enjoy coding in my free time. Sure, it's all ones and zeros when it comes right down to it, but the important thing isn't what the end product does; how you got there is so much more important. Programming gives you a chance to come up with creative solutions. And they don't even have to be real problems!

I spent an entire day obsessed with the idea of programming language quines, which are programs that print out their own source code. Sounds easy, right? "Just keep a copy of the source code in the code, and print it out. Oh crap. How do I recursively store the code in the code..." Quines are, as my office mate likes to say, mental masturbation: they produce no value and in that sense are worthless, but so incredibly amazing at the same time.

I look at quines and other beautiful code with the same curiosity and admiration that I look at pieces of artwork in a museum; I appreciate them not because of their utility, but because of their elegance. And I code as a hobby outside of my day job not as a task of necessity, but the desire to create beauty.

Asynchronous PHP Gotchas

As part of a PixCede re-write (read: correcting damage from a rabbit I chased to far down a hole -- more on that in another post), I decided to modularize the scripts for handling new messages. Previously, procmail was sending the email to a single PHP script that handled extracting attachments, storing the image, and sending the confirmation message. As the script started to get unwieldy, I decided to break it into separate scripts:

• newMailDaemon.php - main script that handles writing the attachment to disk and asynchronously calling additional tasks. This is purposely kept minimal, so that there is less chance of something going wrong (e.g. compilation error). Worst case scenario, the email is written to disk; if the other tasks fail, the unit of work can be replayed from the message dump


• processMail.php - handles extracting the image, creating the shortname, inserting it into the database, and sending the email confirmation. This will be split up eventually

After newMailDaemon.php writes the mail file to disk, it calls exec(..) to asynchronously kick off processMail.php. I was about to pull my hair out until I figured out a couple of things:

• exec(..) does not get called as though through a shell. As a result of that, you need to use absolute paths. That means that `php ./processMail.php` becomes `/usr/bin/php /var/pixcede/xxxx/processMail.php`


• As a corollary to the previous point, absolute paths need to be specified in exec'ed scripts as well; e.g. the database file specified to sqlite3


• Permissions matter! I was using a logger to follow the actions once an email was resolved, and I couldn't figure out why nothing from processMail.php was getting logged. I was logging the command that got exec'ed, and running it manually -- and it always worked. procmail calling newMailDaemon.php calling processMail.php runs at the permissions of the user account that procmail is acting on behalf of, so the script needs to permissions to run from that user. Whatever is running the main script needs permission to exec additional scripts

This design is far from perfect. The different actions need separated more; ideally, having newMailDaemon load and parse an external workflow file wouldbe nice -- then I could lock down newMailDaemon and not have to change it to update the workflow (less chance of it failing). To handle and track the different steps, I would like to keep a table of actions to process, have entries put into that, and have another script act on that table until all tasks are set to "complete"; that sounds more scalable and manageable than exec'ing scripts for each action that needs to be done. Having newMailDaemon schedule tasks and processTasks execute those tasks sounds cleaner.

At any rate, PixCede works again now!

Finally, someone made it dirt easy to transfer Flickr pictures to Facebook

Normally, I don't care about Facebook albums. Flickr has many more features, handles it better, and has a more photo-centric community around it. Unfortunately, I'm vain, and want as many people to look at my pictures as possible. I've played with a handful of Facebook applications that "integrate" Flickr into Facebook, but I've never found one that made the transition completely seamless. In fact, it seems as though the "new" Facebook design broke the last Flickr application that I used.

I could re-upload all my photos to Facebook. But why do that when I've already done it once? I just want some back end service to pull selected albums from Flickr and put the files on Facebook. It looks like dudemeister Matt McNamara felt the same way, so he wrote flickurbook. This is a dead-simple, hacked together way to pull photos from Flickr and put them on Facebook. I love it! It's exactly as unpolished of an application as I would have written. This does what I was hoping Oosah would do; Oosah is a start-up that I saw at the Palo Alto new-tech meetup a couple of months ago that boasted being able to do the same thing (remotely manage photos between different sources). Unfortunately, when I tried it, it didn't work. Looks like it would have been overkill anyway, flickurbook does exactly what I want it to do.

PS: Check out my Tahoe pictures

Mango Lassi: Better than Synergy

About 3 years ago, my friend Lann showed me this sweet, cross platform program called Synergy that lets you share one computer's mouse and keyboard among multiple computer/screens. Why was this cool? This meant that with my laptop, sitting on my futon I could control my TV computer. Yea, I'm that lazy.

As the years go on, I find myself getting lazier. Now that I have an apartment, the couch is even further away from my TV! So, I went out to set up good ol' tried and true Synergy so that I can use my laptop to control my Mythbuntu box. But, it didn't work. Every time I moved my mouse off screen, if showed up back in the center of the original screen.

Now, I don't care why it doesn't work. Really, I don't. I fiddled for about five minutes, and then went to see if there was anything better out there, when I stumbled across Mango Lassi. It was designed specifically to solve the suckitude of Synergy and x2x. Unfortunately, there's not a package for Ubuntu, so we have to do it the old fashioned way:

(following Marius Gedminas's instructions)

sudo apt-get install git-core curl build-essential intltool \
automake1.9 libdbus-glib-1-dev libgtk2.0-dev libxtst-dev \
libavahi-glib-dev libavahi-client-dev libavahi-ui-dev \
libnotify-dev libglade2-dev

git clone http://git.0pointer.de/repos/mango-lassi.git/
cd mango-lassi
./bootstrap.sh
(acknowledge the prompt)
make
sudo make install

You get a nice little GUI:



I learned the hard way: don't try launching this remotely over ssh with X forwarding enabled -- not good things happen. I've got this running on Intrepid Ibex and Hardy Haron, and it works beautifully!

Quick DB2 Express-C 9.5 Update

A while pack, I started a post series about getting Drupal to work with DB2 Express-C 9.5 under Ubuntu. Unfortunately, right after jumping into this project, I updated my laptop from Ubuntu Gutsy Gibbon (7.10) to Hardy Heron (8.04). During this process, DB2 completely broke itself. Not only did it break itself, it broke itself in such a way that it couldn't even be un-installed through apt-get.

I recently found instructions for how to force removal of a broken DB2 package on Hardy Heron. This is good news, because it seems that about 6 months later, there's finally a working version of Express-C 9.5 for Hardy Heron!

Let's just hope that the db2exc package for Intrepid Ibex (8.10) comes out in a more timely manner :)

PixCede gets shorter file identifiers

My initial filenaming convention for PixCede was pure rubbish.

It all started with the best of intentions. After reading about the development of Pastebin and why the database got scrapped, I was inspired to make PixCede rely on the filesystem and not the database. I did, however, need to keep the timestamp so I could sort the images by the order in which they arrived. So my initial naming convention was:

2008-07-04T07:19:03+00:00_f6cfee1f4e477963a3c24d8f9b769722.jpg

That is, PHP's date('c') followed by an MD5 of the file. My logic was, date('c') would keep the time property and, if by chance any two images hit the system in the same second, they would certainly have different contents, and the MD5 would differentiate them (unless of course, the same image hit at the same time, but.. I don't see why it would need to be there twice). Using date('c') was just a bad idea from the start; if I had spent 2 seconds more thinking about it, I would have just used time(), which returns the number of seconds since the Unix epoch. Using an MD5 hash is a dumb idea too, because it's a pretty expensive operation.

So for version two, I used time() concatenated to uniqid(), a function that creates a UID based on the current time in microseconds (it's what the PHP manual pages recommend to use for Session IDs). Without any parameters, uniqid() returns a 13 character string. That brings me to:

1219098558133aaffc6178602.jpg

Substantially better, but.. as the great doctor says, if something's worth doing, it's worth doing right. Ideally, I want PixCede to send a small enough unique identifier back to the user that he can type it into his browser, after receiving an SMS back. Next on the chopping block: the 13 character unique identifier.

In a dream world, I might get 1000 pictures per second with PixCede. Realistically, I think I only need to worry about two at once (and that's a stretch), but 1000 seems like a nice round number. If I use PHP's base_convert(), I can create a random number from 0 to 1,679,616 (36^4), and convert it to base 36 (10 digits + 26 letters) and only use 4 precious characters. This brings it down to:

1219098558xe21.jpg

Which is a lot better! It preserves the ordering in the first 10 characters, and uses 4 characters on the end to make it unique. But why not go balls to the wall? I might as well convert both the timestamp and the random number to base 36; a timestamp in base36 will sort just as well as one in base10. The final code I am using:

$id = (time() * pow (10, 7)) + rand(0, pow(36, 4));
$uid = base_convert($id, 10, 36);


3bnd9c4wf66.jpg

11 character total, a saving of 47 over my original, poorly encoded 58 characters! This, I feel is an acceptable UID to have to type in by hand. If you want to improve on it, base62 is just small function away (there's a user contributed one on the PHP base_convert() page).

EDIT:
I decided to go ahead and switch it over to base62 encoding. The comment on base_convert() actually didn't work for what I needed; the images were mostly sorted, but off a little bit. It turns out you need to switch the upper and lower case letter sets in the dec2any() function ("0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"), and now everything works proper, with a final, 9 character encoding of:

tfebHWV5s.jpg

Adam's Hang Gliding Adventure: Cooler Than What You Did Saturday

As I was going through some envelopes last week, I found two $100 checks that I wasn't counting on. The "Rules of Surprise Money" were very clear on the next step: I had to buy something I didn't need that was awesome, or do something I didn't need to do that would be awesome. I've got enough junk, so I decided to go with the latter. Browsing through Meetup.com, I found a group of people going hang gliding on August 2nd, and the total cost of ground instruction plus a tandem flight at 1000 feet was $195. The choice was clear -- hang gliding and a Chipotle burrito!

This involved waking up early that man was meant to on a Saturday to meet the Meetup.com participants in Santa Clara at 6:15am and then continue the ride down to Trespines outside of Hollister, CA. Once we arrived there, we split into a ground instruction group and the tandem group. The ground training consisted of learning the basics: learning how to run taking long strides (the longer the strides, the less you bounce up and down, and the lower chance you have of rupturing air tension along the wings of the glider), picking up the glider and balancing it on your shoulders, learning how to orient the glider with or against the wind, walking the glider on the ground, running with the glider, and finally, freakin' lifting off the ground with the glider!

The tandem was a little more intense. They had a winch that pulled a tow line connected to the tandem glider, so that it would pick up enough speed to lift up. Man. There are very few things that I have experienced that are as intense as that initial lift -- you go up one hundred feet before you even realize what's going on!
After climbing to cruising altitude of about 700 feet, you see those birds flying around down there near the ground. I taunted them for being wusses. After circling around for six or seven minutes, we landed in the field right next to the runway.

The instructors had all been doing this for years, and had great stories; for example, jumping off of Glacier Point in Yosemite and hang gliding from one side of the valley to the other. Or, the guy who hang glided from Mt. Tam just north of San Francisco about a hundred miles south. Or, the guy who has the longest flight record at 440 miles. Yea. 440 miles. That's roughly equivalent to the distance between Cleveland, Ohio and New York City. In a hang glider.

Satisfied Customer

PixCede: Thoughts on how to proceed

I haven't done any "real" work on PixCede since late last week, but I've done a lot of thinking about it -- which is probably a lot better than implementing without thinking. I've also been using it myself more than I thought I would be, which is really exciting. I've told a couple of my friends and collected some feedback, but Zach is the only one who started really playing with it.

I've thought a lot about whether or not to allow emails from non-phones through. The main concern is the potential for abuse -- who's to stop someone from writing a script to constantly spam the front page with whatever? Zach had a good suggestion of getting a short SMS number (like GOOGL) and only allowing pictures to come in through that. Then, I realized that getting a picture from one of my computers to the other meant emailing it to myself, which for some reason just seems like a huge pain in the ass. I used GMail to email PixCede a picture of myself on the Half Dome, and just retrieved it through the web site on the other computer. Yea, still email on one end, but on the receiving end, I felt incredibly less inconvenienced. Therefore, I think I'm going to implement a simple file upload feature to PixCede so that you can post a picture from one computer quickly, and retrieve it from another computer quickly -- without the hassle of logging into email on both computers.

I know what you're thinking: great idea! TinyPic had that idea YEARS ago! Short answer, yes, long answer... not quite. For one, I really am going to try to model this site after PasteBin more so than TinyPic and other similar services. I really like the dead simple usage of PasteBin, and I also like the content expiration too. I feel like that would discourage people from using PixCede as an image hosting solution -- which it is not. PixCede is a simple way to transfer pictures from one device, be that a cell phone, a workstation, or something else I don't know about, to another. So in that respect, I hope to differentiate myself from TinyPic and Flickr -- this is more of a PasteBin fork.

So, speaking of PixCede, I realized that the owner made all of his code open source. I love not re-inventing the wheel, so I'm going to start reading his blog and poking into the source code. I've briefly scanned his blog, and I was happy to see that he switched from a database back end to just a plain file system -- a decision I arrived at with PixCede a week or so ago. For something this dead simple, a relational database is really overkill. It just seems like the common decision to "my web app needs to persist information" is to just cram it into a database without really thinking about it. A well designed file structure will work just as well in this case.

PixCede: First working version!

It now seems that PixCede is "working enough" to justify putting up that blog I've been avoiding. Thanks to Simple PHP Blog, I was able to get one up really quick without having to install database software.

After fighting with the mailparse extension for PHP, I decided to switch to the PEAR mimeDecode module. From what I understand, it will run a little slower than the extension, but it has the main advantage of working. The move to mimeDecode follows the initial proof of concept that used procmail to pipe new messages to the command line utility munpack, which was being called from within a PHP script. Currently, procmail pipes to a PHP script that does the MIME decoding in the same script. The only other thing I'm doing is creating a thumbnail for the main page, renaming the image, and storing it to disk.

Like I said, functionality is basic right now. I don't have a database running; images are just renamed to {timestamp}_{md5 of content}.jpg. The naming convention takes care of order of submission, and throwing the hash at the end makes sure that unless two people submit the exact same image at the exact same time, there won't be any collisions. But, then again, if the image is the same, and the submission time is the same, would you really need it stored twice anyway?

I still have a huge todo list:
* Allow images to be directly accessible
* Automatically update the main page when new pictures are submitted
* Video uploads
* Confirmation SMS with direct URL
* Browse by time periods

I'm not sure what the "final" product is going to end up looking like; it will just be a continual evolution. While this is completely open and anyone can use it, I'm assuming it's still going to be people I tell by word of mouth for now. If you have suggestions or comments, please let me know!

Introducing PixCede

PixCede is an idea I came up with on the bus into work one day. I've been doing some development on it in my free time, and finally put up a working "proof of concept" last week. There's a development blog on the site, but I've decided to mirror PixCede posts on this blog.

(ripped from my about page)
What is PixCede?
Take a picture with your cell phone camera, send it to submit@pixcede.com, and check pixcede.com to see your picture. Additionally, you can directly view your image using the submission code sent back to your phone.

Ok, that's what it does, but what is it?
PixCede is defined in purely functional terms because I don't care what you use it for. In fact, I'm very curious to see what kind of pictures show up on here. I'll worry about how it works, you worry about what to use it for.

That sounds dumb. Can't you think of anything to use it for?
Sure, I'll tell you exactly what I'm going to use it for: I take pictures with my cell phone camera, and then they just stay there. I'm just going to use this as a sort of Pastebin for my pictures, between cell phones and computers. I'd imagine a lot of the pictures I submit will be either things I think look cool, cool cars, or funny things. But seriously, you should use it for whatever you want. I'm just providing the tool.

Sounds like a cheap way to host pictures!
PixCede is not a traditional image hosting site. I am paying for bandwidth out of my own pocket. If you need a dedicated image hosting site, use S3, Flickr, Picasa, or any number of other services designed for that

So what are you getting out of it?
Entertainment. I, hopefully like you, am interested in seeing what kind of pictures get submitted! A couple of months ago I saw a site about a guy who took a disposable camera and tied it to a park bench with a note that said something to the effect of "take a picture of whatever you want, I'll develop the film once it's all been used." This is my virtual camera-attached-to-a-park-bench.

Am I giving you my pictures? Will you make money off of selling them?
First off, I really don't know who you are -- by design. But I certainly don't think it's right for me to own them either. By submitting a picture to PixCede, you are putting it into the public domain.

Are you making any money off of PixCede?
Not right now. If it really takes off and I have to start putting up AdSense to cover excessive bandwidth costs, I'll let the community know beforehand. For right now though, I'm taking a barebones Craigslist approach.

What does 'PixCede' mean?
The 'Pix' part is pretty obvious: pictures. The suffix '-cede' means to transfer. Altogether now... 'picture transfer.'

BASH Scripting: Inserting text into a fixed location of a file

Ever need to insert some test into the middle of a file? Turns out there's a pretty simple way to do it! Assume you want to put the contents of "addition.txt" into "myfile.txt" at line 10:

head -n 10 myfile.txt > output.txt
cat addition.txt >> output.txt
tail -n +10 myfile.txt >> output.txt
mv output.txt myfile.txt

Of course, 10 can be changed to a variable, and file names can be changed to variables.

Book Review: Here Comes Everybody

Recently appearing on the Colbert Report, Clay Shirky talked about his new book Here Comes Everybody. I thought it was a good interview and sounded like a good book, and then promptly forgot about it. Then I read an article I found on reddit: Gin, Television, and Social Surplus. It was easily the most insightful thing I'd read all day. Then I noticed the URL and decided to buy the book when I flew back to Cleveland for CWRU Commencement. This was one of those rare books that's hard to put down; I really hadn't been this engrossed since I read the Da Vinci Code in two days. Don't let the easy reading fool you; that just proves how well written it was.

One of the topics Shirky explores involves the activity of edits on a Wikipedia page. If you look at the revisions a page has gone through, most people who have touched the page make very small changes, and there's a very active minority who are responsible for the bulk of the work. This distribution follows the power law (I really enjoyed linking that to the Wikipedia page, btw). The power law, in brief, is a sharp curve that shows a skewed distribution (think back to Algebra II when you graphed 1/x on your TI-83). This is odd in that an "average" Wikipedia contributer (not just a viewer) has a really low contribution rate, but that there are a few people that put forth enormous contributions. This doesn't just work for Wikipedia, though -- the economist Pareto discovered this fit a wealth distribution for countries he studied. The interesting thing about this is, with relation to Wikipedia, is that you can't look at just the individual. An "average" user, in the mathematical sense, has contributed very little to the page. You have to look at a page by the group that has worked on it, because it only exists as a result of the group. This explains a number of other online activities: Flickr postings, web blog viewing, e-mail discussion threads. Really cool stuff, and I'm not doing it justice.

The book focuses on what advancements in technology have done, namely, broken down barriers. The cost of organizing people has gone down greatly with the advent of blogs, Facebook, MySpace, Meetup, and any number of other sites. But Shirky doesn't focus on what has been done already; almost quite the opposite. This is what's being done now. Here's the problem people have now, and here's the tools they are using. These tools aren't limited to this problem. Maybe something else will be done using them in the future, who knows? This reminded me a lot of a podcast I heard by Marc Andreessen, co-founder of Ning. Ning is a platform for building social networks, and one of the things Marc mentioned was that they don't try to control what people use it for -- they just enable the community aspect of it. One of Shirky's themes throughout the book was that the internet will change the world the way the printing press did: in an unexpected way. When the printing press came around, the first thought was, "Ok cool, more Bibles." It quickly went far beyond that, to literature, flyers, and newspapers to name but a few. For most of the history of the internet, we've used to to do things online the same way we would in real life. It's just cooler because it's online. Only in recent years have web sites started to harness the power of the internet for what it is, and this is only the tip of the iceberg.

Something that was cool to me was when Shirky mentioned Martin Wattenberg at IBM Research. Martin was the manager of the Many Eyes project that my Extreme Blue team worked with last summer. Martin and crew wrote a paper on visualizing Wikipedia activity. Basically, I felt like a bad ass for sitting on a conference call with him.

It's a great book and a quick read, but chock full of awesomeness.

CBAX 1: Gas Prices

At my girlfriend's suggestion, I'm trying to make this observation into an XKCD:




So yea, apparently, you can make comics that are drawn simpler than XKCD. Also, CBAX stands for "Could Be An XKCD"

Part 2: Drupal 6.2 and DB2 Express-C 9.5 on Ubuntu 7.10

This is part 2 in my series on getting Drupal 6.2 and DB2 Express-C 9.5 to play nicely with each other. In the first part, I installed DB2 Express-C 9.5. In this part, I'll look at getting Apache/PHP5 setup and DB2 working from within PHP.

Step 1: Configuring DB2. There were some additional configuration steps needed for DB2, which are detailed on the developerWorks forum. There were some superfluous steps, but the worst you're going to do is overwrite something with the same thing. In particular, all the user accounts were set up already. If you're not familiar with DB2, you can issue commands from the shell, in the form of:

db2 "create database testdb"

but just make sure you're running these commands as the user db2inst1, or you source db2inst1's db2profile. If you can create `testdb`, then DB2 is working correctly.

Step 2: Install Apache httpd 2.2 and PHP 5.2. The Ubuntu Guide has information on setting up Apache and PHP, but the quick and dirty steps are:

sudo apt-get install apache2
sudo apt-get install php5 libapache2-mod-php5
sudo /etc/init.d/apache2 restart

Step 3: Install DB2 support for PHP. There is a PECL module for IBM DB2, which can be installed with PHP's PEAR (PHP Extension and Application Repository, like apt-get for Ubuntu).

sudo apt-get install php-pear
sudo pecl install ibm_db2

When I first ran this, I got the error: "sh: phpize: not found". After doing a quick search, I realized that I needed the PHP development files to be able to compile the DB2 driver:

sudo apt-get install php5-dev

After the module is compiled, it will ask you where your installation of DB2 is. In my experience, it wouldn't believe me when I told it where DB2 was installed to. Luckily, it ended up not mattering. When you get to this part of the PECL install, just hit Ctrl-C.

Next, I needed to configured PHP to use the IBM DB2 driver. In `/etc/php5/apache2/php.ini`, go to the extension section and add:

extension_dir="/opt/ibm/db2/V9.5/dsdriver/php32"
extension="extension=ibm_db2_5.2.1.so"

Note: correct the path if you have your module installed somewhere else; do a `locate ibm_db2_5.2.1.so`)

After making the changes, I reloaded Apache:

sudo /etc/init.d/apache2 reload

Verify that the module is loaded by making a quick PHP page in /var/www/ with the content:

<?php phpinfo(); ?>

and make sure there's a DB2 driver loaded. I started playing around with DB2 in PHP, but it was getting late and I decided to put that off for another day. IBM has some information on developing PHP5 for DB2, but it's barebones at best. I'll post a sample script next time.

Ok, so at this point, I have Apache, PHP, and DB2 Express-C 9.5 all installed and playing well with each other. In the next article in this series, I'll look at getting Drupal 6.2 to use DB2 on the back end.