The Art of Waiting
My family consists of myself, my wife, my soon to be four year old daughter and my approximately six month old son. Not long ago we were getting the group ready to go somewhere, although I don’t recall where. (Yes, I’m getting old.)
Anyhow, the situation devolved into me sitting on the couch shoes on feet and keys in hand holding my son who was also completely dressed and ready to go. My daughter was rumaging through the shoe box trying to decide what shoes to wear and my wife was doing two or three different things at once in at least two different rooms in the house.
He looked up at me with a big pair of blue eyes asking without words because, well he can’t talk, “What in the world is taking them so long to get ready?” To that I looked down at him, conjurned up my best I’m-about-to-impart-fatherly-wisdom voice and said:
“Boy. Sometimes you just have to wait. There’s no good reason why and there’s nothing you can do about it.”
“Also, save your pennies so you can buy an iPhone. At least then you’ll have something to play around with while you’re waiting.”
I was reminded of this story today when I was setting up a test database. You see I wanted to gauge the level of suckiness of my code when confronted with a massive data set. The problem was that I needed to copy the test data from a network share onto my local machine and that it consisted of a zillion little files. Transferring one huge file, that’s no problem. It comes down in a few minutes just like downloading a bootleg movie, err, business task critical ISO image. Transferring loads and loads of little files though means it is time to go take a walk. Or make a sandwich. Or paint your house.
Worst part about it is that there was nothing I could do to make it go faster. I suppose if I was feeling really abmitious I could have hacked my way into the datacenter, pulled the drive from the server, somehow enclosed it and attached it directly to my Macbook Pro, and then outrun the IT staff. Not a great solution, especially considering that even if I pulled a Usain Bolt on them I’d still be greeted by a pink slip at the podium.
So as I sat, and as I waited, I tried to think of other similar software development tasks that require you to just sit around and wait. Much to my surprise that was the only one I could immediately think of. Even compilation, especially if you are like me and get to watch C++ chug, bend, and contort itself into binary form, isn’t that bad thanks to distributed build tools like Incredibuild or Xcode’s support of distcc. Most other problems have been solved by Moore’s Law, although as shown by my trusty laptop I have a tendency to agree with Wil Shipley about running your code on older development machines:
“IL: Finally, what is/are your main machine(s)? Hardware, software, processor—all the gory details. We’re geeks; we like this kind of stuff.
WS: At the café every day I still use a 1.67 GHz G4 PowerBook, with 1.5 GB of RAM. I’m intentionally waiting to buy a Core 2 Duo machine until I ship Delicious Library 2, because Delicious Library 2 runs so fast on the Core 2 Duos it would be unfair for me to use one day-to-day—I’d never optimize my code, and people stuck on old PowerBooks would hate me. I know a company where they recently bought every engineer a 30″ display, and my thought was, “Oh, man, it’s gonna be hard to use their apps on a portable… .” There are so many problems you don’t solve unless they are bothering you personally. I need to eat the same caviar I feed my customers. (What, you think I’d feed them dog food?)”
— Ars Technica interview with Wil Shipley of Delicious Monster
Well, the progress bar is a little closer to done. I’m going to get something eat. Come to think of it the house could use a little more color…
I had to build a file the other day and get it across town to do verify that we had each and every record on our system that the cross-town people had on theirs. Having no physical access to EITHER system, I did the same thing — split my list o’ IDs up and copied them down one at a time.
Three hours or so into it, I realized I was just downloading a lot of little lists of sorted integers that I could really treat as one big list, so I opened up my trusty shell, wrote a quick perl script to do a half-assed RLE that was really efficient at compressing sequential lists, ran it on the whole dataset, downloaded the much-much-smaller resulting file and wrote a decoder on my laptop.
Of course, by the time I worked out the bugs and had the small file downloaded, the much larger bunch of files had already transferred, so I ended up with two copies of my data in the same amount of time it took me to get one copy!
Seems inefficient in retrospect. That and, in the end, I had to load the files on a USB drive and sneaker-net them across town. Wasn’t technology supposed to fix that by now?
You have overlooked one major software engineering task that requires waiting, though: the pre-code stage. Requirements gathering, all that. Of course, it’s a stage I regularly skip, but I’ve SEEN people waiting on it, so I know it happens.
Comment by ChipMonkey — August 28, 2008 @ 12:19 pm