apenwarr - Business is Programming

2008-08-27: More Artemy Lebedev

Tue, 19 Aug 2008 18:38:35 -0400

More Artemy Lebedev

Art Lebedev writes about his concept of a Unit of Sense: something new which you create and which requires intelligence to do.

He claims that the average "good designer" produces 1-2 units of sense per month. An excellent designer, maybe four. And an artistic director, at least 6. He also claims that a fair way to pay designers is to multiply their produced units of sense by a fixed dollar value.

"Things like good ideas, interesting concepts and fresh techniques are units of sense. A one-hundred-page brandbook may also be considered a unit of sense - not on a per-page basis, but as a whole."

The concept is really interesting, as it offers a way to quantify designer output, something that's notoriously hard to measure. Unfortunately, his description is itself a little hard to quantify.

Obviously this is hard to apply to anything other than designers, as the majority of work that most people do is perfectly valid work, but simply isn't creative. I guess they should still get paid, though.

One way to think of it is like this: as a programmer, perhaps the interest level you have in your job is a function of your natural unit of sense creativity level, as compared to your creativity level at this job. Or maybe your job satisfaction is indicated by your creativity level, not the other way around.

2008-08-25: Art Lebedev on the proper formatting of punctuation

Tue, 19 Aug 2008 14:23:51 -0400

Art Lebedev on the proper formatting of punctuation

This guy continues to blow my mind. Some tips on formatting punctuation in a visually appealing way. Check out the section on how to center things horizontally. And naturally there's a bonus section on the proper formatting of smileys as punctuation.

Reading this stuff is a good way to remind myself why I'm a crappy visual designer.

And also via Art Lebedev: Leonardo da Vinci invented stinkbombs.

He also has a brilliantly simple way of expressing the success of a design in terms of its requirements. "Design is not something to discuss ... the only thing that can be discussed is whether the task has or has not been achieved."

Plus there's an article on designing well by designing badly on purpose. This method is ingenious, since it lets you imagine your users as super-intelligent enemies and then work to defeat them. Considering how hard it is to really believe how helpless users can be, overestimating them could be much easier.

2008-08-23: CSS Still Sucks

Tue, 19 Aug 2008 13:57:28 -0400

CSS Still Sucks

Here's a nicely disgust-filled article about various problems with CSS. He seems bewildered by how something so bad could have been invented, but it's really very simple. The people who invented and then standardized CSS had never used it. After all, they invented it before any web browser supported it.

The best languages are invented by the people who are using them to do real work, while they use them to do real work.

I said something similar, with examples back in 2004.

2008-08-22: gitbuilder

Fri, 22 Aug 2008 23:28:37 -0400

gitbuilder

Well, it seems to be my week for small, handy tools. As a followup to cron2rss from a few days ago, I now present to you: gitbuilder. It's an autobuilder tool for your favourite git-based project, with built-in bisection support.

I've been working on this for a while, but I finally decided to get it production-ready a few days ago after I spent several hours tracking down a problem... that turned out to already be caught by our unit tests. Except nobody had run the unit tests recently. Oops. Well, now I'll run them for you, thank you very much.

Check out a sample of the results at the Versaplex autobuilder page.

Side note

GitHub.com is way more addictive than I thought. It's what SourceForge and Google Code should have been: a really easy way for people to publish, fork, and merge source code, with a few extra hyperlinks thrown in here and there for good measure. The key thing is the non-global namespace, so you can just dump stuff there and give it to people whenever you want; it's like an extension of your local ~/src directory. Thanks to wlach for convincing me to try it.

2008-08-21: cron2rss

Mon, 18 Aug 2008 19:09:04 -0400

cron2rss

At work we have a lot of servers, and sometimes they have problems. Even though *I* mostly just handle development servers, not production ones, it's still kind of a pain to keep track of them all and it's always embarrassing to find out from a co-worker that one of them has stopped working.

So I use a bunch of cron jobs to do a bunch of daily tasks and make sure things are flowing smoothly, but this isn't really all that optimal. The problem is that you don't want cron emails when tasks *work* (some of them are running every ten minutes!). But you *do* want emails when they don't work. On the other hand, you don't need an email every *ten minutes* when something breaks. And moreover, you really want to be notified when the server the task was supposed to run on has crashed and the job doesn't run at all.

Sadly, cron itself doesn't do a very good job of this, particularly that last part, where it's completely useless.

So I spent a few hours today whipping up cron2rss. It reads, saves, and eventually expires the stdout/stderr output from multiple cron jobs on multiple computers, then turns the result into two RSS feeds: one with everything, and one with only the failures. *And* it auto-inserts entries into the feed whenever it's been too long since one of the tasks has produced a log message. The RSS service can also be run on more than one computer at a time, so that if one of your RSS feeds dies, the others can still tell you about a failure.

I leave you with this food for thought:

   0,10,20,30,40,50 *  * * *   ~/cron2rss/add test-website wget -O/dev/null http://versabanq.com

How do you set up something like that if you don't have my tool? What would you do if your test was more complicated? What if you had 50 servers instead of 1?

2008-08-19: Dog on Treadmill

Sun, 17 Aug 2008 17:29:46 -0400

Dog on Treadmill

Yes.

2008-08-17: Being an entrepreneur

Sun, 17 Aug 2008 16:40:27 -0400

Being an entrepreneur

Being an entrepreneur doesn't mean you're smart. A smart guy will figure out how to make a bunch of money while taking on much less risk. Corollary to this, there are a lot of dumb entrepreneurs out there.

-- Ted Dziuba, the uncov guy

It's like he can read my mind, then swears a lot.

2008-08-08: ROE

Tue, 05 Aug 2008 20:18:36 -0400

ROE

Havoc Pennington has an excellent article on Return on Equity (ROE).

2008-08-06: ACPI reprise

Tue, 05 Aug 2008 17:12:36 -0400

ACPI reprise

Wow, my Linux disk-related rant seems to have been featured on Linux Hater. I am strangely pleased, although for the record, I hate all computers and all operating systems, not just Linux.

Now, I've been advised that another hot topic right now is some kind of ridiculous conspiracy theory about a laptop vendor who "deliberately" made Linux's ACPI implementation not work on their system, while Windows works fine. Without going to the unnecessary effort of looking up the article or checking any facts, I can tell you that I already thoroughly debunked all Linux-ACPI conspiracy theories a couple of years ago. You may find it either horrifying or funny.

2008-08-05: Bug trackers don't work

Tue, 05 Aug 2008 16:18:26 -0400

Bug trackers don't work

Weird. The LinuxHater guy wrote an article saying that wide-open bug trackers don't work, and said a bunch of things that I had been planning to say. I also made some related comments earlier. Here are a few more:

Lately, there's been some discussion about distributions like Ubuntu just closing out massive numbers of bugs with comments like "Please try it in the new version and reopen if the bug still exists." Naturally, users are offended by this: I played your game! I filed a bug like you asked! I included reproducible test steps! I even included a patch in some of them! Couldn't you at least do me the basic service of running through my reproduction steps before closing the bug?

The answer is: no, actually, they can't. There are way more users than developers and testers, and those developers are assembling and tracking bugs for all the software that will run on your entire computer. It is just too much work to do properly.

Not that this justifies the crappiness; it just explains it. My point is you can't just say, "They shouldn't have closed my bug!" as if it were a solution, because it's not. Developers also can't say, "Please just reopen this bug if it still exists!" as if it's a solution, because it's not either.

So far, nobody has offered a solution that would actually work.

2008-08-01: ext2resize sucks, but apparently so do I, and also a lot of other things too

Fri, 01 Aug 2008 22:44:28 -0400

ext2resize sucks, but apparently so do I, and also a lot of other things too

This has not been a happy week for my desktop workstation. It started off good: two brand new SATA 750 GB disks to replace my old 80 GB one. What a difference! Plus, this was finally the inspiration I needed to get organized and install Linux as my base system and Windows XP in VMware, instead of the other way around.

So the first thing I did was take out my 80 GB disk and put it aside for later, in order to avoid screwing up, which is what people tend to do during such activities. I booted a Debian Etch CD and proceeded to set up my two disks in a RAID-1 mirror, just like I had planned.

This is where I stop to complain about Debian Etch's installer. What the heck were you people thinking? Steal Ubuntu's, already! And the default mode is to install with LVM but *not* with a RAID device backing it? And in order to just set up a RAID on two identical disks, you have to go through like 25 steps in your awful installer UI? This is really, really disgusting.

pphaneuf's rule for flaming is that you should shut up and not complain unless you at least know a better way to do it. And better still, you should have done it better. Well, I have. Nitix's disk configuration stuff is fully automatic and pretty much foolproof. Sometimes it can be a little tricky to wipe out a disk that already has stuff on it. But that was on purpose. Other than that, it's just plain easy, and your disk gets configured with RAID and LVM layers included even if there's only one disk, because that makes things a heck of a lot more consistent, let alone making it really easy to add a RAID later.

Anyway, I got through the painful disk installer UI after a few tries (you have to reboot if you get it wrong! Genius!), and installed my system, and rebooted, and all seemed okay, for a while.

That's when I installed Firefox 3 and my system suddenly became prone to huge disk grinding delays. Every time a program started writing to the disk, Firefox would freeze, sometimes for many seconds at once. And Firefox 3, you see, also writes to disk, so this was no rare occurrence.

You see, Debian had installed my new ext3 filesystem using the kernel's default data=ordered option. data=ordered is one of those things that I'm shocked was allowed into the kernel at all, let alone made the default. Basically, it means that all relevant data will always be flushed to disk before its journal (metadata) entries can be flushed, so your files will never contain data that was just leftover if your computer crashes between metadata and data updates. Sounds great, right? Well yes, until you think about how you actually have to implement that. The journal is always flushed sequentially, so if someone calls fsync() on a file, we try to flush its data first, then all the metadata changes that were made on disk up to and including this file's metadata. But that metadata leading up to this change, of course, can't be flushed until we include all the data related to all that metadata. This is a long story, but essentially, it makes fsync() of a 4k file turn into essentially a sync() of your entire disk. Result? Firefox 3, which fsync()s like 300 times a minute for no good reason, grinds to a halt. And your system performance is craptastic because basically your disk's write cache is gone.

After reading the mostly retarded discussion in the Bugzilla case, I gave up on the Firefox guys actually fixing the problem anytime soon. fsync() slightly less often?? That's a fix?? What part of "every time I fsync() it syncs ALL outstanding transactions immediately to disk" do you not understand?

So, I made a shared library called libnofsync.so that just makes fsync() do nothing, and used LD_PRELOAD to load that into firefox. By bookmarks are not a bloody ACID database! Nobody cares if they get corrupted! I only ever visit like three sites, and two of them are GMail! Get over it. With this hack, I suppose my Linux Firefox 3 is probably the fastest one around, because data=ordered or not, everybody else is doing multiple synchronous disk writes every time they try to load a page. Good grief.

The next task is to get rid of the completely obnoxious data=ordered setting, which involved messing in my grub configuration file(s). Grub, if you haven't heard me rant about it before, is a total complete pukefest. It does absolutely nothing of value that lilo doesn't do, but it does it in a way that's about 1000x more complicated. Then Debian layers another pile of crud on top. The long and the short of this is that I COULD NOT FIND A WAY to add "rootflags=data=writeback" to my kernel command line in any sort of permanent fashion. Now, you can edit the kernel command line during boot, and there's an option right there that says "savedefaults" that sounds promising but does absolutely nothing, but (at least) one of the 1000 layers of garbage shoveled on top of grub overwrites and/or ignores my config file changes no matter how I try to make them.

So I uninstalled grub and installed lilo, after which things were trivially easy because lilo was not written by morons.

Around this time I tried out a 2.6.25 kernel from backports.org, which happily crashed my system repeatedly. What the heck was I thinking? The kernel hasn't actually been missing any significant features for something like five years. I don't know why I upgraded it, but I sure won't make that mistake again. Anyway, the crashes served mostly to teach me that if you crash your system while the RAID is rebuilding, it might not come back all by itself. Instead, it silently kicks the second disk from the RAID and leaves it idle, without actually telling anyone about this INCREDIBLY SERIOUS RELIABILITY LOSS. But that's par for the course by now. I added it back into my array and off I went.

Fast forward to a few days later; it took some suffering before I bothered to fix the firefox nonsense and got sick enough of the random kernel crashes to downgrade my kernel back to Debian's stable version.⁽¹⁾

That was about when I noticed that my new, fancy-pants RAID was rebuilding at 80 MB/sec, which is fabulously quick, and yet had still not finished rebuilding, after being left uninterrupted for more than a day. What? Well, let's "watch cat /proc/mdstat". Hey, check it out! It's almost done! 98%... 99%... 100%... 100%... 100%... 0%? Hey! What the heck is this! Okay, look at dmesg. Aha, it had a bad sector right at the very end of the disk!

And so it decided to start rebuilding the RAID from scratch!

About once an hour since I installed it at the beginning of the week!⁽²⁾

HA HA HA HA!

Now, that's not actually something that would solve the problem even if I had bad sectors. But of course, I don't really have any bad sectors. What I do have is some partitions that apparently Debian's installer screwed up while creating, so they happily run right past the end of the disk.

Now okay, Debian's partitioning thingy has an excuse for being buggy; probably nobody ever uses it to make a RAID, because it's sure the heck not easy to do. But here's the thing: mdadm let me create a RAID on a partition with inaccessible sectors. Then mke2fs let me create a filesystem on that broken RAID. Did it not occur to anyone to sample a few of the sectors before you decided to actually use them? Didn't any of you ever wonder why Windows does that weird thing about "testing sector accessibility" whenever you make a partition? No, apparently not. Gargle.

Okay, so obviously I need to make my partition a little smaller. Apparently it's possible to resize ext3 partitions and RAID devices now. That's good news, right? Well sure! Let's try that!

So I switched down to single user mode, remounted my rootfs read only, and ran "ext2resize /dev/md0". It told me a magic number, which is the number of sectors it's currently using. I reduced that number by an overly large factor (hey, I've got whole gigabytes of data to waste here!) and ran "ext2resize -v /dev/md0 NUMBER". It grinded away for a while, giving me impressive yet scary messages about how it was moving inodes around, and so on. I figured it couldn't really do anything too harmful, since obvously the space at the end of the disk was nowhere near any of my actual data.

Boy, was I ever wrong.

I foolishly then ran "ext2resize /dev/md0" again to see if it would print out the new size. Except, it seems, that's not what it does. What it does is try to resize the partition again, this time to the maximum size. The maximum size is, as you recall, a size that involves some nonexistent sectors at the end of my partition.

So it moved a few more inodes around and then errored out. Ironically, ext2resize does apparently access that area at the end of the disk, even if mke2fs doesn't. Sadly however, it moves a bunch of crap around before erroring out and aborting midstream.

You might have imagined, as I might have, once, that ext2resize would maybe do a "hypothetical resize" operation, going all the way through the disk and confirming that everything would work - like the last sector it was about to resize into, for example - before it actually starts moving crap around. Or you might have imagined that it would undo those changes before it aborts due to an unexpected error. But if you thought that, then you, like me, would have been completely wrong. ext2resize does no such thing. Instead, it moves a bit of data around, and then aborts when it gets confused, halfway through the process.

As I learned, this makes your filesystem completely unusable. It turns out that, for no reason I can possibly imagine, moving around the completely empty unused section at the very end of my disk also involves rewriting inode 2 (which turns out to be the root directory), as well as an impressive number of other I-woulda-thought unrelated files and inodes. Of course, when you do this wrong, your filesystem stops mounting.

Time to boot the rescue disk one more time, pray a little, run e2fsck, and pray a little more.

e2fsck was not impressed. It correctly noted that inode 2 was, if I recall, "conflicted." Also lots of other horrible things. I asked it to repair everything. It crashed. Well, of course, it didn't crash exactly. It printed messages about "programmer error??" and then restarted the game all over again. I actually went through the game a few times before I finally caught on to the fact that it was the same every single time.

Luckily, when I bravely answered "no" to all the questions about whether it should fix things, it finally fixed things⁽³⁾, complained that Oh God Your Filesystem is Still Broken Though, and exited. But whatever, my filesystem finally mounted again. I had to recover all my root-level folders from their new homes in /lost+found, but oh well, at least I still had my data.

And so I rebooted, and my RAID promptly started rebuilding itself again, and my filesystem was still corrupt, and it couldn't actually be fsck'd because e2fsck would still go into an infinite loop if I tried. Back to square one.

And now, this is the part where I flame myself, because I forgot something I already learned a long time ago and then sold to thousands of people:

Why the heck are you using a RAID in the first place, when you only have two disks, idiot? If you only have two disks, just back up your files to the second disk occasionally using rsync or something. That way, when you ext2resize or just delete a file by accident, the other disk doesn't reflect your idiotic mistakes until a while later. Remember?

So there you have it. All those things were dumb, but I was the dumbest of them all.

I dropped the second disk out of my RAID, repartitioned it correctly, ran mke2fs, copied all the files to the second disk, and booted from that. Done.

Actual Nice Things to Say

Lest it appear that I only hate things:

⁽¹⁾ Yes, Debian's stable 2.6.18 kernel actually works, and thus there's actually a reason they don't carry the highest-numbered one. Good job, guys.

⁽²⁾ The guys who implemented the CFQ disk scheduler are my heroes. Unlike in the 2.4 kernel, where rebuilding your RAID hugely degraded your system performance, in 2.6 this operation happens at "idle" disk priority so it can go at pretty much full speed and yet have zero impact on your disk performance. That's why I didn't even notice for days that the rebuild was going on. Related tool: ionice -c3. Use it for your background compiles and stuff. It's awesome.

⁽³⁾ You know what? I'm a big fan of e2fsck, even though it's supremely un-user-friendly and obviously had some bugs here. But despite those bugs, it didn't abort when it thought it had a "programmer error," and it saved my data from what I was sure by then was certain death. No program is perfect, but at least this one was written by sane people.

2008-07-22: iPhone 3G

Tue, 22 Jul 2008 18:52:15 -0400

iPhone 3G

"If you've ever wondered what Apple will be like after Steve leaves and they've got visionaries like Phil Schiller and Tim Cook running the place, well, now you've seen the trailer. The movie will be worse."

-- Dan Lyons (aka Fake Steve Jobs) on the iPhone 3G release

2008-07-17: It lives

Fri, 18 Jul 2008 00:05:12 -0400

It lives

The company that I co-founded and eventually left lives on as Lotus Foundations.

There's even a guy blogging about it. (He includes some screenshots of the new webconfig and boot screens. The webconfig update looks quite classy.⁽¹⁾

It was also featured in the keynote at Lotusphere 2008.

And speaking of things that aren't dead, I discovered all these links using the beta of Pressflip, formerly Persai, a surprisingly non-sucky news engine from the guy who made uncov. I'm not usually impressed by Web 2.0 startups, to say the least, but this one is pretty cool. Check it out.

Footnote

⁽¹⁾ The boot screen is less classy; when they changed the "Nitix - Autonomic Linux" title in syslinux to say "Lotus Foundations", they centered it incorrectly, probably because they counted the colour code bytes as visible characters. Probably nobody else would ever notice this, except that I've done it myself in the past.

2008-07-04: Risk vs. Calculated Risk

Wed, 02 Jul 2008 15:54:11 -0400

Risk vs. Calculated Risk

It's accepted knowledge that potential returns in business are greater if you're willing to take on more risk. That is, a "safe" investment with guaranteed payoff will generally offer less than an "unsafe" investment that might fail. And so, if you have lots of money, as the wisdom goes, you might want to invest it in a wide variety of risky investments in order to collect some of that potential gain. The idea is that even if some of the risky investments fail, the increased profits from the non-failing ones should more than pay for the loss.

Okay, so far so good, but the problem with risk is that it's random. And worse, there are a lot more worthless risky investments than valuable ones; such is the state of humanity. (In contrast, "safe" investments, by definition, don't really have that problem.)

Enter the concept of a "calculated risk." The idea here is that if you're careful, you'll be able to see which risky investments are better, and thus collect on the returns instead of losing all your money.

But there's a problem with that theory. The problem is that if everyone could just figure out what's a "good risk" vs. a "bad risk," then everyone would want to invest in the same things. Thanks to supply and demand, this basically sucks the profit out of such obvious investments. Various problems with fancy hedge funds and the recent subprime crisis can all be traced to this problem: everyone agreed that a particular risk was "good" at the same time, and over-invested in it until the profit was gone.

But wait! If supply and demand can suck the life out of risky invements, what makes risk such a great thing, again?

My entrepreneurship professor in university used to say that a better term for calculated risk would be "risky calculation," because that puts more emphasis on calculation than on risk. And the more I learn about business, the more I realize how absolutely right that is. Profits don't come from risk. Of course they don't! Who wants to buy pure risk, except compulsive gamblers? Profits actually stem from the quality of the investment, and you can buy into an investment at a good price only when other people don't. Perceived risk is just one possible reason that people might avoid a particular high-quality investment.

I don't have huge boatloads of money, so I don't usually think about investing from the point of view of investing money. I think more about where to invest my time, because my time is the most valuable thing I have. Right now, the cool thing for a programmer like me to do is to form a web startup. But I'm not doing that, because the space is risky and overinvested. It's completely wrong on both axes. If you want to be successful in the market right now, do something safe and underserved.

2008-07-02: Toronto Fringe Festival Suggestions

Wed, 02 Jul 2008 15:54:05 -0400

Toronto Fringe Festival Suggestions

Congratulations, Torontonians, your Fringe Festival starts today and runs until July 13th. Since a lot of shows tour from East to West and Montreal is the first one of the season, my friend Elaine asked me to recommend some good shows based on what I saw in Montreal. So my good deed for today will be to offer you some assorted preview reviews:

Blastback Babyzap. Go see it. It's by Uncalled For. QED.

Teaching the Fringe, by Keir Cutler. Except that Dr. Cutler won't be able to make it to your fringe, so Barry Smith, of Barry Smith's Baby Book (not to be seen in Toronto), will be doing the show on his behalf. That's a bit strange, because both Teaching the Fringe and Baby Book are autobiographical, and so now Barry is doing someone else's autobiography. Oh well, both of them were good. You might as well give it a shot.

American Squatter, also by Barry Smith. Apparently it was popular. I didn't see it, but Smith himself is pretty funny. Who knows?

'Beth. As advertised.

Transcendental Masturbation. Not as advertised.

Wonderbar! and Balls! Both titles are exclamatory, but both shows I missed.

The Diaries of Adam and Eve. Luckily for you, it's cancelled.

The Tricky Part. Didn't see it, heard it was awesome.

Die Roten Punkte. Go see this. Really.

Telegrams from the New Canadian Cinema. Didn't see it. Heard nothing about it.

Time to Put my Socks On. Be sure you know what you're getting into.

Crude Love. Reputedly good. I didn't see this one either. Gee, I'm starting to sense a pattern here.

Greed. Well done. Time-twisted plot exposition. Cute girls.
Update 2008/07/02: Hmm, I've been told there was only one girl. Anyway, she had many cute outfits.

Jem Rolls. Dude, it's Jem Rolls. Of course you have to go see him. If you don't know why, then you have to go see him even more.

Mating Rituals of the Urban Cougar. Pretty good, especially the songs. Unless I got it mixed up with something else, that is. Actually, I'm pretty sure I did. Well, it was pretty good anyway, I think.

The Spy. This came to Montreal for a one-night show. The guy was super nervous and unfortunately it showed. But the show was otherwise excellent, and if he can get into his groove, it'll be awesome. Maybe wait for a later showing just in case.

Totem Figures, by the increasingly infamous TJ Dawe. The guy talks at top speed for 90 minutes and doesn't forget any lines. Worth seeing just for that, although he seems a little oversold to me. Reputedly the only guy to ever make a full-time career out of doing fringe shows.

And two troupes with three shows that you won't be seeing because they're not coming to Toronto but they're so awesome that I have to mention them anyway:

Identity Crisis by Influxdance. More serious and scaled down than previous years, but still excellent. Sadly, they seem to be done for the summer. Too bad.

The Cody Rivers Show: Stick to Glue, and Boom (by one guy from the Cody Rivers Show). Neither of the members is named Cody Rivers, but they're both awesome. Looks like they'll be in Edmonton, if you're lucky enough to live there, or Minnesota, if you're not.

And by the way, if you're going to the Toronto fringe, make sure you mention to them how lame the Fringe Donation Buckets system is, where you basically have salespeople nagging you while you line up for every single show. Please, please, if your fringe festival is going to collapse without these donations, then just charge a service fee per ticket like they do in Montreal. Seriously, guys. It's either optional or it's not. Stop harassing me already. You want people to go to more shows, but the lineup harassment gets less fun every time.

But all that said... you Torontonians have a great selection of shows waiting for you, including a huge number that didn't make it to Montreal. Have fun fringing!

Previously.

Update (2008/07/02): jnc has some alternative opinions.

2008-06-30: A serious case of JustOneMoreBug

Sun, 29 Jun 2008 19:29:23 -0400

A serious case of JustOneMoreBug

I just read Waking Up from the Economy of Dreams, a pleasant essay about how Webmind, Inc. went bankrupt in 2001 while trying to invent artificial intelligence.

The essay is an interesting read, mainly because the author seems to have come out of the experience so... innocently. We were 3/4 of the way to a real, working AI, he insists near the end of the story. It was just bad luck that we ran out of money right as the .com bubble burst...

You might almost be able to believe him, except he also told the rest of the story. The original development was supposed to take a few months. To build the first sentient AI! Unsurprisingly (to me), it took longer than expected. They actually ran out of money multiple times in the process, but managed to land financing at the last minute each time. (Incidentally, that sounds to me like a long string of good luck that finally ended, not any bad luck at all.)

But okay, lots of startups almost run out of money, and just about everyone underestimates the amount of work involved in their project. So what?

Well, the problem is that these guys felt sure, at each failure along the way, that they were just one step away from having it finally work. And when they finished that step and it still didn't work? Well, now we're just one step away. We're almost there. We can feel it!

At NITI, we called this the JustOneMoreBug syndrome, spelled using WikiCaps because it was a page in our wiki. It seems now that we must have been the ones to invent that term, since I can't seem to find it in Google anywhere. So with that in mind, I guess I'd better tell you what it is:

First of all, the fact that this was featured in our internal wiki probably tells you something about our experience with it: ie. we had plenty.⁽¹⁾ JustOneMoreBug is exactly what it sounds like: the feeling that if you can just fix this one last bug, your software will work perfectly, customers will come flocking to you, and riches will inevitably ensue.

The frustrating thing about JustOneMoreBug is that every successful project does have a moment exactly like that: the transition from unreleasable to releasable, from unusable to usable, from unsellable to sellable. And so if you're an optimist, you, like us, would prefer to believe that this bug is your one last bug. And maybe it is.

Unless it's not, which becomes clear one way or the other shortly after you fix it. Then what?

Then, if you're experiencing the JustOneMoreBug syndrome as opposed to the JustOneMoreBug major life-changing event, you realize that there's just... one more bug holding you back.

Fool me once, shame on you. Fool me twice, shame on me. You can get away with claiming JustOneMoreBug once or twice per project - but if you go past that, you have to stop and rethink. Why are there so many bugs that you don't even know about yet? Believe it or not, really good programmers will write programs that mostly work the first time, and the parts that don't work can fairly easily be uncovered and solved. If your program is fooling you about how stable it is, it's because your program isn't very good. You need to stop and find a new strategy: often, by doing some heavy testing and using the results to build a brand new multi-month development schedule. That's bad news, because the project was supposed to be done tomorrow. After all, there was just one bug left.

Webmind was on a multi-year marathon of JustOneMoreBug. I feel sorry for them, because advanced research has a strong tendency toward that; structured engineering principles just don't work for software research projects. But it seems like their investors didn't realize they were doing software research, and their researchers honestly didn't realize - repeatedly - that the JustOneMoreBug feeling is just a feeling, not a reality. And if the essay is any indication, they still don't realize it; they still think they were almost there.

At least they didn't come out cynical.

Footnote

⁽¹⁾ NITI had two kinds of JustOneMoreBug experience. First, with certain internal software projects (which is normal occasionally in any software company, although if we'd been smarter we would have caught them sooner). And second, with our business model: even if your product works, if it's not selling as well as it should, there's a strong tendency to believe that you just need to tweak one more thing about your business and it'll take off. But just like with software, you shouldn't fall for this trick more than once or twice. We fell for it repeatedly, and that was our multi-year marathon. (Eventually the winning fix wasn't a tweak. It was a complete re-analysis and redefinition of our target market, and a corresponding total change in the way we dealt with customers and resellers.)

2008-06-28: Weird things about git, #2: no bug tracking system

Fri, 04 Jul 2008 10:53:42 -0400

Weird things about git, #2: no bug tracking system

The git developers don't track bugs. If you find a bug, you can write about it on the mailing list. You might get flamed. And then probably someone will ask you to fix it yourself and send in a patch.

This is unlike almost all other open source projects. Virtually every project out there has a basic bug tracking system attached, courtesy of SourceForge or Google Code or whatever. It's a key part of the development process, right?

Well, not exactly.

I'm also following the development mailing list for Mono. It has much less traffic than git, which is weird, because at least by lines of code (and probably number of users), Mono is a way bigger project. One of the reasons for the quiet is their bug tracking system. It's a conversation killer. Like this:

Someone reports a bug on the mailing list. "Please file it in the bug tracker." "I can't figure out how to create a bugzilla account." "Do it like this." "Okay, I created the account, but I can't figure out how to file a bug." "Do it like this." "Okay, I filed the bug. Now what?" "Now hopefully someone will fix it someday! Bye!"

So people aren't supposed to discuss bugs on the mailing list, which begs the question: what are they supposed to discuss on the mailing list? Nobody really knows. So the conversation dies. And not everybody sees every bug that comes in. Which is probably good, because there are thousands of bugs filed, and as with most open source projects (and commercial ones, for that matter), most of the bugs never get fixed, because volunteers just aren't actually very interested in fixing every last one of your problems. (This isn't to say the Mono developers don't fix tons of bugs. Mono is awesome. But it seems to be a cardinal rule of bug tracking that the bug database only grows, it never gets smaller.)

Let's be honest: bug tracking sucks the joy out of software development. git developers just work on whatever scratches their itch at that particular moment. If the same question comes up too often on the mailing list, they update the documentation. If people report a bug, they quiet those people by fixing the bug (or convincing those people to fix the bug). If they feel like implementing a new feature today, then they just do. And with multiple examples of this happening every single day, it's easy and fun to subscribe, get a feel for what's happening, and join in.

When you work to a bug tracking system, you're just doing what you're told. Some poor bug tracking janitor spends hours each day shoveling bugs around inside the system and assigning them to people and prioritizing which ones go into which release. Then the developers line up and take their bugs, and work on them in sequence so that the release can go out at the pre-ordained time.

git developers just make a code freeze branch sometimes and let it simmer for a while, then release it. Junio C Hamano is the equivalent of the bug shoveler for git, except he doesn't shovel bugs: he shovels fixes. I expect that's a much more gratifying job.

Update (2008/06/30): YCombinator News has some commentary on this article.

2008-06-25: How to tell the gcc maintainers are all (still) lunatics

Thu, 26 Jun 2008 21:57:43 -0400

How to tell the gcc maintainers are all (still) lunatics

Just got bitten by this insanity today.

There's not being backwards compatible, and then there's... that. What was the point of upgrading, again?

2008-06-24: Weird things about git, #1: premature optimization

Wed, 25 Jun 2008 00:06:46 -0400

Weird things about git, #1: premature optimization

I've been hanging out on the git mailing list for the last little while, and a few things have been striking me as weird. There are hundreds of people on that list, and lots of them are active contributors. Why? It's just a stupid version control system after all.

But there must be some reason. I've thought of a few. Here's one for today:

The git developers are completely obsessed with performance. Apparently nobody there has ever heard that "premature optimization is the root of all evil." (Incidentally, the ACM has a recent article explaining why that statement is a fallacy.)

It's enlightening just to watch the git developers optimize every possible part of their system: syscall overhead, memory allocation, path trees, file size, network turnarounds, etc. And even while they optimize the heck out of everything, they still complain about how inefficient it all still is. Which it is, of course, if you follow the discussions.

Also interesting is that large parts of git are written in sh and perl and it's still fast, because while they obsess about performance, they also know which parts actually matter to performance. It helps to be a Linux kernel developer sometimes, I guess.

There's no doubt that git's optimization is premature: it started out way faster than svn, and it gets even faster with each release. Is that really necessary? Of course not. Everyone was doing just fine with things like svn. But life is just so much better when programs go unnecessarily fast. It's a very strange sensation; pain you didn't realize you were having just goes away. I've been doing a lot of Windows development lately, and let me tell you, Windows developers have a whole lot of pain that they don't realize they have, because it's not overt. Windows development is mostly fine, really. But what kills it is all the little random delays, the crashes, the giant memory-leaky .Net-based IDEs that make you point-and-click fifty things sequentially once a week because you don't have a way to write a script to do it for you. Just give me a text editor and 'make' any day, thanks.

What was my point again? Oh right, unnecessary optimization. While I've been writing this article, I've been waiting for svn to finish committing 20000 tiny files to a fresh svn repository. It's still not done. (Watching a strace of this is hilarious. Among other things, it's creating an XML file for every single one.)

...

And for comparison, git just did the same thing in 23 seconds. (About half that time was spent running my shell script to create 20000 files.)

Who cares? How often do you create a repo with 20000 tiny files in it anyway? Almost never, of course. It's not an important optimization goal.

...

svn finally finished. See? That wasn't so bad.

2008-06-05: I Am Of Literary Significance

Mon, 02 Jun 2008 22:55:05 -0400

I Am Of Literary Significance

Oh yes. Fear me.

Stupid internet.

(The rather dull ancient high school essay that started it all. But this is better. Or maybe this.)

2008-06-03: So... what do you do?

Mon, 02 Jun 2008 14:34:59 -0400

So... what do *you* do?

I got married a couple of weeks ago. We and our guests planted some trees:

Me getting married. Via the London Free Press.
Also note Peter, Luke, Ed, and Anna.

But wait, I'm getting ahead of myself. The story goes something like this:

Erin and I got engaged shortly after she went tree planting last year. (I had been too lazy to get up early.) We thought it would be fun to have a tree planting at our wedding, so we set about trying to find a wedding site that would let us plant trees. Alas, there were none. But one of Erin's friends referred her to a berry farm out in the wilderness somewhere, which we visited and decided it would make a fitting wedding location. Sadly, they already had plenty of trees. We proceeded to the local coffee shop (there's only one) to discuss our strategy. I asked someone there, "So, let's say you happened to have a bunch of trees that needed planting. What would you do?" She looked at me seriously, thought about it for a second, and said, "How many?"

A couple days later, she set up a meeting for us with a local lady, Barbara, who happened to be planning to reforest some of her family's historical farmland with trees that originally grew in the area. (In Southern Ontario, many of the original forests were clear cut and converted to farmland, so a lot of the original species are getting pretty rare.)

We agreed that her land would be a great place for our tree planting, and she said she'd name it "Erin's Dell." She offered to pay for the trees. I said no, no, that's okay, we couldn't ask you to do that. Instead she formed a charitable foundation, got donations, and had the Ontario government match them so that Erin and I ended up paying nothing. She also picked the trees, hired a landscaper, got a local lady to bake fresh bread, and had everything ready for us the day we arrived.

Oh, also, it turns out that before she retired she specialized in public relations and marketing, so she could hardly resist issuing a press release. That's probably why the London Free Press showed up at our wedding (where our guests, by the way, were doing almost all the actual digging/planting work) and took the photo featured above. A couple of smaller local papers picked up the story as well.

So what's the point of this story?

Well, when you meet people, one of the first questions they always ask is, "What do you do?" Erin finds this question annoying, because it isn't the right question.⁽¹⁾

Ask yourself instead: if it weren't for her, how much of the above would have happened?

Footnote

⁽¹⁾ It's not actually the right question for me either. When I answer "I'm a Computer Engineer!" most people's eyes instantly glaze over and the conversation ends. That's not supposed to be the purpose of small talk, so I usually dodge the question too nowadays :)

Epilogue

And then our honeymoon in Salt Spring Island, BC, was almost entirely organized by Juli, my bus driver from high school. I am not making this up.

2008-06-02: A note on drowning in work

Mon, 02 Jun 2008 14:34:55 -0400

A note on drowning in work

Glub.

2008-06-01: Being Right

Sat, 31 May 2008 19:30:56 -0400

Being Right

A few months ago I learned what is, so far, the most fascinating thing I've learned about human nature: the difference between being right, and being able to look someone in the eyes.

If they practice, people can learn to look you in the eyes even when they're lying to you. But it's kind of like a fake smile; there are involuntary muscles up there. If you know what you're looking for, you can still tell.

But what does it mean if they're looking you in the eyes and they mean it? It means that, at least in that moment, they're doing what they really believe is right. That's the definition of integrity.

That part is easy. That's not the surprising thing.

The surprising thing, to me, was that someone can have integrity and still be completely evil. It's kind of obvious in retrospect; the super-villain in an action movie can always look the hero in the eye, and he always does, just to prove it. He has integrity. Evil with integrity is more respectable, somehow, than plain evil.

All it takes to have integrity is to do what you think is right, no matter how stupid that may be.

Beware of people with integrity.

2008-05-29: Ctrl-scrollwheel

Thu, 29 May 2008 10:45:18 -0400

Ctrl-scrollwheel

Today I accidentally discovered the craziness of Ctrl-scrollwheel on a Mac. It also works with two-finger scrolling on a touchpad (although you need an extra finger for Ctrl), so MacBook users need not feel left out. Try it! It's ridiculous.

I'm totally going to use this next time I do a presentation. Which I think is in 15 minutes.

2008-05-25: On the causes of poverty

Thu, 29 May 2008 10:43:21 -0400

On the causes of poverty

To seek "causes" of poverty in this way is to enter an intellectual dead end because poverty has no causes. Only prosperity has causes.

2008-05-20: On projects that don't die

Tue, 20 May 2008 11:25:25 -0400

On projects that don't die

I find it interesting to track which of the things I create that continue to exist and flourish when I get bored with them, and which just drop dead when I stop working on them, and everything in between.

For example, I've posted several times before about random people still do with my obsolete-before-I-even-wrote-it Linux Arcnet driver.

Another one that I find interesting is the Debian Popularity Contest, which I created in... hmm, late 1998, almost ten years ago now. I intended it as a simple way to demonstrate that Debian really didn't need to distribute a 5-CD set; the packages most people wanted would happily fit onto one CD. Also, I was hoping I could forge enough results to get joe installed by default.

The first goal worked out great; Debian and Ubuntu now use popcon to fill their CDs, and your system will be just fine even if you only download a single CD image. The second goal unfortunately still hasn't come to pass... perhaps someday.

But more interesting is where these projects went beyond my original intention. The reason arcnet is still popular is that it's used in quite a few factories, because of its extremely good noise resistance and because it's good enough that nobody wants to replace it.

And popcon? Well, it now seems to have become the de-facto method of resolving things like vi vs. emacs wars or how soon git will be more popular than cvs, including a graph all the way back to 2004.

Amazing.

2008-05-15: jwz's collected bicycle wisdom

Wed, 14 May 2008 16:57:05 -0400

jwz's collected bicycle wisdom

jwz has a remarkably insightful article about bike ownership. Notably, item #6 makes me feel better about getting my own bike stolen last summer:

Get a u-lock. Lock through the frame and the back wheel. Your bike will be stolen, so don't get too attached to it. This also means, don't waste your money on junk like baskets and lights. Just get a backpack. It doesn't matter how crappy your bike looks: any bike is worth stealing for $2 worth of crack. Your bike is temporary. Accept this and move on.

2008-05-13: A note on market research

Mon, 12 May 2008 21:06:57 -0400

A note on market research

I left one company after several years of faithful service. The CEO himself was elected to get me a going-away present on behalf of everyone. He got me... cuff links. (I include the wikipedia link to help out those of you who know as much about cuff links as I do.)

By comparison, where I work now, an office manager was elected to buy me a wedding present. She didn't know what I would want, so she went through my emergency contact information (the privileges of being an office manager!), called my parents, and asked them what I would want. My parents didn't know, so they sneakily called me under false pretenses, got the information out of me, and reported back. What I got, among other things, was a shower squeegee. (I include the patent office link so you can truly understand what an amazing gift that is.)

So what's the moral of this story?

Well, sometimes your office manager knows a lot more about pleasing customers than your CEO does.

2008-05-11: Engineers without Borders

Thu, 08 May 2008 14:57:02 -0400

Engineers without Borders

I went to an Engineers Without Borders conference in Montreal a while ago, and I was very impressed; these people seem to have their goals straight. And little did I know that it was founded in 2000 by University of Waterloo students who were in school at the same time I was. Considering it now has 25000+ members, that's incredibly fast growth.

Anyway, as a sample of the sort of things they do, here's a report of a field study of how a proposed drought-resistant crop worked out in a heavy rainfall season. Summary: not so well. But that's not the point; the point is that they actually went to check, and then reported the results honestly, just like you'd expect engineers to do. I really respect that.

2008-05-09: Great moments in probability

Wed, 07 May 2008 14:22:52 -0400

Great moments in probability

Years ago (around 1999) when dcoombs and I were debugging the first versions of our "weaver" Linux-based server appliances from our apartment in Waterloo, we used to test on the cheapest hardware we could obtain for cheap.

One of these boxes absolutely refused to boot weaver, but the symptoms were strange. We had three ways of booting: boot from a CD, install an image on the hard drive and boot that, or load Etherboot from a floppy and use that to network-boot the kernel over tftp.

The symptoms were as follows:

Booting from CD worked fine.
Installing from CD to the hard drive and booting that worked fine.
Booting a weaver image from the hard drive (with a kernel downloaded via ftp) always gave kernel decompression error.
The etherboot TFTP process would always abort with a timeout after a few packets. (Etherboot of the era would do that occasionally even on a good day, but here it happened every time.)

The obvious conclusion here was that our weaver kernel image was broken, because you could boot the Debian kernel from either CD or hard disk without a problem. Right?

Well... as it turned out, no. The actual problem was a horribly broken network card that would randomly corrupt bits. About 9 out of 10 packets would be corrupted. You'd think that would be obvious, right?

Well, no. In fact, TCP/IP is specially designed to deal with the occasional corrupted packet. TCP and UDP have a 16-bit checksum on every packet, and if it doesn't match, the receiver simply throws the packet away; the sender is supposed to resend (and it does!).

I had noticed the FTP transfers were surprisingly slow, but not *that* slow, and back in those days, you could never quite remember if your network card was 10 MBit or 100 MBit. This happened to be a 100 MBit card, but 9/10 packets were getting thrown away, so we got around 10 MBit performance from ftp.

But here's what killed us: a 16-bit checksum can only detect 65535 out of 65536 possible errors. A 9/10 error rate means you're sending 10x as much data as you think you are, so a 12MB kernel+rootdisk package is actually about 120MB of packets; that is, about 80000 packets at 1500 bytes each. Thus, virtually every transfer was destined to have a tiny number of incorrect bytes! Ha!

Of course TFTP is extra dumb and doesn't deal well at all with packet loss, so it would just time out. But I remain very impressed at how well TCP managed to paper over a 90% broken network. That's the power of the Internet for you, right there.

(Thanks to jwz for having a hopefully-unrelated problem that reminded me of this.)

2008-05-07: Oil usage trends

Wed, 07 May 2008 13:11:58 -0400

Oil usage trends

Every now and then, you see a trendline that actually tells you something cool.

Check out the average number of vehicle-miles travelled in the U.S. since 1983... and how it's finally stopped increasing. Via Paul Kedrosky.

2008-05-03: Architecture Astronauthood

Sat, 03 May 2008 14:04:49 -0400

Architecture Astronauthood

Joel Spolsky has a recent article called Architecture Astronauts take over. The only comments I can seem to find on his article are negative ones, which is too bad.

Now, I'm a bit of an architecture astronaut myself sometimes. Even so, I believe Joel's totally right.

...but he's missed (maybe deliberately for effect) a couple of important points.

One is that "architecture problems" really can be important to customers in the end. Not directly, but because they can make it easier to solve a certain class of problems that customers do have. Synchronization is, I believe, one of those problem classes. I've spent a lot of time thinking about and working on (real customer) problems that come down to the synchronization problem, and wishing there were a magic "synchronize this" button I could just click and have things always work out. There still isn't, and maybe there never will be, but I still hope there will. (rsync is the closest thing we have, and it's a huge architectural step forward - but there's more I need to synchronize than just files, and librsync seems to be largely ignored for some reason.)

The second point he misses is that giant non-user-focussed architectural projects are not always a waste of time. "The only difference between genius and insanity is success."⁽¹⁾ Look at MacOS X, for example; Apple threw out pretty much everything from the original MacOS, and spent years rewriting the entire thing from scratch. It was a total loser for years and years of development, and the pain continued even for years after its initial release, because it was slow and buggy and incompatible. And yet now it's powering some of the world's most popular (or at least coveted) laptops and cell phones.

It was a huge risk. For every MacOS X, there are dozens of examples of failed, mind-bogglingly expensive and time-consuming, attempts to build a new desktop operating system. Even Apple hasn't won more than a tiny fraction of the market. Thus, Joel is exactly right; anybody who would try this basically has no concept of risk vs. return.

And yet, thank goodness for those people.⁽²⁾

Footnotes

⁽¹⁾ Can anybody find a good source for that quote?

⁽²⁾ Not that Microsoft's latest attempt seems any more likely to succeed than the previous ones. I said rsync was a huge step forward, but it's really just a tiny little tool based on a brilliant concept. The "synchronization problem" needs more of those, not another giant multibillion dollar "synchronization platform." An operating system needs to support everything I need to do, while synchronization is just one of the difficult things I need to do sometimes.

2008-05-01: Logic can only help you achieve your goals, not set them

Thu, 01 May 2008 17:11:22 -0400

Logic can only help you achieve your goals, not set them

Today, a quickie on the respective values of logic vs. emotion (intuition).

At one point, a person I know turned down an offer to work for my company in favour of an offer at Google. He had worked both with me and for Google at different points in the past, so he knew a lot about both choices. Our job offer was competitive with Google's in terms of salary and benefits, and as a bonus, he wouldn't have to relocate to a foreign country as with Google. Still, after a lot of deliberation, he went with Google, and I encouraged him to. Why?

Because he felt like it.

You can't argue with logic like that. You shouldn't try. The reason is simple: because doing what you feel like makes you happy. Doing what you don't feel like doesn't make you happy. And that's just the end of it. No amount of thinking it through will change those basic facts.

Unfortunately, a lot of people don't know what they feel like doing; they don't know what they want out of life, and they certainly don't know why. So they try to figure it out. That's where consumerism, among other things, comes from (weird article about escaping from consumerism). "I don't know which job will make me happier, so I'll take the one with the higher salary." "Maybe if I had a bigger house or a nicer car or whiter teeth, then I could be happy." If those things will actually make you happier (and if you're honest with yourself, you already know if they will), then that's it; you win. But if you don't know anything that will make you happier, and you're trying to choose on an "objective" basis, then you'll lose for sure.

This problem - overthinking your desires - is surprisingly widespread, not just among programmers, but for almost everyone. Programmers happen to be especially stubborn at rationalizing, so they sound more convincing than most, but it's all the same game.

So what is logic good for, then? It's good for achieving your desires. If you're honestly having trouble deciding which of two job offers makes you feel better, maybe you honestly don't like either of them. Maybe, regardless of how objectively great they both sound, neither is subjectively good at all.

When that happens - and this is a last resort for only that case - then you can use logic. First, imagine what you want your life to be like. It can be anything you want. Seriously. Imagine it in excruciating detail. Then figure out, in just as much excruciating detail, all the steps you'll have to take to get there. If you do this exercise properly, all the way back to the present, then you'll know what to do next. You'll have figured it out using logic, but that's okay, because your logic was guided by emotion, not the other way around.

For more on logic vs. intuition vs. several other human attributes you can combine for use in decision making, you might want to read the book On Equilibrium by John Ralston Saul. It has lots of big words, but it's enlightening.

Also beware that if you believe what I've said above, you probably now believe in fluffy emotional stuff called the Law of Attraction. Hope you don't mind. If you're like me, you'll find the traditional explanations of it (eg. follow the link) unbelievable and silly, but what I've said here is just a sanitized translation, honest.

2008-04-29: The "culture of quitting"

Tue, 29 Apr 2008 18:29:21 -0400

The "culture of quitting"

Here's an interesting article called Up or Out that talks about how people whose careers have stopped progressing should just leave with no hard feelings... and how this is already the case in legal and accounting firms (ie. if you don't "make partner"), but for some reason doesn't seem to happen in IT organizations.

He even refers to employees that leave happily in this was as "alumni".

This is an interesting point of view, since most people are more obsessed with increasing the quality of their workforce through hiring rather than ~~firing~~ attrition. For comparison, see "hiring above the average" at Google.

2008-04-25: Shameless request

Fri, 25 Apr 2008 23:12:17 -0400

Shameless request

Hello fellow alumnits,

I know that quite a lot of you work at Google now, so I want to ask for a favour: can you weasel me a preview license to Google Appengine? It sounds really fun to play with, and I even have a project in mind.

Way back at NITI, we used to have a system for catching wvcrash output and querying it in various different ways. I wrote a system similar to wvcrash for AmSchedule Express and other Versabanq stuff, but I haven't yet written a handy analysis and tracking system for the crashes. I think this would be a fun project to run publicly with someone else's resources and let all sorts of software send their crash dumps to it. And since I need it anyway, I'd do most of the work. See? Everybody wins!

(Except Google, who's giving all this away for free :))

Update: Well, that was quick. Thanks, mag!

2008-04-20: Linus on Caching

Sun, 20 Apr 2008 18:53:32 -0400

Linus on Caching

So we have other caches and hashes we use, like the index itself, or the name lookup hash into the index, or the delta cache. Maintaining those caches takes some effort too, but those caches aren't there because we're doing something stupid, they are there because they allow us to do something smart.

-- Linus Torvalds on caching repeated operations

2008-04-08: The triumphant return of ion1

Tue, 08 Apr 2008 19:23:58 -0400

The triumphant return of ion1

I really like the ion window manager. However, recent versions (ie. post version 1.0) haven't added anything I wanted, while adding significant complexity and making it so that certain things I liked in the original are nearly impossible to attain.

Specifically, I have "split this frame vertically" defined as ALT-S-V, and "delete the current frame" defined as ALT-S-X. If I do these commands in quick succesion in ion1, effectively nothing happens; my focus is where it started, and the order of my windows is the same as it was when it started. Unfortunately, I can't find a way to do the same thing in later versions of ion; "delete the current frame" always ends up moving my focus around and changing the order of windows. Probably there's a way to program what I want in "lua", the oddball scripting language it uses, but I haven't bothered to learn lua and I don't see the point of doing extra work.

Thus, I hereby announce the stillborn ion1 revival project. I say "stillborn" because I have taken the source code to ion1, built it as a .deb package for Ubuntu Hardy, and I now plan to stop working on it since it already does what I want, and that was the whole point.

Enjoy.

(If you do happen to want to contribute patches, though, please let me know. ion1 *does* have a few bugs that I'd love to see get fixed.)

2008-04-07: The Google Situation

Mon, 07 Apr 2008 16:56:24 -0400

The Google Situation

Fake Steve Jobs

2008-03-28: Gittorrent

Mon, 24 Mar 2008 00:44:24 -0400

Gittorrent

Can your version control system's design do this? Gittorrent.

2008-03-27: A tale of five merges, part 5: darcs and patch theory

Thu, 20 Mar 2008 13:26:26 -0400

A tale of five merges, part 5: darcs and patch theory

This is not really an article about darcs. Actually, I've never used darcs. It's not even really about patch theory, because I'm too lazy to read through the actual article on patch theory in enough detail.

What this really is is a nice way to finish off my series of articles on version control and merging concepts, by discussing some features that a program like darcs might or might not have, because of some theory that patch theory might or might not provide. In other words, here are some good ideas of unknown origin that I hope git will steal. But I'll call our imaginary system that might have these features darcs, because it ought to have a name. My apologies to the actual darcs maintainers.

So far, we've talked about branch merging in CVS, SVN, and git, both with git-merge and git-rebase. We've been addressing each one in terms of four main questions: (A) how automatic is it, (B) what happens to the change history, (C) what happens when you merge back and forth multiple times between two branches, and (D) what happens when you "cherry pick" individual changes, usually bugfixes, from one branch to another.

What we find is that with all the systems discussed so far, most of these answers end up being some kind of compromise. None of the options presented so far has really been entirely what we want. So what would the ideal solution be like?

A note on versioned storage

Before we can talk about the world of perfect version merging, we have to talk about how those versions are stored in the first place. There are two basic philosophies about that: (a) store the files, or (b) store the patches.

Speaking very roughly and inaccurately, SVN and CVS store one version and then a bunch of patches against that version. git stores just stores the files.⁽¹⁾ Systems like darcs, on the other hand, prefer to talk about groups of patches, rather than the files themselves.

The results of these philosophical decisions are very interesting. For example, if you branch and merge a lot in SVN, your repository gets a lot bigger, because you store the patches from A to B, and from B to C, and from A to A', and from B to B', and from C to C', and so on. All those patches will be slightly different, even if the resulting files are mostly the same, so it's impossible to eliminate any redundancy (except trivially with something like gzip). git, on the other hand, stores just the files, and when files are the same (which is very frequently), it doesn't store them over again. That means that in git, branching is basically always free.

darcs, on the third hand, mostly stores each version as a big pile of patches, and it stores them in a particularly efficient way: if it is possible to use the patch from A to A' to convert from B to B', even if the patch application is a bit "fuzzy", then we just add that patch to the pile for B'. Similarly if it will work to convert C to C'. So in darcs, branches are about as cheap as in git.

Now, regardless of which storage method you choose, you can obviously convert quite easily from files to patches and back. Version control systems do this all the time; if you do "git show" or "git diff", it's converting from files to patches; if you do "svn checkout", it's converting from patches to files. The only difference is that certain operations go faster than others depending on your storage format.

Why git is so fast

For example, "git clone" (the equivalent of "svn checkout") goes much faster than checking out of any of the other systems, because no conversion is necessary. On the other hand, "git diff" might be faster or slower than a diff on one of the other systems; if you're asking about a patch that darcs has already stored, darcs will probably be faster, because git has to generate a patch. But if you're asking about the difference between two arbitrary sets of files, darcs will first have to figure out what those files look like, then generate the patch, which is two steps to git's one, so git will be faster.

And this is where the fun starts. Because of git's storage system, some operations are just plain amazingly fast: such as creating a commit. The secret to git's speed is that making a checkin just involves copying the changed files verbatim into the repository; it doesn't think about deltas, or patches, or anything. And checking out any version, or creating a branch, involves simply copying a file directly out of the repository. Even finding the differences between any two versions simply requires reading the index to see which files have different SHA-1 hashes, and doing a diff between them. And the "git merge" operation does exactly that: it figures out the differences from BEFORE to AFTER, and applies them to TARGET, and checks in the result. All of these operations are about as fast as they get.

Where git gets slow is exactly where darcs gets fast. In darcs, every merge is like a "batch cherry picking" operation in git. When there are no patch conflicts - and darcs' "patch theory" will tell you in advance if there's a conflict - then darcs can cherry pick a batch of patches in the same time it takes for git to make an empty commit. For git, on the other hand, every single cherry you pick takes the same time as a git-merge: checkout, diff, patch, and checkin.

git-rebase is just a big batch of cherry picking, and it's slow in git. It would be a non-event in darcs.

git-svn uses git-rebase extensively, because it pulls patches out of svn and then rebases all your work on top of it. In darcs, the patches would just go straight into the repository, no questions asked, and the conflicts could be resolved later.

But much more importantly than the speed, cherry picking patches (and thus "rebasing" branches) in darcs retains the history of those patches; after all, they're just the same patches in another place, and all there is is patches. Where history in SVN is a line, and history in git is a helix, history in darcs... well, we don't like to talk about "history" in darcs. We talk about... Patch Commutation and Inverses.⁽²⁾ Erm, okay, maybe another time.

The major difference in the way darcs thinks about these things is in what happens during a merge. In git, we noted that merging two branches in either "direction" always results in the same merged result, ie. set of merged files. But in darcs, you don't talk about files, you talk about patches, and the merged result is not the same: the two merges have the same set of patches, but the patches are in a different order.

The order of the patches is the missing information that SVN's simple-minded merge history emphasizes, and git's merge history hides. Moreover, git-rebase is all about changing the order of patches, and since git doesn't really know anything about patches, that's exactly why it makes git-merge so angry.

How does darcs do back-and-forth merges between two branches? Well, that's easy: you simply look at the list of patches in the source branch, and the list of patches in the destination branch, and you add the patches to the destination that weren't already there.

How about reordering and recombining patches, like in git's interactive rebase? Well, if "patch theory" says there aren't any conflicts, then feel free to reorder them; darcs already knows it'll work, so it doesn't actually need to calculate anything. It finishes instantly, and doesn't mess up future merges to or from other branches.⁽³⁾ That's way better than git-rebase.

A perfect world

At last, we come to the end of our series! Let's put it all together. What would an ideal system look like?

My suggestion would be to base the system on git - because it's fast for everyday operations like checkout, checkin, diff, and merge - but add some elements of patch theory. Most importantly, it's not okay for a git-cherry-pick or git-rebase operation to forget the old commits that the new commits were based on. They need to store a "patch parent" field that basically says, "this commit is what happens when you take the difference from BEFORE my patch parent to AFTER my patch parent, and apply it to my PARENT." Then git-merge will have some extra work to do sometimes to unwind the criss-crossing patches, but it's not the end of the world: after the next git-merge between a given two branches, we're entirely caught up and everything is easy again. It seems a small price to pay in order to have git-svn, git-rebase, and git-filter-branch all magically work properly with git-merge and git-pull.

So how does our imaginary new system stack up with respect to our four key questions?

Our newfangled merging system: (A) allows you to rearrange patches as easily as git-rebase, and yet still merge as easily as git-merge; (B) allows you to shuffle and clean up change history at will, while still tracing (via patch-parent) into the detailed original patches if you want; (C) easily handles back-and-forth merges as well as git-merge or darcs, *and* even if you've been using git-rebase; (D) easily supports cherry picking without disrupting future back-and-forth merges.

Yeah, yeah, I know what you're going to say. I should go write it myself. Well, maybe someday I will. But don't hold your breath. :)

Footnotes

⁽¹⁾ This is really true, honest. The existence of "git repack", which rewrites groups of files using xdelta encodings, is really an optimization. It's not how git actually thinks about things.

⁽²⁾ That's probably why most people don't use darcs.

⁽³⁾ There are actually huge theoretical problems with doing this, so you wouldn't actually implement it that way. Imagine you had a patch to add a function declaration to foo.h, and another one to add the function body in foo.c, and another one to call the function and include the header from blue.c. If you check in the changes in that order, you'll have a compiling program at each step. And these three patches are "independent" according to patch theory, since they all change different files. But if you reorder them and change blue.c before foo.h and foo.c, your program won't actually work. So arbitrarily reordering patches produces software versions that never actually existed and were never tested and thus might not work, which is of limited use. That's why "patch theory" as I've described is mostly "theory" and not "practice", even in darcs.

2008-03-25: A tale of five merges, part 4: 'git rebase'

Thu, 20 Mar 2008 13:26:24 -0400

A tale of five merges, part 4: 'git rebase'

So far, we've talked about branch merging concepts in CVS, SVN, and git. Of the three, git's is by far the most convenient and magical... when it does what you want. When it doesn't do what you want, you find yourself in a bit of trouble.

Rebasing git branches

Last time, we brought up one example of a major problem with the "git merge" command: if you create BRANCH1 from MASTER, and then BRANCH2 from BRANCH1, and you merge BRANCH2 into MASTER, then you end up merging BRANCH1 into MASTER as well by accident. That's because "git merge" finds the first common ancestor of BRANCH2 and MASTER, which is MASTER, and merges all the changes from there to BRANCH2. In between those two points lies all of BRANCH1.

This is exactly the problem that "git rebase" was designed to solve. Here's what you do:

git checkout BRANCH2
git rebase --onto MASTER BRANCH1

What the above command does is re-apply all the changes from BRANCH1 to BRANCH2 onto MASTER. Sound familiar? Yes, that's right, it's exactly what "cvs up -j BRANCH1 -j BRANCH2" would have done. And with "git rebase", you specify revisions just as manually as you would with CVS or SVN.⁽¹⁾

Here's the problem: the rebase command generates entirely new patches onto an entirely new branch. The old BRANCH2 is gone⁽²⁾, and the new BRANCH2 has a bunch of commits on it that look similar to a person, but as far as git knows, have nothing in common with the old ones. If you've ever shared your old BRANCH2 with anyone, they will no longer be able to merge to and from your new BRANCH2. If you use git-rebase, the only sane thing you can do is throw away all the pre-rebase versions of BRANCH2... and that's tricky, since people all over the world might have a copy. They might even have merged it with their own branches. Oops.

This combination of inconveniences, by the way, is the absolute most frustrating problem with using git-svn. Every time a new revision comes in from SVN, you need to git-rebase your patches on top of it, screwing up all git branches in the entire repository.

The good news is that you can continue to work on your new BRANCH2, and when you merge it (with "git merge") back and forth with MASTER, at least you won't have a problem.

Interactive Rebase

Another concern that people run into with git's merging style is more about their sense of aesthetics. git makes it really easy to create local branches and check in frequently - it even encourages this, since it has such easy merging and branching and such fast checkins.

The problem is if you take advantage of this, you end up checking in code every ten minutes, and sometimes it doesn't work. When you later want to prepare a set of patches for submission to an upstream maintainer, you don't want to send in 57 patches, none of which work independently. You probably want to send maybe three patches, each of which adds a separate feature.

For this, git introduces "git rebase -i", known as "interactive rebase." I won't go into detail, since the git-rebase man page is detailed enough. But the idea is that you can take your original patchset and shuffle it around and split and join patches however you want, making it look like that's what actually happened as you developed your project. So if you added feature #1, then feature #2, then realized there was a bug in feature #1 and fixed it, then fixed another bug in feature #2, you can join patches #1 and 3, and then #2 and 4, so it looks to the world like you just did everything perfectly the first time. As a bonus, when people are doing code reviews of your patches, they won't have to review code that you already knew was broken. That ought to save some time.

Interactive rebase is an innovative feature that certainly isn't available in something like CVS or SVN, and it can be addictive. Of course, before you have it, it's hard to imagine that you'd need it. Personally, I find myself reshuffling my patches a bit unnecessarily just for the aesthetics of a clean repository where each patch does what it says. This is both fun and time consuming, and it's debatable whether it's worth it or not.

To some degree, the reason interactive rebase is popular is that "git merge" history is too detailed. In svn, you'd make your mess on a branch, get things all cleaned up, and then do a one-shot merge from your branch into the trunk. The merge would just say "added features X and Y." Nobody was forced to look at the inevitable long string of screw-ups that actually led to features X and Y getting added. Interactive rebase exists primarily to get back the feeling of cleanliness that SVN had and git took away.

Git rebase summary

For each merging system, we've been asking: (A) how automatic is it, (B) what happens to the change history, (C) what happens when you merge back and forth multiple times between two branches, and (D) what happens when you "cherry pick" individual changes, usually bugfixes, from one branch to another.

"git rebase" is unusual for a merging system, but it's basically a paraphrased version of old-style CVS and SVN merge commands. Thus, the answers it gives to these questions are roughly the same: (A) it's semi-automatic, as long as you provide the right revision numbers; (B) it's mostly used to make the change history look cleaner, at the expense of losing the detailed history git-merge could provide; (C) it actively complicates merging back and forth between branches, and produces more conflicts than CVS or SVN would have; and (D) it's highly compatible with patch cherry picking, since that's pretty much all it does.

In short, a combination of "git merge" and "git rebase" either gives you the best of both worlds, or the worst of both worlds, depending on your point of view. But having to choose between them, and trying to figure out what crazy thing happened when you chose the wrong one, is one of the scariest and most confusing parts of git as far as new users are concerned. In fact, for me, it's much harder to sort out the combination of "git merge" and "git rebase" than it is to figure out two-way merging in CVS... and that's saying a lot.

Next time: darcs and patch theory.

Footnotes

⁽¹⁾ One cool thing about "git rebase", however, is that it keeps your original patches as separate checkins. With CVS or SVN, you would get one big checkin with a merge of all your patches; with "git rebase", it keeps your patch series intact. There is no technical reason that CVS or SVN couldn't do the same trick, though.

⁽²⁾ Usually you don't want to actually delete the old BRANCH2, just in case you screw up. Perhaps you'll call the new one BRANCH2a. But you'll still never be able to merge between BRANCH2 and BRANCH2a, even though they contain the same set of changes.

2008-03-23: A tale of five merges, part 3: 'git merge'

Thu, 20 Mar 2008 13:26:22 -0400

A tale of five merges, part 3: 'git merge'

In the last two articles from this series, I talked about merging branches in CVS and SVN, and how they're largely the same.

Today is the first of two articles about merging in git. git is most definitely not the same as CVS and SVN.

Remember, the questions we're investigating for each merging system are: (A) how automatic is it, (B) what happens to the change history, (C) what happens when you merge back and forth multiple times between two branches, and (D) what happens when you "cherry pick" individual changes, usually bugfixes, from one branch to another.

Merging with 'git merge'

Compared to working with CVS or SVN, merging branches in git feels like magic. First, you check out the branch that you want to merge into, say MASTER. Then you type "git merge BRANCH", where BRANCH is the name of the branch you want to merge from. And that's all.

...When it works.

Aha. You see, "git merge" is really just doing exactly what you were doing in CVS or SVN by hand. That's why I went into CVS merging in such detail before.

Here's what really happens: given the name BRANCH and the implicit name MASTER (since that's the one you have checked out), git traces back in the history of both until it finds the branchpoint, that is, the last commit before the two diverged in the first place.⁽¹⁾ If we were using CVS, we'd call that point BEFORE, and BRANCH would be called AFTER. git then simply takes all the changes from BEFORE to AFTER and adds them to MASTER.

Now here's where things get especially clever. Remember when we were discussing bidirectional merges in CVS, and I pointed out that merging all the changes from BEFORE to AFTER into HEAD was the same as merging all the changes from BEFORE to HEAD into AFTER? The resulting tree is the same in both cases, but of course, in CVS and SVN, the actual commit you make is different in each case. After all, in one case, you're committing to the HEAD branch, and in another, you're committing to the AFTER branch.

git is smarter than that. In git, there really is no difference between the two operations. You generate a single commit that has two parents: AFTER and HEAD (or BRANCH and MASTER). Now, it so happens that git updates the MASTER branch pointer to point at your new commit in one case, or the BRANCH branch pointer in the other case, but the commit itself is exactly the same in both cases.

And that simple fact is why repeated merges and bidirectional merges work. Next time you're merging MASTER and BRANCH, git will trace back, and the most recent checkin that is shared between the two branches will be one of the parents of that magical two-parent merge checkin. We use that one as BEFORE, and merge the changes from BEFORE to BRANCH into MASTER. Or else we merge the changes from BEFORE to MASTER into BRANCH; again, it's the same thing.

This concept of a two-parented commit is what makes git merging totally different from SVN, and it's the difference between SVN's "linearized history" (in which every commit has exactly one parent, even if we made it that way artificially) and git's non-linear history.⁽²⁾ It's also the reason git merging can be automatic, while svn's cannot.

Why it doesn't always work

The problem with git's automatic merging is, well, that it's automatic. When it's working for you, it's amazingly powerful. But people who are used to CVS or SVN merging sometimes feel like git is taking their power away.

For example, perhaps you don't want to merge all the changes from BRANCH; perhaps you only want the changes up until sometime last week. With 'svnmerge' this is easy enough to do, but with git it requires multiple steps, since git-merge won't accept arbitrary revisions on its command line, only branch names.⁽⁴⁾

Or say you create BRANCH1 from MASTER, and start to add feature #1. Then you create BRANCH2 from BRANCH1, and add (unrelated) feature #2. Now you decide that feature #2 is ready to merge back into MASTER, but feature #1 is not. No problem, right? You checkout MASTER and then type "git merge BRANCH2".

Oops! git has just merged all the history of BRANCH1 *and* BRANCH2 back into MASTER, which is not what you wanted at all! Unfortunately, because git's merging is fully automatic, there is no easy way to get the behaviour you wanted.⁽³⁾ With CVS, on the other hand, what you would do is obvious: "cvs up -j BRANCH1 -j BRANCH2", to merge all the changes from BRANCH1 to BRANCH2 into MASTER. With SVN, it would be similar, except that BRANCH1 and BRANCH2 involve revision numbers and URLs.

In git, the solution to this sort of problem is to use the "git rebase" command, which we'll discuss in more detail next time. But any use of git-rebase or git-filter-branch, both of which are very useful, fundamentally change the commit IDs on the modified branch. "git push" and "git pull" stop working as expected. Essentially, these commands generate entirely new commits for the part of the tree they change, so now there are two sets of commits in your repository that do the same thing in two different ways, with nothing tying them together. Worse, after a rebase, the point where your two branches diverged might be no longer known to git - they no longer have a single commit id in common - so it just gives up.⁽⁵⁾ And trying to merge the new MASTER back into the original BRANCH2 later will result in a ton of conflicts.

The really bad news is that git-svn, which is otherwise a great tool to help with migrating between git and svn, uses git-rebase pretty heavily. That confuses git's automated merging features altogether, and git's manual merging features are not as straightforward as SVN's, so things get confusing very fast.

Git merge and cherry picking

git includes another command called "git cherry-pick". Its job is to take the changes from one specific commit and apply them to the current branch.

Unfortunately, the existence of git cherry-pick is incompatible with git merge's view of the world. git's history may not be linear, but at least it's continuous: branches may swerve into and out of each other as they pass through time, but individual changes aren't supposed to simply jump from the middle of one tree into the middle of another.

Since there's no way to represent what really happened, a cherry-picked patch instead produces an entirely new commit with exactly one parent (the destination branch), so a future git-merge from that branch has no idea the patch already existed, and usually produces a conflict.

Cherry picking in SVN with 'svnmerge' definitely works better than this.

How the git changelog tracks merges

Last time around, I talked about how SVN's "linear" change history tracks merges, which is simple enough: every time you do a merge, it puts a single checkin into the destination branch which contains all the changes. And if you want to know the details of every patch from the source branch, well, you'll have to go look at the source branch yourself.

git definitely doesn't do that. Instead, if you ask for "git log" it will show you the complete set of changes in all branches leading up to the current commit. That is, all of the checkins to either of the merged branches show up in the log, with nothing in particular in the "git log" output identifying which patch came from where.⁽⁶⁾ This is technically correct, but is not actually what most users wanted to know.

Most people who ask for the history of the MASTER branch want to know the "simplified" form of its history: Added feature X; added feature Y; fixed feature X; added feature Z. But instead, they confusingly see all the individual 57 commits making up "added feature X", even though if they were ignoring your featureX branch all along, "added feature X" was really a one-time event for them.⁽⁷⁾

A related problem is that git doesn't have a way of naming branches globally. So if you merge from the "featureX" branch, you're not really merging from featureX; you're merging from 2bed2ad4845eceb8ee650f34e476a60f9fbecc7c. As far as the computer is concerned, that's the same thing, but when it comes to tracing back through your history, it's not actually what you wanted to know.

Worse still, git doesn't remember at branch time where you diverged; it only figures it out at merge time. That means there is no equivalent to SVN's "svn log --stop-on-copy", which shows only the changes you added in this branch.⁽⁸⁾

git-merge overall

Now let's go back to the questions we were trying to answer in the first place:

git-merge: (A) does merges fully automatically, when it works, but complicates manual merges; (B) retains the history of each change, but that often turns out to be more than you wanted; (C) easily supports back-and-forth merges with no more trouble than single merges; and (D) does not really support cherry picking without introducing future merge conflicts.

Phew!

Next time: how 'git rebase' tries to solve some of the things people don't like about git merge.

Footnotes

⁽¹⁾ This is actually greatly oversimplified. Since git history is nonlinear, just tracing back through it is a bit complicated. On top of that, you can play tricks with selecting different branchpoints for different files, and so on.

⁽²⁾ I guess if SVN's history is "linear", then git's history is "helix shaped," with endless splitting and remerging of histories.

⁽³⁾ git proponents would say that what you should have done is branched BRANCH2 from MASTER in the first place. But not everyone is lucky enough to get everything right the first time.

⁽⁴⁾ There seems to be no actual reason for this restriction. Perhaps it will go away eventually.

⁽⁵⁾ That's the source of the mysterious note that "You should understand the implications of using git rebase on a repository that you share" in the git-rebase man page. Of course, the man page doesn't really tell you what the implications are. It just says it "will cause problems."

⁽⁶⁾ The "gitk" and "gitweb" programs attempt to represent this graphically, which helps. The "git show-branch" program is also somewhat useful here. But all of these are much harder to understand than "svn log".

⁽⁷⁾ There is actually a "--squash" feature of git-merge that tries to resolve this problem. Essentially it creates an entirely new commit with only one parent - the destination branch - and doesn't tie in at all to the history of the other branch. In other words, it does exactly what "svn merge" would do, but that has exactly the opposite set of tradeoffs, as discussed last time, and makes bidirectional merges very ugly. Alternatively, "git log --first-parent" displays only the first parent of each merge, which is closer to what most users actually want.

⁽⁸⁾ If you manually keep track of which branch you came from, you can use "git log BEFORE..MYBRANCH" to see the list of changes between BEFORE and MYBRANCH. But if you're going to manually keep track of BEFORE, you might as well be using CVS. "git show-branch" also attempts to help here, but if you ever use "git rebase" its output will be hopelessly cluttered and confusing.

apenwarr - Business is Programming

2008-08-27: More Artemy Lebedev

2008-08-25: Art Lebedev on the proper formatting of punctuation

2008-08-23: CSS Still Sucks

2008-08-22: gitbuilder

2008-08-21: cron2rss

2008-08-19: Dog on Treadmill

2008-08-17: Being an entrepreneur

2008-08-08: ROE

2008-08-06: ACPI reprise

2008-08-05: Bug trackers don't work

2008-08-01: ext2resize sucks, but apparently so do I, and also a lot of other things too

2008-07-22: iPhone 3G

2008-07-17: It lives

2008-07-04: Risk vs. Calculated Risk

2008-07-02: Toronto Fringe Festival Suggestions

2008-06-30: A serious case of JustOneMoreBug

2008-06-28: Weird things about git, #2: no bug tracking system

2008-06-25: How to tell the gcc maintainers are all (still) lunatics

2008-06-24: Weird things about git, #1: premature optimization

2008-06-05: I Am Of Literary Significance

2008-06-03: So... what do *you* do?

2008-06-02: A note on drowning in work

2008-06-01: Being Right

2008-05-29: Ctrl-scrollwheel

2008-05-25: On the causes of poverty

2008-05-20: On projects that don't die

2008-05-15: jwz's collected bicycle wisdom

2008-05-13: A note on market research

2008-05-11: Engineers without Borders

2008-05-09: Great moments in probability

2008-05-07: Oil usage trends

2008-05-03: Architecture Astronauthood

2008-05-01: Logic can only help you achieve your goals, not set them

2008-04-29: The "culture of quitting"

2008-04-25: Shameless request

2008-04-20: Linus on Caching

2008-04-08: The triumphant return of ion1

2008-04-07: The Google Situation

2008-03-28: Gittorrent

2008-03-27: A tale of five merges, part 5: darcs and patch theory

2008-03-25: A tale of five merges, part 4: 'git rebase'

2008-03-23: A tale of five merges, part 3: 'git merge'

2008-06-03: So... what do you do?