The “Break it Down” Blog

Why is it that whenever you start a new programming project the initial design, in your mind, is always immaculately laid out, symmetrical and performant while the end result ends up being a shadow of that overall projection?

I don’t know what it is about the actual “development” phase of a project that ruins all designs, good ones and bad; detailed and vague. No matter how detailed your initial design is on paper once you get your hands dirty and start coding, start writing those stored procedures, start storing those records and start loading those POJOs something happens. It usually starts in the form of a quick hack just to see if what you are doing is right… but then that hack ends up staying put because you need to try just 1 more tiny little thing to make sure that works too. Then there may be a handful more of quick additions just to check something and next thing you know you have an almost working system… that is of course assuming you don’t want to back up and rewrite all those hacks. So you leave them and continue on.

There is of course the alternative route when you are working on something new and different that you haven’t experienced before. Your document is all theory: “we will store people in this table and they will have these 3 attributes that are FK references to these other tables”. Then you are told 6 weeks later that the person doesn’t so much need attributes as they need to belong to groups… well shit. That’s a little different.

I’m not sure yet what to blame this phenomenon on. Is it misunderstood user requirements? Is it lack of insight into the process by the original designer? Is it a limitation in software development itself?

I’ve been part of a lot of teams that had to work on conceptually straight forward projects. Lot of data warehousing and processing. These types of projects tend to suffer from performance bottlenecks, not designer bottlenecks. The frustrating part is that no matter what original design we came up with, even though all seemed fantastic, the end result was always a hacked mess compared to the idea we had. At the time we always blamed it on the pressure management was pushing down on us and chopping development schedules in half… yea that’s it, it’s THEM. Then I moved to my next job where our development schedules were padded quite generously with time… that helped, but the same phenomenon still occured. So we blamed that one on changing user requirements… you know, THOSE. Well now I’m in another position to start a new project where the user requirements are a bit clearer. I have my fingers crossed that no wires will get crossed and no mistakes made, but that would be naive to think so. There will be some hickup in this process, I’m not sure what or when, but there will be.

It’s too bad we can’t design software in theory only. Think of a site like Digg or Slashdot. Just content, nothing but categorized, commented on, aggregated content. In theory, no problem! You’d have a database, represented by a cylinder of some sort on paper. That database would be emcompassed with web services and other techniques of abstraction. You would have client code that could make using these services easier. You’d have a data tier, a web tier, a client tier and a series of lines and entities all connecting to eachother to show the interactions. God what a nice clean design! Now we just need to start writing the site; because it’s a piece of cake that’s not going to be a problem.

We all know where this is going. You fast forward a few months and you have a spaghetti-bowel full of good code combined with crap code combined with code that should get you prison time. You have a hack for creating the screenshots because the technique you tried crashes the virtual machine when too many people post at once. You end up dumping avatar icons into a folder on the server instead of the nicely designed database table you had because it was 8x slower and 15% higher load on the CPU to load the avatars from the DB under high load. You have to completely denormalize the vote and comment tables because joining that information each time a story is displayed causes a major performance bottleneck for you. The list goes on and on. You learn your lesson though and swear it will never happen again, but it will.

It will happen again and again because each project you tackle will never be the same. You learned rewriting Digg that storing images in a database and performing joins instead of straight selects 1000s of times a minute were too much overhead for high volume sites. That’s great, but what about the next site you do? Something a little bit different, like Weather.com. The problem isn’t insanely high traffic anymore, the problem is data aggregation and synchronization. Who cares about joins anymore, those machines are fast enough, but what about aggregating all that information? Not all the weather stations use the same API. Some services offer web service access, others are still stuck in the 80s and will upload a CSV file to a FTP server every 30mins with a snapshot of the weather. Challenges around ever corner and no end in site. As soon as you nail down how to do a weather site, your next project will be writing a site like YouTube and MySpace combined with feedback like Digg.

I suppose that’s the fundamental reason why software design is so hard to get right; it’s a moving target. Even when you think you have it figured out, a user requirement can change or some performance testing may show you results you had no idea would popup or the next project you start is nothing like the ones you have done so far. The good news is that after each project, you take a little information away with you to use on the next and the next and the next. Some day someone will ask you to rewrite a project you already did and you can rejoice in celebration, knowing that this time around it will be perfect… maybe. (Digg it)