denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote2011-03-29 05:34 pm
Entry tags:

Technical debt and the making of payments on it

I saw an entry posted the other day where someone said sie was disappointed with (among other things) DW's development pace slowing down: new features being released more slowly, things that we were working on delayed/postponed, etc. And there were totally some valid criticisms in there, don't get me wrong! (In fact, I'm not linking it because I don't want there to be an overwhelming impassioned defense of DW in hir comments.) But that's one criticism that made me realize I've been doing a poor job of explaining precisely what's been going on in DW development and why there's been a paucity of user-facing changes, which can look to an outsider like there's a massive slowdown going on in DW development.

The answer is at once both very simple and very complicated: we've spent the past six months or so concentrating on paying down our technical debt.

Technical debt, as detailed in that Wikipedia article, is the collective IOU you-the-developer write to your future self. Let me just do this fast now, and do it right later, you think: let me do the quick but messy way now and do the correct way later. But of course, the minute you ship, you're moving on to the next big thing, and you never do go back to fill in those FIXMEs and TODOs in the code, until the next time you have to touch that area and what would otherwise be a five-hour fix has turned into a twenty-hour fix because you have to "pay back" the debt you incurred last time.

More than that, you've got to pay the debt plus interest: it's a common truism that code you yourself wrote six months ago is as impenetrable as code written by a complete stranger, and you have to spend a great deal of time puzzling out what the heck you were thinking back then. (Code that is brilliant, flawless, and crystal clear when you write it slowly morphs into idiotic, bug-ridden, and clear as mud over time. This is a well-known process. I suspect pixies in the source code repository, working their anti-magic while nobody's looking.)

Now, like real-world, financial debt, sometimes there is prudent technical debt: sometimes it makes sense to incur those future obligations in order to accomplish something you otherwise couldn't have. We saw a lot of instances of that during the ramp-up to open beta, where we did it one way then to "get it out the door" and opened bugs for later refactoring so we didn't lose track of the debt we were incurring. (There's a good article about the technical debt quadrant that discusses these various types of technical debt, and when they might be useful.)

There's also scenarios where you don't realize you're committing to technical debt until later: you do something one way based on your best understanding at the time, and down the road -- due to new technology becoming available, new people joining your team with new strengths that allow you to work on things you couldn't before, new sources of funding appearing, new progress in hardware capabilities, etc -- you realize, oh, hey, we could have done it this way instead, and it would have worked much better.

In short, there's no way to completely eliminate technical debt in software design, and more than that, you wouldn't want to even if you could. Technical debt, like the responsible use of revolving lines of credit or obtaining a mortgage to buy a house in the real world, is an important part of the software lifecycle, and responsible use of technical debt is a tool that can enable a software project to succeed where they otherwise might have failed.

In a 'typical' two-year-old software project, the amount of technical debt you've built up -- if responsibly managed, which I like to think we've been pretty good at with the things we've made so far -- is fairly negligible, and you can probably get away with spending 10-20% of your coding time and effort to making payments on your technical debt without problems, thus leaving you 80-90% of your time to advance new user-facing features and improvements.

The problem is, Dreamwidth isn't a two-year-old software project. We've only been open for two years, true -- but when we opened, we forked from LiveJournal, which sprang into life in Brad Fitzpatrick's dorm room in 1999. In forking our code from LiveJournal's, we inherited that decade's worth of features, fixes, and improvements, but we also inherited a decade's worth of deferred technical debt. Dreamwidth isn't a two-year-old software project; it's a twelve-year-old software project. And the face of technology has changed quite a bit in those 12 years.

In the decade LJ was under development before we forked our code from theirs, there were of course payments being made on that technical debt; it's necessary in order to move forward. Really big payments, though -- the technical equivalent of paying down your credit card in one lump sum because you've come into a windfall -- were mostly deferred.

So, one of the things we need to do in order to move forward in a lot of instances, to take advantage of the advances that have been made in technology since various features and bits of site design were first coded, is to do all the work necessary to get to a place where we can take advantage of them.

I'll give you an example: when we forked from LJ, the code "out of the box" would only work on the Apache 1.3 series of web server software -- and early releases of 1.3, at that. (The web server is the program that runs on your machines and handles how to serve web pages to your browser when you ask for them -- if you don't have a web server running, nothing else works.)

The first release of Apache 1.3 was June 6, 1998 -- it was cutting-edge when Brad started LJ in 1999. The latest version that the LJ code would work with was (I believe -- I may be wrong) 1.3.17, released January of 2001. Subsequent versions of Apache wouldn't work with the LJ code -- it would cause horrible errors and prevent the site from running at all. To upgrade to later versions of Apache (which had more features, fixed security holes, and in general were more technically advanced) would have taken a lot of work on the code.

Even at that time, Apache was working on Apache 2.0 -- a much more technologically advanced version. The first 2.0 release was in March of 2000. The problem was, it was mostly incompatable with the optimization tricks used under Apache 1.3, and LJ code was highly optimized to take advantage of the 1.3 series. To port LJ over to the Apache 2 series would take an incredible amount of effort -- aka, technical debt.

By the time we forked the code, in mid-2008, Apache 1.3 was nearing the "end of life" -- the point past which Apache would refuse to support it, refuse to issue any additional updates, and generally say, look, c'mon, it's been nearly a decade, upgrade already. (It was formally EOL'd in February of 2010, but everyone agreed it was well past time.) So if we accepted the requirement for Apache 1.3, we were already tying our hands, and making it incredibly hard for our developers: the Apache 2 series had been the standard since at least the middle of the decade, and if someone wanted to run DW, either as a production website or to do development work, they would first have to spend hours actually downgrading their server in order to make the code work.

Before you guys even saw DW, before we could even get to the point where people could install and start hacking on the code, [staff profile] mark had to spend months making the code work under the Apache 2.0 series -- he had to sit down and pay the technical debt. The same thing happened with Perl, the language DW is written in. LJ had been written in an earlier version of Perl, and later versions had some backwards-incompatabilities; DW had to modernize before we could upgrade.

The things we need(ed) to do aren't all that obvious, of course. Those of you who came from LJ know that each page you see on LJ has the suffix ".bml" on the end of it. BML stands for "Better Markup Language" (or "Brad's Markup Language"), a templating system that produces the framework to generate a page. (Things like the site skins --Tropospherical, Celerity, Gradation, etc -- are a function of BML; the contents of the page are the same no matter what, but the templating system builds the various 'looks' of the page so you can just swap them in and out and the contents don't change but the display does.)

In 1997-1998, when Brad was working on FreeVote (his project before LJ) and the first iterations of what would become LiveJournal, there wasn't anything better out there, so he had to "roll his own". He got it to a place where it would work for LJ, and then -- because there wasn't any real need to advance things further, since it already did everything he needed it to do -- mostly stopped work on it. The face of the web changed a lot since that point (this may be a wee bit of understatement), and today there are a lot of incredibly powerful templating systems out there that do way, way more than BML does -- and are under active development, so there will be future awesomeness coming out of them.

Additionally, making people learn BML -- which is a fairly impenetrable system -- in order to contribute to DW would be silly -- it's a barely-documented custom language that only exists on less than half a dozen websites in the world. The other templating systems out there are in wide use, and it's way more likely that someone will already have the skill set necessary to contribute. (Not to mention, it'd be nice to have a templating system that has actual books written about it rather than a few web pages here and there.) Switching to a more standard version makes an incredible amount of sense.

It's also an incredible amount of work. (We've been plugging away at it, bit by bit, for over a year. At this point, the pages you see on DW itself are half generated by BML, half by Template Toolkit (the templating system we chose) -- you never see the difference, if we do our work right, but our end goal is to get rid of BML entirely.)

There are a lot of examples of things like this, from the "big project" issues to smaller things (like the need to take duplicated code -- where the same block of code is repeated in multiple places -- and move it into a function that can be called from anywhere instead, so that people only have to update one area instead of many). There are cases where functions and features that were revolutionary when they were implemented on LJ have aged over time while the technology has advanced in leaps and bounds -- a good example there is the implementation of the inline cut-tag expander; when "lj cuts" were first introduced on LJ waybackwhen, that technology didn't exist, or would've been too much of a pain in the neck to implement. Over time, it became easy; from becoming easy, it became expected, until the point where a site that didn't implemented it started to look clunky or backwards.

People have come to expect a lot of things from their social websites, in terms of 'standard' technical abilities -- I'm sure you can think of about a dozen things that other, more recently-designed websites implement that DW doesn't. We'd love to have those features. (I probably curse their lack about twice a week.) The problem, again, is that the backend code for those pages was written so long ago that before we can drop in those features and functions, we have to modernize everything. You'll never see the work. But we have to do it before we can move forward.

Some people have asked us why we committed to the LJ platform when we knew we'd be accepting the IOUs of ten years of the programmers who came before us. The answer is twofold:

1) In many cases, we were those programmers. We knew the code, knew what it could and couldn't do, had a reasonable perspective on what we were in for, and were incredibly familiar with the way it worked, the way it ran, and the way it was put together. (I say 'we', but I mostly mean [staff profile] mark there. I didn't get much into development until we started DW. But I still followed along with the technical discussions, and I had a pretty strong grasp of the technical end of things even though I hadn't been doing the coding myself.)

2) In addition to the ten years of technical debt, we also inherited the benefit of ten years of bugfixes, security fixes, architecture/performance improvements (in another 10 years when historians write the history of the early 2000s on the internet, I fully believe they will point to LJ as the technical pioneer that made a very great deal of Web 2.0 happen; the problems LJ solved back then are universal to any high-load system, and the solutions they/we came up with are still in use today), and feature development.

We believed, and continue to believe, that the LiveJournal system and code contains some of the most incredible social features out there, to the point where even today, ten years later, there is no other site that does everything the LJ code does and does it as well. A lot of that amazingness is buried, now, under a lot of "usability problems" that are actually relics of the fact that nobody went back to modernize things once the first draft was released. (To be fair, there are tons of usability problems that are actual usability problems, and were at the time the feature was released, as well. And I don't want to intimate that LJ-now is ignoring these problems either; they've been doing a lot of work on their own technical debt lately, as evidenced by the number of people who accuse them of not working on any new features either.)

One of our major goals with DW is to take the awesomeness that is inherent in the LJ codebase and bring it into the "modern era" of web design and function. We've made some great strides, but we're still only part of the way, and every time we set out to do something new, another whole chunk of problems that we have to address first pop up. It's the technical equivalent of having to learn to crawl before you can learn how to walk: before we can complete the new update page redesign, for instance, we need to do an incredible amount of work in order to make it possible to intermingle JQuery (the Javascript library that we need for the modernized widgets on the update page) with the existing Javascript the site uses. (Among many other things.)

So, when you see a code tour that's full of nothing but backend improvements, don't think of it as "DW isn't doing any feature development". Think of it as "DW is doing the necessary background work to enable awesome feature development in the future". The work we're doing now is going to pay off in the future, and it's going to allow us to do epic things.
reddragdiva: (geek)

[personal profile] reddragdiva 2011-04-06 11:28 am (UTC)(link)
This inspired me to start a page called "Technical debt" on our intranet wiki. So far I've got the technical debt I've run up or noticed personally ...
technoshaman: Tux (Default)

[personal profile] technoshaman 2011-04-06 01:47 pm (UTC)(link)
Nice writeup. I'm a sysadmin by trade, but I've dabbled in dev in the past, and this was a very interesting concept, very well explained. A lot of adminning (and DBA'ing, the other thing my group does) involves writing scripts to make our jobs easier, and the concept of technical debt definitely applies there!

Methinks I shall link to this to give the gang a little light reading. Especially since we're about to enter a major project phase...
soc_puppet: Words "Language Barrier" in yellow (Language Barrier)

[personal profile] soc_puppet 2011-04-06 02:26 pm (UTC)(link)
This was a fascinating explanation, thank you for taking the time to write it up :)
vlion: cut of the flammarion woodcut, colored (Default)

[personal profile] vlion 2011-04-06 02:51 pm (UTC)(link)
A great description of technical debt.

One thing I wish DW had was a +1 or Like button. Often there's just this slew of "Seconded, IAWTC" posts, and it raises the noise::signal ratio.

Other than that, I am very happy with the DW capability set. New features are just, well, icing. Reliability is very high and that makes me happy.
Edited 2011-04-06 14:52 (UTC)
fyreharper: (Default)

[personal profile] fyreharper 2011-04-06 11:10 pm (UTC)(link)
I just saw a suggestion about that! It's at if you haven't seen it and want to join the discussion.
lizvogel: Good / Bad (Good Bad)

[personal profile] lizvogel 2011-04-06 08:16 pm (UTC)(link)
One of the reasons I signed up with DW in the first place was this very attitude about the code -- the commitment to fixing rather than just patching, the idea that digging in and making it work right rather than just right now was the way to go. Having worked (support, ghod help me) for a couple of software companies whose approach to technical debt was "I can't be overdrawn -- I still have checks left!", I for one am deliriously happy to see DW holding off on some of the user-facing changes until the foundation is solid.
rabid_bookwyrm: Black and white illustration of an anthropomorphized margay cat (Default)

[personal profile] rabid_bookwyrm 2011-04-06 08:31 pm (UTC)(link)
I'd just like to thank you for your use of gender neutral pronouns at the top of the post.
hirez: (Default)

[personal profile] hirez 2011-04-09 02:08 pm (UTC)(link)
I keep coming back to re-read this.

I'm a reasonable fan of hacker-histories like 'Soul of a new machine', 'A computer called LEO' and 'Backroom boys' because they're part of the tiny number of books about hackers/hacking[1] that actually speak to me about my own life and don't completely lose track of the technical stuff.

Your piece is easily as good as any one of those and has made me wonder quite hard about the nature of code as a wonky sort of meta-narrative about the mental state of the people writing it. I think one would need to find someone good at both critical theory and debugging to give a proper go at (re)interpreting The Text, though.

[1] I use the word in the traditional sense, rather then the l33t/scriptk1ddy one.

(no subject)

[personal profile] majorshipper - 2011-12-29 02:52 (UTC) - Expand
love: (Default)

[personal profile] love 2011-04-10 05:59 am (UTC)(link)
This is the post that has convinced me to buy a seed account. I have always appreciated DW's transparency. I'm one of those people who adopt a wait-and-see attitude to new things and then switch when it seems like I should. You have just convinced me I should. Just commenting to tell you that this is why DW's userbase is growing, even though resistance from my LJ-only friends continues to be strong.
dimrub: (Default)

[personal profile] dimrub 2011-04-11 10:16 pm (UTC)(link)
Thanks for writing this up. Awesome stuff.
lovingboth: (Default)

[personal profile] lovingboth 2011-04-14 09:54 pm (UTC)(link)
We believed, and continue to believe, that the LiveJournal system and code contains some of the most incredible social features out there, to the point where even today, ten years later, there is no other site that does everything the LJ code does and does it as well.

I believe this, but please please please have some examples of why it is better than some later arrivals somewhere prominent: either on the front page or one easy to find click away.
sircaliban: (Default)

[personal profile] sircaliban 2011-06-02 05:48 pm (UTC)(link)
Thanks for writing this up.

It definitely makes me want to chip in and help. I'm a programmer, but I come from the dotnet/windows world. over the last year, I started working with jquery and it's definitely the direction in which to go.

I got a dreamwidth account a while ago. Maybe it's time for me to roll up my sleeves and start helping.

I think I've got a pretty steep learning curve since I come from a different world of programming.

floatboth: (Default)


[personal profile] floatboth 2011-07-26 04:00 pm (UTC)(link)
I feel your pain (or is it?). Just because I know that the admin panel of a very popular Russian real estate site only supports… wait for it… Internet Explorer 5.5.

Re: OH

[personal profile] floatboth - 2011-07-26 16:18 (UTC) - Expand
chaos_by_design: (Default)

[personal profile] chaos_by_design 2011-08-07 02:25 am (UTC)(link)
This was really interesting to read, since I'm a web developer myself. Thanks.
radiantsoul: (Default)

[personal profile] radiantsoul 2011-12-24 03:15 pm (UTC)(link)
This is an interesting read.
I came back to this after the "heat" around livejournal has recently faced when changing their comments page. I am guessing that this is what livejournal mean when they state that, our new S1 comment system was completely rewritten from scratch. The comment system has never been updated on this scale before. In the past, code was being piled upon old code, which turned out to be messy and unstable. A complete rewrite means an updated, cleaner and faster system - which is a good thing for everyone. The default comments pages are the first step in an update to our overall site design, which is meant to improve performance and user experience.

Anyway I remember seeing this post on Technical Debt before and thought it tied in quite well. I thought this was an interesting read when I first saw it, but didn't comment at the time. So belatedly, thanks for writing this!
majorshipper: (SW - Emperor's Hand)

[personal profile] majorshipper 2011-12-29 02:59 am (UTC)(link)
I have to say, this was a truly fascinating read. It, as well as the multitude of other posts I've had the pleasure of reading over here has completely and utterly sold me on Dreamwidth. There is so much amazing information to be found and learned, largely because you try so hard to be as transparent as possible and I just can't get enough.

(On an only marginally related note; this post in its explanatory and transparent nature totally reminded me of the Diversity Statement, which actually brought me to tears a couple days ago because of its simple acceptance and the way it tried to give everyone a way to participate or at the very least understand.)
snowpuppies: (Default)

[personal profile] snowpuppies 2012-02-02 03:14 am (UTC)(link)
Thank you so much for this. I'm not a techie at all, but I was able to understand and maybe even learn a bit.

I love how open and honest you guys are about this project. It makes me love DW just that little bit more because it feels like home.
juliet316: (Star Trek TOS: The needs of the many)

[personal profile] juliet316 2013-11-25 12:59 pm (UTC)(link)
Late post is late, but thank you for this explanation.
icelightning: XXY male bettafish, started off looking girlish (Default)

[personal profile] icelightning 2014-04-12 03:04 am (UTC)(link)
This is a late comment, but I just found this post.

Technical Debt - I needed this term. It makes so much sense. I've been dealing with it since I became a team's new programmer, and now I understand what it is. Thank you for that!

[personal profile] jazzyjj 2014-05-05 03:12 am (UTC)(link)
Thanks for explaining yourselves. I just registered on here and know a little bit about this more technical stuff but not much. This website is very good. I am a screen reader user and one thing I like about the site is that while it does contain a lot of bells and whistles, you and your team have done a brilliant job breaking it down so that my and hopefully others' assistive technology can easily make sense of it all. I actually read about DW on another website, and you folks came highly recommended. So I signed up for a free account, and I'm glad I did. If and when I'm able to do so, I will upgrade to a paid account.
ext_104554: Tron Bonne from Megaman Legends (Default)

[identity profile] 2014-08-14 09:21 am (UTC)(link)
Found this at AO3 on a page where they explained why they don't have private messaging. I wanted to thank you for the interesting read. I especially liked that note about debt + interest. Specifically:

it's a common truism that code you yourself wrote six months ago is as impenetrable as code written by a complete stranger

I've taken two Java classes, but not in consecutive semesters. I remember being in the second class and looking at my old code, refreshing my memory, recycling some of it for homework in one case, etc. Even with the small amount of experience I have, I have to agree with that statement.

On a side note, I just glanced at your profile, since I'm not familiar with DW and I wanted to confirm that the little guy-with-DW-logo-on-him was a staff icon like I thought it might be.

Before I glanced at your profile, this side note was going to be a "Haha, your parents must have had the same great taste in names that my parents did!" little joke.

And then I saw that, in addition to sharing my first name, you share my birthday, which is kind of a freaky little coincidence.

Well, as long as you aren't also 26 years old right now, I'll refrain from asking if you're my clone. ;)

Page 3 of 3

<< [1] [2] [3] >>