Friday, November 9, 2007

Technical Debt in Agile Development

In a previous post, I talked about technical debt and how you can use that concept to understand the real costs of design and implementation trade-offs. Being a believer in Agile development, I wanted to explore what technical debt means in an Agile world.

I think one of the interesting questions around this is if it is possible to implement a feature maintaining the principle of Simplicity is Essential and still manage to take on debt.

For instance, if I have to implement a logging system, the feature calls for logging to a database, and I start by logging to the file system as a prototype, there are three options I can now take with this feature, none of which I would call debt. My choices in making a change in scope are to 1) change the requirement such that file logging is OK, mark it complete, and add RDB logging as a seperate feature in the backlog; 2) not call the feature complete and continue implementing the RDB features; or 3) pull the feature from the code altogether and put the feature back into the backlog. None of these introduce a debt in the code.

Any new feature that gets partially implemented will fall in that category: either the requirements get refined to handle it, the requirements are completed, or the code is removed. Every other case that comes to mind is in the category of "I would like to do X because it feels right, but it is not the simplest thing." Do you agree?

Actually, I don't. Read on to find out why...

There are several kinds of debt that can build up in an Agile project. We'll start with the most basic and work our way up to the harder to quantify debt.

  • Poor Unit Tests

    In agile development, this may be the worst possible debt you can accumulate. This is check-cashing, 35% interest plus weekly fees debt. Failure to put proper emphasis on unit tests will always come back with huge future costs. In test-first methodologies, there is never a time where it is appropriate to take on this debt. In test-soon methodologies, there may be one time where the cost of the debt won't be felt exponentially: at the end of a particular release. The caveat here is that your organization has to have the maturity to pay the debt immediately following the release, or the interest will rack up.

    In my opinion, the only time this would ever be acceptable is if the fate of the business/product rested on it. The cost of poor unit tests is far too high to risk in any but the most extreme circumstances.

  • Failure to Refactor

    Now we get into some of the optional aspects of software development. It is possible to forgo refactoring on projects in favor of just pushing the code out. The challenge with this is that systems get brittle as pieces start connecting in more tenuous methods. "I'll just tack this on" is a familiar sentiment, especially when refactoring forces not just application code changes, but *gasp* unit test changes. A lot of work for one little feature, but, ironically, each time the decision is made not to refactor, the pain of refactoring grows.

    This is debt that can be easily tracked. Strong developers will know when a piece of code should be refactored. Any time it isn't refactored, it should be tracked.

  • Over-Engineering a Solution

    The Dot-com boom was a marvel of over engineering. I worked for a dot-com that built, from scratch, every piece of infrastructure it needed to build a portal. These included a web templating engine, an object-relational mapper (built on a directory server, no less), a mail server, a COM-like framework in Java, and many others. In addition to building all of these, they were built in some theoretical "perfect" mindset, such that they were slow, difficult to use, and didn't really meet the needs of the clients (other developers) who were trying to get an application built.

    On top of that pain, we had to maintain the code going forward. As things changed, we were forced to update code that seemed to only exist to make our own lives more difficult. We had a huge over-engineering debt that significantly impacted our ability to produce features for customers, which impacted our own business success.

    That debt is harder to quantify, because no developer likes to believe that he is over-doing something. Worse yet, you pay for it at least twice: once to spend more time building it and again to maintain it. I believe this kind of debt should never be incurred and the only way to prevent it is through continual code reviews...imagine that, another tenant of Agile programming coming to the front. I have never experienced over-engineering coming back and paying for itself in the future: don't do it.

  • Over-Simplifying a Solution
    This is debt because an expectation has been made to a customer and the solution does not meet those real needs. This generally happens when there isn't a solid understanding of requirements, and it causes debt through customer service issues and other interruptions to the development plan. This is the hardest one to quantify, because in some cases you just don't know what you don't know, but it is also the most obvious debt from a customer perspective.

    My experience has made this The Money Pit debt. It is debt you have no idea you are getting in to, always has to be paid immediately upon discovery, and generally causes much pain and suffering. The only way to prevent this debt from occurring is close customer relations and regular feedback (does this sound familiar?).

What other forms of debt do you see as you do Agile development? How do you minimize that in your work? What is your favorite color? All comments welcome.

No comments: