Paying Down Technical Debt

This article is the second part of our series on why COBOL and lack of COBOL developers aren't the real problem in maintaining legacy systems. See the first piece here.

In the first part of this series, we headlined the most common mistake non-technical people make with legacy systems and technical debt: blaming the programming language the system is written in. We laid out in great detail why the programming language is only a small part of the problem with maintaining large systems.

In short, programming languages tend to be very small and easy to learn, at least for experienced programmers. (In contrast, actually learning how to program is quite difficult, regardless of language.) Existing programs, on the other hand, can be very difficult to learn and understand, not least because they often introduce more application-specific words and ideas than exist in the programming language they are written in. 

However, both the programming language itself and the size of the programs are trivial problems compared to the difficulty of understanding a system that was built poorly. Experience has shown that long-lasting systems need a kind of constant maintenance known as refactoring. Without this maintenance, systems tend to build up a complexity that is very expensive to fix and becomes more expensive with time. Because the price of repair grows with time, it has become common to call this lurking problem technical debt.

Now that we have given a high-level overview of technical debt, we are going to discuss ways that decision-makers can approach technical debt. Technical debt is not inherently bad so much as it is real and demanding, much like monetary debt. None of these solutions are a silver bullet; technical debt is just as real as monetary debt. 

So what do you do? We have a few suggestions.

1. Don't play the blame game

Technical debt comes from many places. It may have been created by former members of your team. It may have been created by the current team. It's normal for code that was fine when it was written to gain technical debt as the world changes around it. 

Even if the code was "bad" when written, it may have been the right choice given the deadlines when it was written. It could have been working around a strange mistake in a third-party product, now patched. Or it may have been written by an intern on their first important project. Or maybe the developer had a bad week.

In short, technical debt is normal and is the cost of doing business with computers. Barring truly damaging incompetence, as measured by their peers, shaming developers for creating technical debt is shaming them for having done their job, and developers know it.

Blaming previous management—or even yourself—is counterproductive. Your current state is what it is. Don't worry about why or how it happened, as long as you can make sure it doesn't happen again. All you can change now is the future.

2. Know what technical debt is and how much your organization has

The old cliche is true: Knowledge is power. In this case, simply knowing that technical debt exists in general, and where it exists in your organization in particular, is a good start. The worst-case scenario for an organization is to have looming technical debt and not know about it. (Don't say you don't have any without a serious audit. Nowadays, you'd probably be wrong.) 

While technical debt can be hard to measure, we recommend you try to quantify it as much as possible. Don't try to find any fancy theoretical definitions or metrics of technical debt. Since all estimates of technical debt are going to be wrong, you are best served using accessible metrics like "time" and "money." Ask your technical employees practical questions like "If we followed your suggestion, how much time would it take to implement the new framework?" or "How many hours a week are our programmers working on fixing bugs versus adding new features?" In particularly extreme cases, it makes sense to ask "How much would it cost to throw this whole system out and replace it?" 

Don't worry if it takes time to get the correct answers. These things are difficult and take experience to get right.

3. Manage technical debt intentionally

It's not a bad business decision to take on technical debt any more than it's wrong to take out a loan from the bank to buy a car. There are often important business concerns or deadlines that make it more important to get a system that works rather than something that matches some standard of beauty. That said, informed leadership should always know that that is what they are doing.

In addition, you should have a deliberate plan for both how much debt you consider acceptable and how you are going to pay it off. This is part of why quantification is important; you need to have the metaphorical system light set to turn on when the debt is getting too high.

4. Follow the plan

When the plan requires you to pay down technical debt, you need to actually do the maintenance. As we've been discussing, maintenance will have many business benefits in the medium to long term: you will have fewer bugs and failures, adding new functionality will be much cheaper, the developers will be generally happier, and all in all, things will just work better. In the short term, though, it can be hard to see these benefits. You need to do the maintenance anyway. 

In practice, this means you will be explicitly, consciously letting the technical folk (and sometimes forcing them to) spend time doing work where the only obvious benefit right now (if you can't read the programs) is improving their morale. 

What's more, there must be no hidden penalties for having your employees refactor. No holding back promotions, no requiring it to be outside of standard work hours. It's real, valuable work. Treat it like the business necessity it is and reward your employees for honestly doing their job.

5. Keep the balance sheet honest

An awful lot of tech debt can be built up with incomplete accounting. 

In many businesses, the costs of important refactoring are counted in full. Technical debt takes a significant amount of time and effort to pay down. If the company officially takes time to have their employees work on it, it becomes very visible that they aren't working on other things. If individual employees try to take the initiative on their own, it can damage their career or promotion prospects if management perceives that they are "wasting time." It's also very hard to actually make an impact as a single employee, so most won't bother.

On the other side of the ledger, all of the day-to-day slowdowns, bugs, brand damage, employee turnover, and missed product opportunities are consistently ignored. The status quo of slow feature development, excessive debugging, and a general lack of progress becomes accepted as normal, leaving the business wide open to disruption when somebody comes along that doesn't have all those problems. This can be good for short-term success, but in the long run, it is a good way to run a company into the ground.

Like most other kinds of debts, tech debt doesn't have to hit the people who created it, just the people who own it. If the organization intends to survive in the long term, it needs to structure itself in such a way that short-term incentives don't create too much tech debt. If you have leadership that either doesn't know or doesn't care about the tech debt being built up, you are building trouble.

6. Trust the technical staff

Technical folks can sometimes be overzealous about code beauty and truth and perfectionism. They are also often wildly optimistic about their own abilities and the potential of new technologies. (The kind of mind that can talk to a computer can sometimes be a bit odd.) Because of these quirks, it can be difficult to get unbiased information about your system from your technical staff, even if they are being completely open and honest.

All that said, none of these problems are reasons to ignore what your engineers are actually saying. These engineers are your only window (however imperfect) into what actually needs to be refactored in your system. When you need to get estimates for how much technical debt you have, their estimates are the only ones worth hearing.

Note that we are deliberately suggesting you use primarily your own developers for doing this kind of assessment. Far too many businesses and governments have tried to save money by relying on third-party contractors, hired for as long as it takes to solve the problem. The problem with contractors is that practically all of their incentives are short term by design. "Building a pile of sheds" (i.e., a badly structured but technically functional program) is a viable business model for them because they aren't left holding the bag. It's the difference between a homeowner and a house-flipper.

In general, communication between technical people and leadership (or people on the business side) is a learned skill, the result of deliberate design and effort. Management needs to learn to ask the right questions. The technical people need to learn how to make the business case for any proposed changes. Both must learn to trust the other side and learn how they work and think. It's not uncommon for technical debt to build up, with management blissfully unaware of it, because developers are, for whatever reason, afraid or unable to tell the truth.

More specifically, and much like dealing with actual computers, both management and engineers need to learn how to break down projects into small enough pieces that the technical estimates are reliable. Management needs to learn how much to pad the engineers’ estimates. The developers need to learn to correct their numbers, temper their optimism, and remember that business priorities are vitally important. Management must remember that the developers are the primary reliable source of information. If they ignore them (or incentivize wrong information), all they'll get is garbage.

7. Realize that there’s no silver bullet

Programming is hard, and dealing with technical debt is far harder. If a company has built up a serious technical debt over the years, their only real choices are to pay it down (spend time and money fixing the problem), default (stop getting the benefits of the software), or pay the ever-overwhelming interest (waste engineering talent on constant small bandaids rather than new features or product improvements).

There are rare occasions where you can just replace old software. Sometimes yesterday's in-house software has become today's open source or business software. This will rarely make a serious dent in technical debt, though, because off-the-shelf applications are practically never flexible enough by themselves to match the complexity of old but useful existing programs. Even if they are, the transition to the new application will almost certainly require transferring data and logic. Transferring data and logic is generally a complex operation that requires programming skills, which requires a company to hire developers or similar professionals. In this case, the transfer is how you pay down the tech debt in this case.

There are other popular options that actually can mitigate tech debt, but they are sometimes sold as a full solution. For example, you can hire contractors. The obvious problem with hiring contractors is that you are paying them; that is part of the cost. The larger problem is that by their very nature, it is difficult to incentivize contractors to make the code maintainable. Another commonly proposed solution is automatic code translation or other automated tooling. The reality is that while good tools can help a properly trained development or IT team, currently these tools are nowhere near smart enough to understand or fix complex code by themselves. Effectively using these tools requires having competent developers using them. The debt will come due.

This can be a hard message for many businesses or governments to absorb. There has been an entire industry built out of selling silver bullets, and experience has shown that they do not work. Like many other problems, technical debt will be solved through hard work and dedication, not magic.

Conclusions

Even though we started these articles by referencing COBOL, I think it should be clear to most readers that the problems with old and complex systems face don't actually have anything to do with that language. The real problem of technical debt is actually far worse and much more applicable than any single language, no matter how old. Instead, it is the size and structure of the code itself that make old or poorly designed systems hard to maintain and liable to failure. 

The good news is that this debt can be managed. In this article, we addressed several ways that, if management is willing to acknowledge its existence and handle it deliberately, technical debt can be paid down, including vital tools for managing technical debt. If these principles are applied consistently and honestly, organizations will find that they can keep their systems running much more smoothly and with far fewer nasty surprises or wasted time and money.