Technical Debt: The Good, the Bad, and the Tao

What is technical debt? In the usual sense, technical debt refers to the technical problems we introduce when, while developing a product or feature, we work quickly (often amid chaos and various constraints) under limited time and resources to finish work that could not normally be finished. These problems can make it difficult to add new features later, or cause the features we build to hit capacity bottlenecks very quickly. This kind of technical debt—incurred because limited time and resources forced us to sacrifice quality—is the most important kind.

The second kind arises when the development team fails to do sound design up front, resulting in a confused architecture or over-engineering that plants technical problems which erupt later.

The third kind emerges over time: an originally well-designed architecture keeps accumulating debt of the first kind, until the code becomes a stack of case-by-case patches. No one dares delete the old cases, new cases keep growing, and the codebase swells to an unmanageable size.

Yet another kind comes from the limited ability and skill of the development team: the code they write is poor in quality, planting technical problems that erupt later.

Most software teams carry several of these debts at once. All technical debt must eventually be replaced by a solution with a sounder architecture, clearer interfaces, better efficiency, and greater capacity—that is what we mean by repaying the debt.

Note that technical debt always describes work that was completed but left regrets or hidden risks. Work that is not completed is called tech failure, and is outside the scope of this essay.

The good

Debt is one of the greatest inventions in human economic activity; its importance ranks alongside paper money. We can say with confidence that wherever there is an economy, there is debt. As members of the peasant army who hoe weeds under the noonday sun, we coders are most familiar with mortgage debt. A mortgage lets us buy today a house we could only afford in the future, even when our current cash flow is short. Without mortgages, we could not live in homes we own.

Technical debt is called debt for a reason: it has the drawbacks of debt, but also its advantages. Programmers do not live in a utopia where they can pursue architectural perfection without concern; we live in a real, bloody commercial world. Sales needs a product to compete and win deals; marketing needs to weave a beautiful, colorful story for a launch; customers demand a feature be finished within a deadline, even though they themselves do not know when they will use it (usually it is just to assert the Party A swagger of “everything I say, you do”). Engineers must deliver within the deadline. If they fail, the product does not sell, marketing cannot promote it, and the customer walks away. Pay the mortgage? Buy baby formula? Forget it. At such moments, incurring technical debt is a forced and smart move.

In startup circles this practice has a beautifully chilling name: MVP (minimal viable product). The respected old hand Bill Gates—now nearly forgotten in tech circles—was a master of it: the BASIC interpreter was one case, and so was DOS. If he had refused to pay technical debt, he probably would have died before earning his first pot of gold, and Microsoft might have had no place on the map of the software world.

Another classic, and also lamentable, case of successful “debt-financed operation” versus failure for refusing debt is MongoDB / RethinkDB. As a database, MongoDB’s play in its first two versions was appalling: a database that relied on mmap for efficiency and on fsync for durability (if you did not enable the oplog, you simply left things to fate), and then had the nerve to ship 1.0 all the way through 2.x—give me a break! Tech veterans could not stand it; trenchant critiques poured out, such as the classic “call me maybe” (Google it yourself: call me maybe MongoDB). You can see that MongoDB pushed technical debt to the extreme, and in a part that is life-or-death for other software companies: the database. RethinkDB went to the opposite extreme: for the sake of technical correctness, it kept its head down and cultivated its internal strength. Both products released their first versions in 2009. By 2011, MongoDB was already being used in production; only by 2015 did RethinkDB coyly claim it was production-ready. The result was that RethinkDB missed the NoSQL boom, could neither earn nor raise enough money to sustain operations, and had to disband its team. MongoDB, after acquiring the WiredTiger engine, largely solved its most criticized technical debt and has traveled the software-service road fairly smoothly.

So do not fear technical debt. On the contrary, dare to incur it—debt-financed operation—especially before a new product or feature has proved its value. By incurring technical debt, we trade the room to maneuver that software will have in the future for time to develop now. In this sense, MVP applies not only to startups but to companies of any size.

The bad

The drawback of debt is, naturally, interest. A mortgage may fulfill your dream of a settled life, but a recession-induced layoff can suddenly rob you of your ability to repay. The bank will take your house, ruthlessly smash your pipe dream that “life is ever upward,” and reset everything to zero. So with debt like a mortgage, you must watch short-term solvency, keep the current ratio or quick ratio within a reasonable range, and hold enough free cash flow for emergencies.

For a company’s operations, too much short-term debt can disrupt normal operations and even make the company insolvent, forcing bankruptcy or debt restructuring. For software development, the interest charged by technical debt is longer development cycles for new features, or old features quickly hitting a ceiling. This affects the company’s next stage of growth. If the interest on accumulated technical debt cannot be paid for a period, the company may go technically bankrupt and then operationally bankrupt. In Twitter’s early development, its MVP chose Rails. Rapid development brought rapid product validation, but Rails’ inefficiency soon made Twitter hit a technical ceiling (despite countless optimizations): in 2007–2008, Twitter went down at the drop of a hat, and once went down for three days ( https://techcrunch.com/2008/04/22/twitter-may-not-have-to-care-about-uptime-any-longer/ ). Later, after a change of technical leadership, it learned its lesson, carried out drastic debt restructuring, abandoned Rails, embraced the Java ecosystem, rewrote many core services in Scala, and finally stabilized the service.

A particularly nasty problem with technical debt is that debts stack. In life we rarely stack debt: if you have a mortgage, you cut expenses and avoid simultaneously taking on high-APR credit card debt. In software development, we stack debt very easily. Stacked technical debt makes interest grow exponentially, which then turns on us.

Another very serious problem with technical debt is backward compatibility. Some flaws in interface design are treated by users as features, and are used so widely that developers cannot repay the debt in a new version; they can only passively preserve the debt forever. During Windows’ evolution, a great many flaws and design mistakes were treated by developers as features, making it impossible for Microsoft to change those unreasonable interfaces. This is a lesson from history, and we must be careful with this kind of technical debt. One important principle is to design external interfaces (APIs, SDKs, etc.) very carefully. If you must incur debt, make the implementation behind the interface incur it.

The tao

Technical debt is no flood or beast; we should control it reasonably and let it exercise debt’s advantages. The way to handle it:

1. Embrace MVP. Solve food and warmth first, then think about repaying debt.

2. Outsource technical debt. To use the fashionable phrase, tech debt as a service. Give infrastructure to AWS/Azure, monitoring to Datadog/New Relic, search to ELK/Algolia, analytics to Mixpanel/Sensors Data (a friend’s company—free unabashed plug), payments to Stripe/WeChat/Alipay… Transfer your non-core technical problems and hard problems to more suitable teams. Besides the monthly service fee paid as interest, this debt carries another hidden and expensive interest: vendor lock-in.

3. Hire the best people you can get, and give them the authority that best unleashes their abilities. Technical debt caused by people is the most painful, speechless, heartache-inducing technical debt of all. No explanation needed.

4. Embrace Conway’s Law. Your organizational structure determines your code structure. If you want fast, independent feature delivery, you need end-to-end feature teams that include all roles and hold direct decision-making power—not separate development, testing, and operations teams reporting to different VPs, with priorities completely out of tune, each fighting its own battle. If you want to use microservices, then organizationally you need to build parallel service teams. Much technical debt appears because technical direction and organizational structure are out of alignment.

5. You can incur more debt in implementation, but minimize debt at interfaces. Have you ever asked yourself: apart from learning the art of writing code from others and educating newcomers, why do we need code review? Why should I care how someone else’s code is written? Do I care how memcpy is implemented? Do I care how Twilio actually sends an SMS to my users at the code level? I do not. I care only about the interface. And what matters in software is precisely the interface. Why do we not have interface review? Why is such an important interface review mixed into the far less important code review? Code can be discarded and rewritten, but once an interface is confirmed, changing it is costly, especially an external interface.

A digression. If we did only interface review and no code review, would there be serious consequences? It seems not—if the code is badly written, throw it away and rewrite it. If we built software with this mindset, would we still need the whole development team to use one or a few programming languages? Would we still need to set up peer backup for every engineer? Once an engineer leaves, if their work can be rewritten when needed, is that enough? On this point, Bezos—who does not understand software—had remarkable foresight: in 2002 he proposed a series of demanding ideas about service instances and forced R&D teams to follow them. See my article:

6. Realize that software is an organism with metabolism, and build software around this idea. Code needs metabolism: new code keeps being added, old code keeps being deleted. In a living organism, broken cells are remade and dead tissue is regenerated. The organism does not try to “repair” cells that are born, age, sicken, and die; it kills and metabolizes them, while regenerating cells with the same function. When we write code, we should likewise keep every part independent (like a complete cell or tissue), leaving room for ourselves or others to delete and rewrite it completely later. Then, when we actively or passively incur technical debt, we will not panic—because we know we have retained the right to delete and rewrite.

Why, when dealing with technical debt, should we consider rewriting rather than refactoring? Refactoring code with too much technical debt carries the cost of refactoring: to refactor code, you first have to figure out the bizarre logic of your predecessor (or of yourself a few months ago), and you have to be careful not to introduce problems yourself. Often, code with too much accumulated technical debt also suffers from problems that are hard to scale and extend; rather than cautiously refactoring while looking over your shoulder, it is cleaner to rewrite, and it avoids a half-baked project. Of course, if you can reach the ideal where every feature or component can be rewritten at no great cost, then choose an architecture centered on that, such as “microservices,” and make related rules—for example, a service should be no more than one programmer-week of work, so it can be rewritten at any time.

Rewriting code instead of refactoring is a crazy enough idea, but we have a perfect example close at hand: Unix. Does anyone argue that Unix is not a system made of a series of microservices? Probably no one. We should organize the systems we build as a core surrounded by individual ls / grep / awk / vim… As a great piece of software, Vim’s code quality is truly hard to praise, so someone simply started from scratch, following Vim’s interface to build Neovim.