A plea to Engineering Managers: Let’s talk about invisible tech debt!

I want to talk about “invisible tech debt”; a particular subset of tech debt, which is much harder for project managers or system owners to be aware of. And even the developers can stop noticing it after a while.

Tech debt in a project is inevitable, and honestly…not inherently a bad thing if well managed. Just as in the analogy it’s drawing from: financial debt is crucial to setting up many businesses, and tech debt accruing shows you that your platform has changed and progressed. As much as your developers say they’d love to fix all the tech debt, they probably know there are things on their lists that don’t actually deserve to be actioned, even in the most fastidiously clean of projects^[1].

…which makes the question of which tech debt to address, and when, all the more interesting.

You’re already talking about tech debt

Before I go into more detail about invisible tech debt, a few words on tech debt more broadly:

A great deal has been written about the concept of “tech debt” in general terms, and different perspectives include different things under the label. For anyone unfamiliar with the concept, or who’d like more information about the concept in general, here are three well-regarded takes on the subject.

For this article’s purposes it suffices to say that “tech debt” refers to aspects of a system or codebase that make it hard to work with, or to change; they are things that “cost” you productivity – each new feature must pay a small “tax” to move forward, to mitigate the impact of the tech debt on that new feature. And that cost continues to be imposed until a chunk of time is spent solely on addressing the underlying cause of the debt – hence the analogy with financial debt.

The common caricature is that clients don’t care about tech debt: they like new features, they like them fast, they like them cheap, and they like them without notable user-facing bugs. Whereas the caricature of devs is that they’d like zero tech debt: they like robust, maintainable code and a comprehensive gold-plated test suite that can detect every whisper of an issue. As with all caricatures, neither of these is fully true…but also, neither is fully false!

This is fine – people just value different things to different degrees and see different impacts from different points of view; it’s not about Right and Wrong. All we can do in an interdisciplinary team is communicate clearly so managers and decision-makers have the best possible information upon which to base product/platform/system decisions, and so developers have the best understanding of the business-facing scope of changes to that product/platform/system when modifying it.

But it’s hard to discuss something that’s invisible to you

One kind of tech debt that most commonly becomes contentious when discussed is the kind that slows the team down just a little bit for week-to-week work. It’s not the sort of tech debt that prevents you from building a new feature; it doesn’t block a major system upgrade that’s needed for security; it doesn’t even change the estimate of any single card – it’s the tech debt that’s just a bit of metaphorical grit in the development wheels. I’m calling this “invisible tech debt”, and it can feel pretty draining.

As with all other forms of tech debt, the nature of the root problem can vary wildly:

• a core library that’s difficult to use
• a migration system that makes it tedious to change the database schema
• something that makes your tests painful to debug
• and many more examples

These examples of invisible tech debt can be contentious because they’re so hard for the client to visualise some concrete “return” on fixing, so the team may not be as likely to bring them up as other, more concrete, issues. Especially if they’ve had failed attempts to pitch similar work in the past. And worse, examples like these are completely invisible outside the development team. They likely wouldn’t even be visible to an overseeing Technical Principal or Architect because there’s never really any obvious reason to discuss them. Developers, like many other job roles, get used to a certain amount of grind, and can give up on hoping for better.

Issues like these aren’t big enough to specifically delay a ticket by a substantial margin, so they don’t come up in sprint status updates. They aren’t blocking the work such that they need to get fixed, so they don’t get brought up during estimation or sprint planning. They aren’t complex enough to work around that they need major technical escalation to help with a work-around, so the Principal/Architect doesn’t get called in. They’re the things that the new joiners mention briefly at stand-up in the first month of being on-project, saying could someone help with a weird error, and four different mid-level devs all say: “Oh yeah, I know that one; it’s a thing; I can show you”.

They’re the things you only notice when you are working on the code, and after a month of doing so, they’re just the norm – the stuff you don’t mention any more – so they never get into your work-tracking systems or scoping conversations. But they’re still slowing your team down, they’re still potentially valuable to fix, and they should definitely be included in the discussion of what tech debt does and doesn’t make it onto the to-actually-do list.

The nature of the right fix varies a lot with the specific cause of each problem. Sometimes it’s about choosing the right libraries and testing approach upfront; sometimes it’s a dev adding a small helper function, which the rest of the team can then use everywhere in the future; sometimes it’s starting a suite of helpers to make that annoying inner library easier to work with; and sometimes, yes, it’s “spend a bunch of time as a one-off to fix a deep underlying issue that’s causing particular pain”.

But even though the cost can vary quite a lot, if issues like these are happening at all frequently then fixing them can be worth it in the long-run (and often even in the medium- or short-run!). The time spent on fixing these problems properly can be saved multiple times over by the time not spent on trying to deal with every time thereafter.

And the developers on your team will certainly thank you for the improvements to their quality of life, which always saves money in the end – a long-term stable and happy team who have years of experience with your codebase and your personal business domain will have a big advantage over an irritated team with constant churn, where the most experienced dev has only been there nine months.

But any painful tech debt will get fixed naturally over time, won’t it? Not necessarily!

You might assume that pain-points like these will get logged in your general work tracker and will get prioritised alongside all the other bugs, tech debt and features; that if you have a healthy process to prioritising work then you won’t have this issue. But that’s frequently not the case.

Most of the problems with invisible tech debt aren’t bugs that end up getting tracked in the bug tracker, and even if they were, they probably wouldn’t get prioritised, because their individual impact to a project looks almost negligible – no angry users, remember? No error messages in live log files. No tickets sitting blocked for multiple days. Just…oh, that ticket took me an hour longer than expected, and a team of slightly sad developers. But these small impacts stack up easily during development and could well be costing you money that you haven’t factored in (or, more hopefully, are costs that your team are factoring into estimates as a statistically present overhead, which could therefore be lower than it is!)

No individual dev has the need, time, or authority to fix a complicated cross-cutting problem, and the small simple inefficiencies can seem too trivial to bother with in isolation.

To start seeing invisible tech debt, you only need to a) care and b) know how to ask

Fortunately, getting visibility of the cost of invisible tech debt is easy once you’ve decided you care about it: ask your dev team(s)! They probably already know.

Ask them what their pain points are in day-to-day work, try to work out how much each issue is costing the whole team over the course of, say, a year and then consider giving the team some time to address the root cause properly. Note that some tech debt is really hard to quantify the cost of – but that doesn’t mean it isn’t worth doing. If your devs say something is definitely valuable to fix, please take that as a strong prior it’s something important!

Or alternatively, (and this may not be palatable for every organisation!), decide on a small amount of baseline investment that you want to spend on what might be called “routine-maintenance”, or “lights-on upkeep”. Give that monthly pot to the developers, to be spent on whatever things they find most painful, and let them spend it as they see fit. The goal is to fix the pain points; they have every incentive to fix the pain that they’re experiencing, as efficiently as possible with the time available! And they’ll love you for giving them the autonomy too.

The time you invest in fixing invisible tech debt will probably pay back well within the lifetime of your system, and once it’s paid back, you’re left with a team that can develop faster – and a team less frustrated by day-to-day quality of life issues. You probably already have at least one dev on the team who has been wanting to dig into “why did that bug cause this seemingly unrelated error over here”, or “make it so that they never have to deal with that damn class again”.

Here are some example questions to get the conversation started:

• Which classes are annoying to work with?
• What weird tricks do you have to teach new team members, to work around speed bumps?
• What things make experienced devs look weirdly at you during on-boarding, when you say, “Oh yeah…that’s just a thing we do on our project”?
• Which parts of the system are most likely to make a ticket run over the estimate a little bit?
• Where do you most often introduce bugs which you catch yourself?
• Which test errors say one thing, but actually mean that something different has gone wrong, elsewhere in the code?
• What would you fix first if you had a bunch of time to yourself?

Give a developer the time to dig into their problems, and then let the rest of the team enjoy the fruits of their labour once they’ve resolved them.

And if your team can tell you what hurts, but you don’t have anyone on the team who wants to dig into it, or has the necessary depth of knowledge to fix it…well, hey, we know a couple of folks who could come and fix it for you 😉

---