How does technical debt get built up - a tech exec perspective
Wear the exec lens, and how they get out of technical debt.
Let’s be brutally honest: tech execs do not go to their teams and yell, “let’s rack up some technical debt!”. Yet, tech execs make split-second decisions that stack up liabilities faster than you can say “code freeze”.
It’s not about being reckless; it’s an unconscious trade-off we make under pressure — pressure from the competition and the burning runway. When you’re racing against time, competing for market share, and burning money, speed often trumps perfection.
Emails from Turing
Turing started out in 2018 with the core business matching remote job opportunities to remote developers. It is now better known as the AGI powerhouse that powers complex cognitive abilities offered by OpenAI, Meta, Google, Mistral AI, and Nvidia. For better or worse, when the pandemic hit in 2020, Turing’s core business grew exponentially.
Turing got clients through hot and cold marketing intros. Initially we had paid for an email service provider to market our service — very common. Eventually, the need to scale and survive meant the exec team decided we needed “warm emails,” ASAP.
The unsung hero that made Turing successful was this in-house email service, which was capable of doing just that. It was not glamorous, but it was absolutely necessary. Unbeknownst to the exec team, as a consequence of their “ASAP” decision, the email service was built at breakneck speed just to get it working, which meant significant technical debt piled up. It was just raw, fast code that handled sending emails as quickly as possible, connecting to multiple email service providers, managing templates, and featuring a bolted‑on subscription management. Yes, it did not occur to the team at the time that some people would like to unsubscribe from marketing emails. Bolted-on code is usually very buggy.
From that point on, every engineer or engineering manager received at least one email from Turing — or some frequent duplicate emails from Turing. People either embraced it, endured it, or a minority complained vocally about it. We will come back to that group.
When things are built that way to support business needs, it works — but it has a lot of bugs. The consequences, in hindsight, were hilarious: someone received an email every minute from Turing because of a while(true) loop; some users frequently got dozens of emails in a single day because product managers decided it was just one more email; sometimes that one more email did not get sent out; when people tried to unsubscribe, the unsubscription process threw an exception and failed to unsubscribe users.
Remember the minority group who complained about the emails? By early 2022, Turing’s business had grown so much that these people were no longer a minority. Each of these people would spend a precious amount of their lifetime venting their anger to Turing. I will spare the reader from the threats and profanity (yes, I have read your emails myself).
The Moment of Reckoning
Emails are fundamental to Turing’s business. The email service, once capable of handling tens of thousands of emails, was now buckling under the pressure of millions. The problems ended up causing GDPR complaints, hits on IP and domain reputation, additions to undesirable lists, and the problems went on and on. This turned into an operational nightmare just to keep things going. It threatened our business model enough that it caught the CEO’s attention, who had been preoccupied with more important matters.
It was easy for me to criticize all the issues I have seen, but the solution was not to dismantle everything and start from scratch. As a tech exec in the core R&D team, that would have been as impractical as it sounds — and a failure of my responsibilities.
Solving the problem, not the tech.
(feel free to skip if the next part is too in-depth!)
Execs may not always care about the technology, but they definitely care if the problem gets solved ASAP.
I decided to take a surgical approach. I reverse-engineered the code enough to write a design doc, and isolated the most problematic parts of the email system—1) the subscription management and 2) email mailer components—and transformed them into a dedicated microservice. Then, 3) I introduced a rate limiter that that imposes hard guardrails on the number of emails that can be sent to a recipient over a period of time, akin to a QuotaServer commonly used inside Google.
1) Subscription Management
Why Microservice?
This was what I wrote, verbatim, from the design doc from 2022:
Many of Turing's code does not have adequate foresight and proper tests. The monolithic codebase makes things very hard to add tests and understand.
By moving some of the core and critical code to a microservice, we leverage the principle of separation of concerns: We enforce the new project to be very well-tested and start with good practices, and then gradually "hollow out" the monolithic code.
The microservice will start with the proper template and is used as a forcing function to ensure that there are proper unit tests, interface design, code coverage, etc.
That being said, there is a balance to strike: breaking a codebase into too many smaller microservices makes the code harder to trace (if done too granular and too extreme), and coordinated rollouts will be tricky. The goal is not to get rid of all monolithic code, but focus on the core few parts.
Remember the 80/20 rule, by fixing a few problematic modules of 20% effort, you can get 80% of the impact already.
I remember not even knowing how to test the code in the first place, because it was just not testable. Regretably and surprisingly, repositories without any tests were not uncommon in Turing back then.
As the DRI, I enforced the team’s adoption of Google Protocol Buffers to implement a clean interface for the unsubscription protocol, and I was able to dig up the code below to show you how much easier it was to reason about.
By isolating this component to be developed, tested, and deployed separately, we were able to ensure all those idiosyncratic cases were handled. Back in the monolithic email service, it was really hard to debug and fix the problems — look at this long list of functions and unhelpful names:
2) Mailer Service
Similarly, I had the team start extracting the mailer into a separate microservice that just sends emails, rather than a sender that also contains subscription links, formatting, tracking, and routing (not shown) all mixed together.
3) Rate Limiter
Theoretically, this would have been a standalone service, but engineers ended up deciding that it could be separated cleanly within the mailer service. Instead of remaining a single function, it was separated out into a testable component. BTW, rate limiter is also a frequently seen interview question.
The Result
With clearly defined interfaces for subscription management, mailer, and rate limiter, and over 90% test coverage, we addressed the most pressing vulnerabilities. This was not about chasing a 100% microservices utopia that proponents raved about; it was about making a targeted, pragmatic improvement where it mattered most. You know what? This solved over 90% of the problems—my job was done there!
Even into March 2025, no further re-architecture or heavy development was needed—it just works!
Final Thoughts
From a tech exec perspective, technical debt isn’t something we deliberately accumulate—it’s the natural fallout of making tough, time-sensitive trade-offs. When you’re in the fast lane, shortcuts are inevitable, but if left unchecked, they become a serious drag on innovation. At Turing, our email system was never our flagship—it was just the indispensable engine keeping our core mission (matching remote talent with opportunity) alive. The takeaway is clear—especially if you are the DRI and, more importantly, an exec: identify these compromises early, tackle them with precision, and never let a “good enough” mindset fester into long-term liability.
In the end, it’s all about balancing the raw speed of growth with the discipline to invest in durable, scalable infrastructure. Smart tech execs know when to floor the accelerator and when to hit the brakes for a tune-up—before technical debt turns into a roadblock.
very insightful, enjoyed the reading
Thanks for the example and details. We built an automated way to do decoupling of production software so a modular system emerges. Including protobufs.(kodou.io) I’ll share this link as a use case.