Architecture and Technical Health

Features get celebrated; technical health gets ignored until it’s a crisis. One of a staff engineer’s quieter, more important jobs is to keep the systems alive and evolvable over years, so the team can keep shipping features next year, not just this one. This is unglamorous work with a long time horizon (Lesson 2), which is exactly why it needs someone senior to champion it, nobody gets a quick dopamine hit from paying down debt or finishing a migration, so it falls to the person who can see and argue for the longer game. This lesson covers the main levers: managing debt, running migrations, choosing technology, and building paved roads.

Tech Debt Is a Portfolio, Not a Monolith

“We have too much tech debt” is one of the least useful sentences in engineering, because it treats all debt as one undifferentiated blob to feel bad about. Debt is a portfolio, and you manage it by triage:

Plot each specific piece of debt on two axes, how much pain it causes (slows people down, causes bugs, blocks the roadmap) and how much it costs to fix:

• High pain, low cost → fix now. The obvious wins. Something that annoys everyone daily and takes a day to fix, just do it. These are pure value; the only reason they linger is nobody prioritised them.
• High pain, high cost → plan and schedule. This is a real project (a re-architecture, a major migration). It needs a proper business case and a roadmap slot, not a weekend. This is where you write the doc (Lesson 4) arguing what fixing it buys the business.
• Low pain, low cost → fix opportunistically. Clean it up when you’re already working in that code (the “boy scout rule”). Don’t make a project of it; just leave things a little better as you pass through.
• Low pain, high cost → leave it, consciously. Document that you’re choosing to live with it, don’t gold-plate something that doesn’t hurt, and revisit if the pain rises. Not all debt should be paid; some is a reasonable loan.

The staff move is to make debt specific and legible. Instead of a vague sense of doom, you have a named list: this debt hurts here, costs this to fix, and buys that. Now you can argue for the high-value fixes with a real case, and consciously defer the rest, rather than either ignoring debt until it’s a crisis or trying to fix all of it and shipping nothing. Debt you’ve chosen to carry, with eyes open, is fine; debt that accumulates invisibly is what kills teams.

Big Migrations: Expand, Migrate, Contract

Much of the high-cost debt work takes the form of a migration, moving from an old database, API, framework or infrastructure to a new one. These are where staff engineers earn their reputation, and where big-bang cutovers earn outages. The safe pattern is always expand, migrate, contract:

1. Expand: add the new thing alongside the old. Both run at once, dual-write, or dual-read. Nothing breaks yet; you’ve just added capacity to run the new path.
2. Migrate: move traffic or data over gradually, a slice at a time, behind a feature flag. Measure each slice. Every step is reversible, if something’s wrong, you shift back.
3. Contract: once nothing uses the old thing (verified with logs and metrics, not hope), remove it. This is the scary step, made safe because you’ve proven there are no users left.

Why this beats a big-bang cutover (“we’ll switch everything over Saturday night”): the big-bang has no safe rollback and discovers its problems in production, at the worst possible time, with everyone tired at 2 AM. Expand-migrate-contract is always reversible until the final step, and each slice is small enough to verify before moving on. (This is the same pattern as the database and Kubernetes migration lessons elsewhere on this site, because it’s the pattern for safe change.)

The hardest part, and the one that separates finished migrations from zombie ones, is the contract step. Teams enthusiastically build the new thing, migrate most traffic, and then… never remove the old. The project stalls at “80% migrated”, and now you’re carrying both systems forever, which is worse than either alone. Finishing is the discipline. Track ”% migrated” as a real metric on a dashboard, make the new path the default for all new work early (so the old path stops growing), and treat the final cleanup as a real, owned deliverable, not an optional nicety. A migration isn’t done until the old thing is deleted.

Choose Boring Technology on Purpose

There’s a strong pull, especially for talented engineers, toward new and exciting technology: the latest database, framework, language, paradigm. A staff engineer’s job is often to be the voice of “boring technology.” The idea (from Dan McKinley’s well-known essay): you have a limited budget for innovation, a few “innovation tokens”, and you should spend them very deliberately.

The reasoning: a proven, boring technology has known failure modes, a deep pool of people who know it, mature tooling, and years of bugs already found. A shiny new one has unknown failure modes, few experienced people, immature tooling, and bugs you’ll discover in production. For the important parts of your system, the boring choice is usually right, not because new things are bad, but because the risk of the unknown is highest exactly where reliability matters most.

So: spend your innovation tokens on the one or two places where a new technology gives you a real, differentiated advantage, and use boring, proven tech for everything else. “We’ll use the database we know, on the cloud we know, in the language we know, and spend our one innovation token on the actually-novel part of the product” is a mature, staff-level position, even though it’s less fun than adopting five new things at once. Your judgment about where to innovate and where to be boring is worth more than enthusiasm for any particular tool.

Build Paved Roads

The most durable way a staff engineer shapes architecture isn’t by mandating standards (you can’t, Lesson 7); it’s by building paved roads, the well-supported, easy, default way to do a common thing. A paved road for creating a new service, for adding observability, for deploying, for handling auth, means the right way is also the easiest way.

This works because of human nature: people follow the path of least resistance. If doing it correctly (with proper monitoring, security, testing, deploy safety) is harder than doing it ad hoc, people will cut corners, no amount of policy stops them. But if you make the correct way a single command or a template that already includes all the right defaults, people take it gladly, because it’s less work. You’ve turned “please follow our standards” into “here’s the easy button”, and influence becomes effortless.

Paved roads are also the ultimate durable artifact (Lesson 8): they encode your architectural decisions into tooling, so they scale to every team and every new hire without you being in the room. Building and maintaining the paved roads for your org is some of the highest-leverage architecture work there is. It makes good practices the default, bad practices the friction-ful exception, and your influence a permanent feature of the environment rather than a one-time argument.

Pitfalls to Avoid

1. Treating tech debt as one blob. Triage by pain vs cost; “we have debt” isn’t a plan. Name the specific debt and what fixing it buys.
2. Trying to fix all debt. Some debt is a reasonable loan; consciously tolerate low-pain, high-cost debt.
3. Big-bang migrations. No safe rollback, problems found in production. Use expand-migrate-contract.
4. Never finishing the contract step. Carrying both old and new forever is worse than either. Finishing is the discipline.
5. Spending innovation tokens freely. Use boring, proven tech for important things; innovate deliberately where it differentiates.
6. Mandating standards instead of paving roads. You can’t mandate; make the right way the easy way.
7. Ignoring technical health for features. Someone senior must champion the long game, or next year’s velocity collapses.

Quick Recap

• Manage tech debt as a portfolio: fix high-pain-low-cost now, schedule high-pain-high-cost as a real project, clean low-cost opportunistically, consciously tolerate the rest.
• Run big changes with expand-migrate-contract (reversible until the last step); the hardest and most important part is finishing the contract.
• Prefer boring, proven technology for important systems; spend innovation tokens sparingly and deliberately.
• Build paved roads so the right way is the easy way, the most durable way to shape architecture without authority.
• Technical health is long-horizon, unglamorous work that needs a senior champion, or next year’s team can’t ship.

Next lesson: when systems fail anyway, being the calm in an incident, running blameless postmortems, and the judgment that handles risk.