Everything breaks eventually, no matter how good your architecture. When it does, the response matters as much as the system, a well-handled incident builds trust and makes the system stronger; a badly-handled one compounds the damage and scars the team. As a staff engineer, you’re often the most senior person in the response, and here’s the thing that surprises people: your most important job usually isn’t to be the one who finds the bug. It’s to lead a calm, coordinated response, and then to make sure everyone learns from it without anyone getting burned. This lesson is about incident leadership, blameless postmortems, and the broader judgment of managing risk.
In an Incident, Be the Calm
When the site is down and adrenaline is high, the senior-most engineer sets the emotional temperature of the room, and emotions are contagious. If you panic, everyone panics, and panicked people make bad changes, miss obvious things, and step on each other. If you’re calm and structured, that spreads too, and calm people debug well. Your composure is a tool.
The practices that make a response calm and effective:
- Establish roles, out loud. Name an Incident Commander who coordinates (not necessarily the best debugger, coordinating and debugging are different jobs and one person can’t do both well), a comms person who handles updates, and responders who investigate. Saying “I’m IC, Priya is on comms, Raj and Anita are investigating the database” instantly reduces chaos.
- Stabilise before you diagnose. Stop the bleeding first, roll back, fail over, shed load, put up a maintenance page, then find the root cause. Users don’t care why it’s down; they care that it’s down. The instinct to understand before acting is wrong in a crisis; restore service, then investigate from a calm place.
- Communicate on a clock. Post updates every N minutes, even if the update is “still investigating, next update in 15 minutes.” Silence is the enemy: it breeds panic, and it generates a flood of side-channel “is it fixed yet?” pings that distract the responders. Regular updates, even empty ones, keep everyone calm and out of the responders’ way.
- One change at a time. Under pressure the urge is to try five fixes at once. Don’t, change one thing, observe, record it. If you change ten things and it recovers, you’ll never know what worked, and you can’t write a real postmortem. Discipline here is what turns a chaotic scramble into a diagnosable sequence.
- Protect the responders. Pull in the help they need, and push out the spectators and anxious VPs who want status (that’s what the comms person is for). Order food. Rotate people before they’re exhausted, tired responders make mistakes that extend the incident.
- Write it down live. Keep a running timeline in the incident channel: what you saw, what you tried, what happened, with timestamps. This becomes the postmortem and saves hours of painful reconstruction later, memory of a 3 AM incident is unreliable.
The reframe worth internalising: you don’t have to be the hero who finds the bug. Often a more junior responder finds it, that’s fine, even ideal. Leading a calm, coordinated response, so that the right people can find the bug fast and safely, is worth more than being the smartest debugger in the room. Incident leadership is the staff skill; incident debugging is a senior skill that others can own.
Blameless Postmortems: Fix Systems, Not People
After the fire is out, the postmortem (or incident review) decides whether the organisation gets stronger or just scared. The single most important property is that it’s blameless:
The difference is stark:
- Blameful (“who pushed the bad deploy?”, “why didn’t you test it?”) makes people hide their mistakes, stop reporting near-misses, and get defensive. The same class of failure then recurs, but now in the dark, because nobody will surface the warning signs. A blame culture gets fewer reported incidents and more actual ones.
- Blameless (“what made it easy to ship this?”, “what would have caught it?”) assumes everyone acted reasonably given what they knew at the time, and focuses on fixing the system: the missing guardrail, the test that would have caught it, the alert that didn’t fire, the confusing default, the process gap. People surface problems freely because they won’t be punished, so you actually fix the root causes.
The mental model: human error is a starting point for investigation, not a conclusion. “Operator error” isn’t an explanation; it’s a question, why did the system let an operator make that error so easily? A good system makes the right thing easy and the dangerous thing hard. If someone could take down production with one wrong command, that’s a system failure (no confirmation, no canary, no guardrail), not a person failure, even though a person’s hand was on the keyboard.
As a staff engineer, you set this tone, and the most powerful way is to model it on yourself. When something breaks on your watch, say openly: “here’s the mistake I made, here’s the system gap that made it easy, and here’s the fix.” That makes it safe for everyone else to be honest, which is the whole point. Ask “how” and “what”, never “who” and “why didn’t you”. And make sure the postmortem produces a small number of concrete, owned, dated action items that someone actually tracks to done, a review that generates no fixed action items (or twenty that nobody owns) was theatre. One or two real fixes, done, beats a long document filed and forgotten.
A reusable incident-review template
# Incident review: [short description] — [date]
**Severity:** ... · **Duration:** [start] to [resolved] · **Author:** ...
**Status:** Draft / Reviewed
## Summary
2-3 sentences: what happened, the user impact, and how it was resolved.
## Impact
Who/what was affected, for how long, and how badly (users, revenue, data).
## Timeline (from the live notes)
- HH:MM — what happened / what we observed
- HH:MM — what we tried / what changed
- HH:MM — resolved
## What happened (root cause, blameless)
The system conditions that allowed this. Assume people acted reasonably
with the information they had. Focus on the "how", not the "who".
## What went well
Things worth keeping (fast detection, good rollback, clear comms).
## What we'll change (action items)
| Action | Owner | Due | Status |
|--------|-------|-----|--------|
| A specific system fix (guardrail/test/alert/default) | name | date | open |
(Keep it to a few HIGH-VALUE items with real owners and dates.)Manage Risk as a Portfolio
Beyond incidents, judgment about risk is a constant part of staff work, and the mature view is that risk is to be managed, not eliminated. You can’t make a system risk-free; trying to would mean shipping nothing. Instead, treat risk as a portfolio of conscious bets:
- Make risks visible. The dangerous risks are the ones nobody named (Lesson 6’s “risks & unknowns”). A risk you’ve written down, with an owner and a mitigation, is far safer than one lurking unspoken.
- Match the response to the stakes. A reversible, low-stakes risk (two-way door, Lesson 5) you just take, learn, adjust. A one-way, high-stakes risk (data loss, security, a public commitment) you de-risk deliberately: test it, stage it, get more eyes, have a rollback.
- Buy down risk cheaply where you can. A canary deploy, a feature flag, a load test, a dry run, these are small costs that convert a scary unknown into a measured known. Spend a little to avoid a lot.
- Accept some risk on purpose. Sometimes the right call is to knowingly take a risk because the upside is worth it and the downside is survivable. That’s fine, as a conscious, recorded decision (an ADR, Lesson 5), not as an accident. The goal isn’t zero risk; it’s no unexamined risk.
Good risk judgment is quiet, it mostly shows up as incidents that didn’t happen, which is invisible, like most of the best staff work. But it’s one of the things people most rely on a senior engineer for: the calm, calibrated sense of “this one we can just try, that one we need to be careful about.”
Pitfalls to Avoid
- Panicking, or trying to be the hero debugger. Your job is to lead a calm, coordinated response; let others find the bug.
- Diagnosing before stabilising. Stop the bleeding first (roll back, failover), then investigate.
- Going silent during an incident. Communicate on a clock, even “still investigating”, or panic and pings fill the void.
- Changing many things at once. One change at a time, or you can’t tell what worked or write a real postmortem.
- Blameful postmortems. Blame drives failures into hiding; fix the system, not the person.
- Postmortems with no owned action items. A review that changes nothing was theatre; produce a few real, owned fixes.
- Trying to eliminate risk. Manage it as visible, conscious bets; match your response to the stakes.
Quick Recap
- In an incident, the senior engineer sets the emotional temperature; be the calm. Establish roles, stabilise before diagnosing, communicate on a clock, one change at a time, protect responders, write a live timeline.
- You don’t have to find the bug; leading a calm response is the staff skill.
- Run blameless postmortems: assume reasonable action, fix the system not the person, because blame hides failures.
- Model blamelessness on yourself; produce a few concrete, owned, dated action items and track them to done.
- Manage risk as a portfolio of visible, conscious bets; de-risk the high-stakes ones cheaply, accept some risk on purpose.
Next lesson, the last: putting it all together into your own path, getting to staff, staying effective, and a concrete 90-day plan.