Alerting & On-Call

An alert is a promise to interrupt a human — to pull them out of dinner, out of sleep, out of focus. That makes alerting a place where good intentions (“let’s monitor everything!”) do real harm, because an avalanche of alerts trains people to ignore them, and then the one that mattered gets missed. This lesson is about alerting that respects the human on the other end: few, meaningful, actionable pages that people trust.

Alert on Symptoms, Not Causes

The single most important alerting principle:

Alert on symptoms that users feel, not internal causes. The difference:

• Causes (bad to page on): “CPU > 80%!” — so what? Maybe the service is happily busy and users are fine. “A pod restarted” — it probably self-healed. These fire constantly, often signify nothing actionable, and bury the real signals. Causes are for dashboards (Lesson 7), where a human investigating can use them — not for pages.
• Symptoms (good to page on): “checkout error rate > 2% for 5 minutes,” “p99 latency over the SLO,” “users can’t log in.” These mean users are actually hurting, which is the only thing worth waking someone for. They’re few, meaningful, and actionable.

The test: does this alert mean a user is having a bad time? If yes, page. If it’s just an internal metric that might lead to a problem (high CPU, a restart), put it on a dashboard and let the symptom-based alert fire if and when users are actually affected. A high CPU that hurts nobody should never wake anyone.

Alert Fatigue Is the Real Enemy

Here’s why this matters so much: alert fatigue is the failure mode that makes alerting worthless. When alerts fire constantly for things that don’t matter — every CPU blip, every restart, every transient — people learn to ignore the channel. They mute it. And then the one alert that was real gets missed, because it’s buried in noise nobody reads anymore. A noisy alert that gets ignored is worse than no alert at all, because it gives false comfort (“we have alerting!”) while actually hiding problems.

So treat your alert set as something to tune relentlessly:

• Every page should be actionable: when it fires, there’s something a human should do. If an alert fires and the response is “nothing to do, it recovered,” that alert shouldn’t page — downgrade it to a dashboard or delete it.
• Every page should have a runbook: a short “what is this, how do I investigate, what are the likely fixes.” The 2 AM on-call engineer shouldn’t have to figure out from scratch what a cryptic alert means. Link the runbook from the alert.
• Delete or fix flapping alerts — ones that fire and resolve repeatedly. They’re pure noise and they erode trust in the whole system.
• The goal is that when an alert fires, people believe it and act. That trust is the entire value of alerting, and it’s destroyed by noise.

Tie Alerts to SLOs and Burn Rate

The most refined approach (building on Lesson 9) is to alert on your SLO error budget burn rate rather than on instantaneous thresholds. Instead of “page if any request is slow” (noisy — single slow requests happen), you alert on how fast you’re consuming the month’s error budget:

• Burning the budget slowly? A ticket, maybe, not a page — you have time.
• Burning it fast (you’ll exhaust the whole month’s budget in hours)? Page immediately — this is a real, urgent problem.

This elegantly solves the “how do I set the threshold?” problem: the alert fires based on user-facing impact over time (the SLO, Lesson 9), not an arbitrary number, and it distinguishes “a minor blip” from “a genuine incident eating our reliability budget.” Burn-rate alerting is why Lesson 9’s SLOs aren’t just a planning exercise — they become the foundation of good alerting.

On-Call, Humanely

A few practices that make being on-call sustainable rather than miserable:

• Few, trustworthy pages. The combination of symptom-based + actionable + runbook-backed means on-call gets paged rarely, and every page is real. That’s a humane rotation.
• Severity levels. Not everything is a wake-you-at-3 AM page. Distinguish “page now” (users down) from “notify, handle in the morning” (degraded but okay). Match urgency to impact.
• Runbooks and blameless culture. When an alert fires, a runbook gets the responder moving fast; afterward, a blameless postmortem (Lesson 10) improves the system (and often the alert) rather than blaming a person.
• Review alerts regularly. Which alerts fired? Which were actionable? Which were noise? Prune and tune. An alert set is a living thing, not set-and-forget.

The overarching mindset: alerts are a cost (human interruption), spend them only where they buy something (preventing or shortening user pain). Every alert should earn its place by being a real, actionable signal that a user is affected. Keep that bar high and on-call stays trustworthy and humane; drop it and you drown your team in noise and miss the real problems.

Pitfalls to Avoid

1. Alerting on causes. CPU/memory/restarts often don’t mean user harm and create constant noise. Alert on symptoms users feel; put causes on dashboards.
2. Alert fatigue. A flood of alerts gets muted, and the real one is missed. A noisy ignored alert is worse than none. Keep pages few and meaningful.
3. Non-actionable pages. If there’s nothing to do, it’s not a page. Downgrade to a dashboard or delete it.
4. No runbook. The 2 AM responder shouldn’t decode a cryptic alert from scratch. Every page links a runbook.
5. Arbitrary thresholds. “Page if latency > 500ms” is noisy and ad hoc. Alert on SLO burn rate (Lesson 9) — user impact over time.
6. Set-and-forget alerts. Alert sets rot. Review regularly; delete flapping/noisy alerts; keep the human’s trust.

Quick Recap

• Alerts tell a human to look when they’re not watching — make them few, meaningful, actionable.
• Alert on symptoms (errors, latency over SLO, unavailability) that users feel — not on causes (CPU, restarts), which belong on dashboards.
• Alert fatigue is the real enemy: a noisy, ignored alert is worse than none. Every page must be actionable and have a runbook.
• Tie alerts to SLO burn rate (Lesson 9): page fast when you’re burning the error budget quickly, not on arbitrary thresholds.
• Tune relentlessly — delete flapping alerts, keep the human’s trust. On-call should be few, trustworthy pages.

Next lesson: the number that defines “reliable enough” and underpins good alerting — SLOs, SLIs, and error budgets.