Every engineer who’s run software in production knows the feeling: an alert fires, or worse, a customer complains, and you’re staring at a system that was working, with no obvious reason why it isn’t now. How quickly you get from “something’s wrong” to “here’s the fix” is largely decided before the incident ever happens — by whether your system was built to be understood. That property is observability, and this first lesson is about what it means and why it’s different from the monitoring you may already have.
Monitoring vs Observability
These words get used interchangeably, but the distinction is real and it shapes how you build.
Monitoring watches for failure modes you anticipated. You decide in advance what could go wrong — the server could run out of memory, the error rate could spike, the site could go down — and you set up dashboards and alerts to watch those specific things. Monitoring answers known-unknowns: you know the question (“is CPU high?”), you just don’t know the answer until you look. It’s excellent at telling you “is it broken?” and it’s necessary — but it only catches the failures you thought to watch for.
Observability is the ability to explore questions you didn’t anticipate, by interrogating rich telemetry your system emits. The classic test: can you answer a question like “why is this specific user’s checkout slow, only on mobile, only since 2 PM?” — a question nobody wrote a dashboard for — by slicing and drilling into your data? Observability answers unknown-unknowns: you didn’t even know to ask the question until the weird failure appeared. It’s what lets you answer “why is it broken?”
The relationship is complementary, not competitive. Monitoring tells you something is wrong (an alert fires); observability lets you find out why (you drill into logs, metrics and traces to explain a failure you never pre-planned for). You need both. A system with great monitoring and poor observability pages you reliably and then leaves you guessing.
Why This Matters More Than It Used To
You could get away with just monitoring when systems were simpler — one big application on a few servers, with a handful of well-understood failure modes you could enumerate and watch. Modern systems broke that:
- Microservices and distributed systems fail in the gaps between components, in combinations, in ways that are genuinely novel. A request might traverse ten services; the failure is an emergent interaction you couldn’t have predicted.
- Many dependencies — databases, caches, queues, third-party APIs — each a potential source of a failure you didn’t anticipate.
- Scale and concurrency surface rare conditions (the tail, Lesson 11) that only appear under real production load, in patterns no staging environment reproduced.
In this world, you cannot pre-plan a dashboard for every way the system might fail, because you can’t enumerate them — there are too many, and the interesting ones are the ones you didn’t foresee. So the strategy shifts: instead of watching a fixed set of known signals, you instrument richly (emit detailed logs, metrics and traces — the three pillars, Lesson 2) and build the ability to explore that telemetry to answer whatever question the current failure demands. You trade “watch these specific things” for “be able to ask anything.”
The Core Truth
Everything in this course follows from one principle, worth stating plainly: you can’t debug what you didn’t make observable. If your system doesn’t emit the signal you need — if there’s no log at the point that failed, no metric for the thing that’s saturated, no trace to show where the time went — then when the novel failure arrives, the data simply isn’t there, and you’re reduced to guessing, adding logging, and redeploying while users suffer. Observability is a property you build in ahead of time, not something you can bolt on in the middle of an incident.
So the work has two halves, and this course covers both:
- Instrument for understanding (Lessons 2–6): emit the logs, metrics and traces — tied together — that let you ask questions later. This is an up-front investment, paid back every incident.
- Use what you see to debug fast (Lessons 7–12): read dashboards, respond to alerts, define SLOs, and run a disciplined debugging workflow when it matters.
A useful mindset shift to end on: treat “can I understand this when it breaks?” as a first-class requirement of every service you build, alongside “does it work?” and “is it fast?“. The service that’s easy to observe is the one you’ll fix in minutes at 2 AM; the opaque one is the one that ruins your night and your users’ day. Observability isn’t overhead on top of the real engineering — for anything you run in production, it is part of the real engineering.
Pitfalls to Avoid
- Treating monitoring as enough. It only catches failures you anticipated. Modern systems fail in novel ways; you also need observability to explore the unexpected.
- Bolting on observability during an incident. If the signal isn’t already emitted, it’s too late. Instrument before you need it — it’s a build-time property.
- Confusing “is it broken?” with “why?” Monitoring answers the first; observability the second. Build for both.
- Enumerating failure modes in complex systems. You can’t list them all. Instrument richly and build the ability to explore, rather than pre-planning a dashboard per failure.
- Only instrumenting the happy path. The failures you need to see are in the error and edge cases. Make those observable especially.
- Thinking observability is a tool you buy. Tools help, but observability is a property of your code — whether it emits what you’ll need. The tool can’t show data you never sent.
Quick Recap
- Monitoring watches anticipated failures (known-unknowns, “is it broken?”); observability lets you explore unanticipated ones (unknown-unknowns, “why is it broken?”). You need both.
- Monitoring tells you something is wrong; observability lets you find out why by drilling into logs, metrics and traces.
- Modern distributed systems fail in novel ways you can’t pre-plan dashboards for, so you instrument richly and explore rather than watching a fixed list.
- The core truth: you can’t debug what you didn’t make observable — it’s a build-time property, not an incident-time fix.
- Treat “can I understand this when it breaks?” as a first-class requirement of every service.
Next lesson: the raw materials of observability — the three pillars (logs, metrics, traces) — what each is, when each helps, and how they work together to take you from “something’s wrong” to the fix.