Distributed Systems in Practice — Failures, Time, Consensus and Correctness
What really happens when your code runs on more than one machine. Timeouts and retries, idempotency, clocks, replication and quorums, consistency models, Raft, consistent hashing, distributed locks, delivery guarantees, the outbox pattern and cascading failures. Every failure is reproduced on a real etcd cluster, real Redis, or a simulation you can run.
Every backend engineer today builds distributed systems, whether they call it that or not. The moment your app talks to a database over the network, calls a payment gateway, publishes to a queue, or runs on two pods behind a load balancer, you are in distributed systems land.
And distributed systems fail in strange ways. A payment shows “failed” but the money is debited. A user sees their old address after updating it. A lock that was supposed to allow only one worker allows two. A five-second database slowdown turns into a 30-minute outage. None of these is a bug in a single line of code. They come from the gaps between machines.
Most material on this topic is either very academic (papers and proofs) or very hand-wavy (“just use Kafka”). This course sits in between. I explain each idea in simple language, then break a real system to show it: I kill an etcd leader and time the failover, pause a Redis lock holder until two workers both “own” the lock, crash a consumer before it acknowledges a message and count the double credits. You can run every experiment on your laptop.
What This Course Covers
| # | Lesson | What you will learn |
|---|---|---|
| 1 | Why Distributed Systems Fail Differently | Partial failure, the 8 fallacies, tail latency, and setting up the lab |
| 2 | Timeouts, Retries and Idempotency | Why a timeout means “unknown”, backoff with jitter, idempotency keys |
| 3 | Time, Clocks and Ordering | Clock skew, last-write-wins data loss, Lamport and vector clocks |
| 4 | Replication and Quorums | Leader, multi-leader and leaderless replication; N, R, W in practice |
| 5 | Consistency Models, CAP and PACELC | Linearizable to eventual, in plain words, with a real etcd test |
| 6 | Consensus With Raft | Elections, log replication, and what a leader failure looks like |
| 7 | Partitioning and Consistent Hashing | Sharding schemes, rebalancing, hot keys |
| 8 | Distributed Locks, Leases and Fencing Tokens | Why a Redis lock is not enough, and how to make it safe |
| 9 | Messaging and Delivery Guarantees | At-most-once, at-least-once, “exactly-once”, ordering |
| 10 | Transactions Across Services | 2PC, sagas, and the transactional outbox |
| 11 | Cascading Failures and Resilience | Queueing, retry storms, circuit breakers, load shedding |
| 12 | Observability + Capstone | Tracing, debugging a distributed incident, a design checklist |
How This Course Is Different
The site already has a System Design Masterclass that walks through designing systems like URL shorteners and news feeds. This course goes one level deeper: why the building blocks behave the way they do, and what goes wrong when the network, the clock or a process doesn’t behave. If system design is about drawing the boxes, this course is about the arrows between them.
Who Is This For?
- Backend engineers working with microservices, queues, caches and replicated databases
- Engineers who have seen a “this should never happen” bug in production and want to understand it
- Anyone preparing for senior or staff-level interviews, where these topics come up a lot
Prerequisites
- Comfortable with one backend language. Code examples are in Python because it is short and readable, but the ideas apply everywhere
- Basic idea of HTTP APIs, databases and queues
- For the experiments: a Linux or Mac machine with etcd, Redis and Python 3 installed (setup is in Lesson 1)
Let’s start with the question that explains almost everything else in this course: what changes when a function call becomes a network call?