Consensus and Coordination

Replication told you how to keep copies of data in sync. This lesson answers the harder question underneath it: when machines disagree — because one crashed, or the network split in two — how does the system decide who is right? That is the consensus problem, and it is the foundation beneath leader election, distributed locks, and strongly consistent databases.

You will almost never implement consensus from scratch in an interview. But you must recognise when a design needs it, and name the tool that provides it.

The problem: split brain

Imagine a cluster with one leader that takes all the writes. The network partitions, cutting the leader off from half the cluster. The isolated half can’t reach the leader, assumes it died, and elects a new one. Now two leaders both accept writes. When the partition heals, their histories conflict and data is corrupted. This is split brain, and it is the nightmare consensus exists to prevent.

The naive fixes don’t work. “Just ping to check if the leader is alive” fails because you cannot distinguish a dead leader from a slow network — the classic impossibility at the heart of distributed systems. What you need is a rule by which the cluster can agree on exactly one leader even when some nodes are unreachable.

Raft: consensus you can actually follow

Raft is a consensus algorithm designed to be understandable (its predecessor, Paxos, is correct but famously hard to reason about). It has three moving parts.

1. Leader election. Nodes are followers by default. If a follower hears nothing from a leader for a random timeout, it becomes a candidate, increments the term number, and asks the others to vote for it. A node votes for at most one candidate per term, and a candidate becomes leader only if it wins votes from a majority. Because a majority of a fixed cluster can only be formed by one candidate at a time, there is never more than one leader per term — this is what defeats split brain. The isolated minority in a partition simply can’t elect a leader, so it stops accepting writes.

2. Log replication. All writes go to the leader, which appends them to an ordered log and streams each entry to the followers.

3. Quorum commit. An entry is committed only once a majority of nodes have stored it. So a 5-node cluster commits as soon as 3 nodes (including the leader) have the entry, and can tolerate 2 failures while staying available. This is why consensus clusters are odd-sized (3 or 5): an odd number gives a clear majority and the best failure tolerance per node.

5-node Raft cluster:
  majority = 3   → tolerates 2 failures
  partition 3 | 2 → only the side with 3 can elect a leader and commit

The result is a single, consistent, replicated log — the primitive that strongly consistent databases, queues, and lock services are built on.

Coordination services: what you actually use

You don’t build Raft into your application. You run a coordination service that has already solved consensus and exposes it as simple primitives. The standard choices are ZooKeeper, etcd (used by Kubernetes), and Consul.

These services give you four things that come up constantly in design problems:

• Leader election. When exactly one instance of a service should be active (a single writer, a scheduler, a primary), the instances race to create a key in the coordination store; the winner is the leader, and the others watch for it to disappear.
• Distributed locks. When only one worker should process a resource at a time, it acquires a lock. Two details separate a correct answer from a broken one:
  • a TTL / lease so that if the holder crashes, the lock is automatically released and the system doesn’t deadlock;
  • a fencing token — a monotonically increasing number handed out with the lock — so that a stale holder (one that paused, lost its lease, but woke up thinking it still held the lock) is rejected by the resource, which only accepts the highest token it has seen.
• Configuration. A consistent, watchable place to store feature flags and settings that every node reads and is notified when they change.
• Service discovery and membership. Which nodes are currently alive and where to reach them.

Correct distributed lock:
  acquire(lock, ttl=30s) → get fencing token N
  do work, tagging each write with N
  resource rejects any write with a token < the max it has seen

The cost

Consensus is not free, and you should say so. Every committed operation needs a quorum round trip, which adds latency, so you don’t route high-throughput data-plane traffic through a coordination service. You use it for the control plane — the relatively rare, must-be-correct decisions about who is in charge. And you run an odd-sized cluster (usually 3 or 5 nodes) to get a clean majority.

Interview tip: Reach for a coordination service the moment your design needs exactly one of something — one leader, one writer, one owner of a shard, one job runner. Saying “I’ll use ZooKeeper for leader election so only one scheduler is active, with a lease so a crashed leader is replaced automatically” signals that you understand coordination, not just storage.

Key Takeaways

1. Consensus prevents split brain — it lets a cluster agree on one value (or one leader) even when nodes crash or the network partitions.
2. Raft does this with leader election, an ordered replicated log, and quorum commit: a majority must ack, so there is never more than one leader per term. Clusters are odd-sized for a clean majority.
3. You use a coordination service, not hand-rolled consensus — ZooKeeper, etcd, or Consul for leader election, locks, config, and discovery.
4. A correct distributed lock needs a TTL and a fencing token so a crashed or stale holder can’t deadlock or corrupt the resource.
5. Consensus is for the control plane. It costs a quorum round trip, so use it for the rare “exactly one owner” decisions, not high-volume data traffic.