Consensus With Raft

I once watched a team spend two months building “a simple leader election” for their job scheduler. Each instance wrote its hostname to a database row with a timestamp, and whoever had the freshest timestamp was the leader. It worked perfectly in testing.

In production, a long GC pause on the leader meant its timestamp went stale, so a second instance took over. Then the first one woke up, still believing it was the leader, and both scheduled the same jobs for about 40 seconds. Customers got duplicate SMS reminders. They patched it. Then a network blip produced the same thing a different way. After the third incident, they replaced the whole thing with a 20-line etcd lease and never had the problem again.

Agreeing on something across machines, when any of them can pause, crash or get cut off, is a famously hard problem. It’s called consensus. The good news is that it’s been solved, and you almost never need to solve it yourself. But you do need to understand how the solution behaves, because its behaviour shows up in your system: in Kubernetes, in your database failovers, in your locks.

What Consensus Gives You

Consensus means getting a group of nodes to agree on a value (or a sequence of values) such that:

• Once a value is decided, it never changes.
• All non-failed nodes eventually learn the same decided value.
• The system keeps working as long as a majority of nodes are up and can talk to each other.

If you can agree on one value, you can agree on a sequence (a log), and with a log you can build almost anything:

• Leader election: “who is the leader for term 7?”
• Distributed locks and leases: “who holds the lock on job X?”
• Configuration and service discovery: “what is the current list of shards and their owners?” (Kubernetes stores all cluster state in etcd for this reason.)
• Replicated databases: replicate the log of writes, and every replica applies the same writes in the same order. CockroachDB, TiKV, YugabyteDB and Kafka’s KRaft mode all do this.

Paxos (1989) was the first widely known solution, and it’s notoriously hard to understand. Raft (2014) was designed specifically to be understandable, and it’s now the most common choice. Let’s look at how it works.

Raft: Roles and Terms

Every node in a Raft cluster is in one of three states:

• Follower: the normal state. Accepts log entries from the leader and votes in elections.
• Candidate: a follower that hasn’t heard from a leader for a while and is trying to become one.
• Leader: handles all client writes and sends regular heartbeats to followers.

Time is divided into terms, numbered 1, 2, 3 and so on. Each term starts with an election, and has at most one leader. Terms work like a logical clock (Lesson 3): every message carries the sender’s term, and if a node sees a higher term than its own, it updates and, if it was a leader or candidate, steps back to follower. Stale leaders discover they’re stale this way.

Elections

1. A follower that doesn’t receive a heartbeat within its election timeout becomes a candidate. In etcd the defaults are a 100 ms heartbeat and a 1,000 ms election timeout.
2. It increments the term, votes for itself, and asks the others for votes.
3. Each node votes for at most one candidate per term, first come first served, and only if the candidate’s log is at least as up to date as its own.
4. A candidate that gets votes from a majority becomes the leader and immediately starts sending heartbeats.
5. If the vote is split (two candidates at once, neither has a majority), they time out and try again. Raft randomises election timeouts slightly, so this usually resolves in one more round.

The majority rule is the heart of it. With 3 nodes, a majority is 2. With 5, it’s 3. Any two majorities share at least one node, and that node only votes once per term, so two leaders can never be elected in the same term. Split brain is impossible by design.

Log Replication and Commit

Once there’s a leader, every write goes through it:

1. The leader appends the command to its own log.
2. It sends the new entry to all followers (piggybacked on heartbeats or sent immediately).
3. When a majority (including itself) has stored the entry, it’s committed.
4. The leader applies the committed entry to its state (the key-value store, for etcd) and replies “OK” to the client. Followers apply it when they learn the new commit position.

Two rules make this safe:

• Voters refuse candidates with older logs. A node only votes for a candidate whose log is at least as complete as its own. Since a committed entry is on a majority, and a winning candidate needs votes from a majority, every possible new leader already has every committed entry. Committed writes are never lost, whatever minority of nodes fails.
• Followers’ logs are forced to match the leader’s. If a follower has extra entries from an old leader that never got committed, the new leader overwrites them. This is where Lesson 2’s “timeout means unknown” comes from: a write that reached the old leader but wasn’t committed before it died may simply disappear. One that was committed is permanent, even if the client never got the “OK”.

Real Test: Killing the Leader

Theory is nice, but I wanted to see what a client actually experiences. I ran a small Python client that wrote a counter to the 3-node etcd cluster every ~10 ms, trying each endpoint in turn if one failed, with a 0.5-second timeout per attempt. After two seconds, the script found the leader and killed it with SIGKILL: no graceful shutdown, like a machine losing power.

leader before: http://127.0.0.1:32379 term 2
killed leader pid 14762
new leader: ('http://127.0.0.1:22379', '3', 1.21 s)
writes before kill: 135  all ok: True
failed writes after kill: 1
first successful write after kill at +1.014 s
normal write p50 4.1 ms  p99 8.6 ms

What happened:

• For about one second, no write could succeed. The followers were waiting for heartbeats that would never come. That’s the election timeout.
• Then one follower became a candidate in term 3, got the other surviving node’s vote (2 of 3 is still a majority), and became leader.
• Writes resumed at +1.01 s. (My status check, which polled every 20 ms, saw the new leader a little later, at +1.21 s.)
• Not a single committed write was lost, and the client only saw one failed attempt, because it retried on the next endpoint.

This is what failover looks like in Kubernetes, Consul or any Raft system: a pause of roughly one election timeout, then business as usual. Your application needs to ride through such pauses with timeouts and idempotent retries, and your alerts shouldn’t page anyone for a single one-second blip.

Then I tested quorum loss: with only one of the three nodes running, every write and every linearizable read timed out (you saw this in Lesson 5). A Raft cluster stops accepting writes rather than risk two sides accepting conflicting ones. That’s the “C” choice from CAP.

Real Test: The Lonely Node Problem (PreVote)

This one I found by accident. During an earlier experiment, I paused the wrong process (my own shell, as I mentioned in Lesson 1), which left one etcd node alone for about two minutes while the other two were unreachable. When things came back, its log showed:

b71f75320dc06a6c is starting a new election at term 3
b71f75320dc06a6c became candidate at term 4
...  (89 elections in total)
b71f75320dc06a6c became leader at term 92

The isolated node kept timing out, starting elections, and bumping its term: 89 elections, term 3 to 92. Nobody could vote for it, so it never won while isolated. But the moment the old leader came back, it saw term 92, which was higher than its own term 3, so it stepped down, and the lonely node won the next election.

I then reproduced this on purpose. I cut one follower off from the other two for 10 seconds, then reconnected it. I ran it twice: with etcd 3.4’s default settings, then with --pre-vote=true:

PreVote off:  term 96 -> 103 after rejoin, leader CHANGED (the lonely node won)
PreVote on:   term 108 -> 108 after rejoin, leader unchanged

With PreVote, a node that wants to start an election first asks, “would you vote for me if I did?” The others say no, because they’re happily receiving heartbeats from a working leader. So the isolated node never bumps its term, and its return causes no disruption.

Why this matters in real life: a single node with a flapping network link (up, down, up, down) can trigger a leader change every time it comes back. Each change pauses writes for about a second. In a Kubernetes control plane, that shows up as random API server slowness that’s very hard to trace. etcd 3.5 and later turn PreVote on by default, but plenty of clusters still run older versions or other Raft implementations. It’s worth checking.

Cluster Size: Why 3 or 5

A Raft cluster of N nodes needs a majority to work, so it tolerates (N - 1) / 2 failures (rounded down):

Notice that 4 nodes tolerate no more failures than 3, and need more nodes to agree on every write. Even numbers are a waste. And more nodes isn’t automatically better: every write must reach a majority, so a 7-node cluster is slower than a 5-node one.

The usual setup: 3 nodes for most uses, 5 nodes when you want to survive two failures, or one failure during planned maintenance. Spread them across 3 availability zones. And remember that consensus systems are sensitive to disk latency (every log entry is fsynced before it counts) and network latency (every write needs a round trip to a majority). Put etcd on fast SSDs, don’t stretch a cluster across distant regions, and watch etcd’s own fsync and leader-change metrics.

Use It, Don’t Build It

Consensus algorithms are short to describe and very hard to implement correctly. The papers leave out many practical details (membership changes, snapshots, log compaction, read optimisations), and the bugs only show up under rare timing. Even mature implementations have had subtle bugs found years later by tools like Jepsen.

So, in practice:

• Need leader election, locks, or small critical config? Use etcd, ZooKeeper or Consul. Their client libraries have leader election and lease APIs ready to use. In Kubernetes, use the built-in Lease objects (that’s how controllers elect leaders).
• Need a replicated database? Use one built on consensus (CockroachDB, YugabyteDB, TiDB, Spanner) or a single-leader database with a consensus-backed failover tool (Patroni for Postgres).
• Don’t store large data or high-volume writes in etcd or ZooKeeper. They’re for small, critical coordination data, a few GB at most.

Leader election with an etcd lease looks roughly like this (with the python-etcd3 style API):

import etcd3

client = etcd3.client(host="127.0.0.1", port=2379)
lease = client.lease(ttl=10)                          # dies if we stop refreshing it

won, _ = client.transaction(
    compare=[client.transactions.create("/scheduler/leader") == 0],   # key doesn't exist
    success=[client.transactions.put("/scheduler/leader", "pod-7", lease)],
    failure=[],
)
if won:
    run_as_leader(refresh=lease.refresh)              # must keep refreshing within 10 s

Even this has a catch: a leader that pauses longer than its lease is no longer the leader, but doesn’t know it yet. That’s exactly the problem from the story at the top, and Lesson 8 shows how fencing tokens solve it.

Pitfalls to Avoid

1. Building your own leader election with timestamps in a database row. Use a consensus system.
2. Even-sized clusters. 4 nodes are no more fault-tolerant than 3.
3. Stretching a Raft cluster across far-apart regions. Every write waits for a majority round trip.
4. Slow disks under etcd/ZooKeeper. fsync latency becomes your write latency and can cause false elections.
5. Treating a one-second leader election as an outage. Design clients to ride through it.
6. Running without PreVote (or an equivalent) where a flapping node can disrupt the cluster.
7. Putting big or high-volume data in a coordination store.
8. Assuming “I am the leader” stays true without checking. Leadership can be lost during a pause.

Quick Recap

• Consensus lets nodes agree on values despite failures. It’s the basis for leader election, locks, config, and replicated databases.
• Raft: one leader per term, elected by a majority; entries are committed once a majority has them; committed entries are never lost.
• Real test: killing the etcd leader paused writes for about 1 second, the new leader was in place by 1.21 s, and nothing committed was lost.
• Without PreVote, a node that was cut off can force a leader change when it returns (term 96 to 103). With PreVote, no disruption.
• Use 3 or 5 nodes on fast disks, and use etcd, ZooKeeper or Consul instead of writing your own.

Next lesson: splitting data across machines. Hash vs range partitioning, why hash(key) % N moves 80% of your keys when you add a server, consistent hashing, and what to do about hot keys.