Replication and Consistency

Almost every scalable system keeps more than one copy of its data. The moment you have two copies, you have a new problem: how do you keep them in sync, and what does a reader see while they’re diverging? This is the heart of replication and consistency — the single topic that separates candidates who can recite components from candidates who actually understand distributed systems.

This lesson gives you the vocabulary and the decision framework to reason about copies of data under failure. You’ll use it in every design problem that follows.

Why replicate at all

Keeping a single copy of your data is simpler, so you should know exactly why you’re giving that simplicity up. There are four reasons, and in an interview you should name the one that applies:

• Availability. If the one machine holding your data dies, the data is unreachable. Replicas let another machine take over.
• Read scaling. A read-heavy workload (most are) can spread reads across many replicas instead of hammering one node. This is the most common reason to replicate.
• Durability. More physical copies on more machines means a single disk failure, rack failure, or data-centre outage doesn’t lose your data.
• Geo-locality. A replica in Mumbai serves Indian users with 10 ms latency instead of 200 ms across an ocean.

Each of these pulls in a slightly different direction, which is why there isn’t one replication strategy — there are several, and they trade off differently.

Leader–follower (single-leader) replication

The default and most widely deployed model. One replica is the leader (also called primary or master); the rest are followers (replicas or secondaries).

• All writes go to the leader.
• The leader applies the write, then streams the change to its followers.
• Reads can be served by the leader or any follower.

This is what PostgreSQL, MySQL, MongoDB, and most managed databases do out of the box. It’s popular because writes have a single, unambiguous ordering — there are no write conflicts, because only one node accepts writes.

The critical question is when the leader considers a write “done.”

Synchronous vs asynchronous

Synchronous replication — the leader waits for at least one follower to acknowledge the write before confirming success to the client. If the leader then dies, that follower is guaranteed to have the data, so failover loses nothing. The cost: the write is only as fast as the slowest follower you wait for, and if that follower is down or slow, writes stall.

Asynchronous replication — the leader commits locally and confirms to the client immediately, then ships the change to followers in the background. Writes are fast and a slow follower can’t block them. The risk: if the leader crashes before a change reaches any follower, that acknowledged write is lost on failover. For many systems (social feeds, analytics) that’s an acceptable, rare event; for others (payments) it is not.

Semi-synchronous is the common middle ground: wait for one follower synchronously and let the rest replicate asynchronously. You get durability against a single-node failure without waiting for every replica.

Synchronous:   client → leader → (wait for follower ack) → confirm
               safe on failover, slower, a slow follower blocks writes

Asynchronous:  client → leader → confirm → (ship to followers later)
               fast, but a leader crash can lose the last few writes

Replication lag is the gap between a write landing on the leader and showing up on a follower — usually milliseconds, but it can spike to seconds under load. Lag is the source of most “I just saved it but I don’t see it” bugs, which brings us to consistency.

Handling replication lag in practice

Lag is why asynchronous read replicas, left naive, produce confusing bugs. Three standard techniques fix the common cases without paying for strong consistency everywhere:

Read-your-writes:  after a user writes, route THEIR reads to the leader
                   (or the replica known to have the write) for a few seconds.

Monotonic reads:   pin a given user to one replica (sticky routing), so they
                   never bounce to a more-stale replica and see time go backwards.

Bounded staleness: only serve reads from replicas whose lag is under a threshold
                   (e.g. < 1s); route the rest to the leader.

These give each user a coherent experience even while the global system is only eventually consistent — the sweet spot most products actually want.

Multi-leader and leaderless

Two other models show up, usually in more advanced designs:

Multi-leader — more than one node accepts writes (often one leader per region for low-latency local writes). The hard part is that two leaders can accept conflicting writes to the same key at the same time, so you need conflict resolution. Use it only when you genuinely need multi-region write availability.

Leaderless (Dynamo-style) — there is no leader; the client (or a coordinator) writes to several replicas directly and reads from several replicas. This is how DynamoDB and Cassandra work, and it’s built around quorums (below). It trades the simplicity of a single write ordering for high availability and tunable consistency.

Both multi-leader and leaderless buy availability at the price of dealing with conflicting writes — a cost single-leader systems never pay.

The consistency spectrum

“Consistency” isn’t on/off. It’s a spectrum of guarantees about what a reader can observe, from strongest (and most expensive) to weakest (and cheapest).

• Strong / linearizable — every read returns the most recent write, as if there were a single copy. Example: your bank balance must reflect the withdrawal that just completed. Expensive: it usually requires coordination (reading from the leader or a quorum).
• Read-your-writes — a user always sees their own writes, even if others see them later. Example: after you post a comment, you see it immediately; a friend might see it a second later. Often achieved by routing a user’s reads to the leader for a short window after they write.
• Monotonic reads — once you’ve seen a value, you never see an older value on a later read. Example: you refresh a page and the like-count never goes backwards. Achieved by pinning a user to one replica.
• Eventual — if writes stop, all replicas eventually converge to the same value, with no guarantee about when. Example: a follower count that’s briefly stale across replicas. Cheapest and most available.

In an interview, don’t say “we’ll make it consistent.” Say which guarantee each piece of data needs. A single product can span the whole spectrum: a chat app wants read-your-writes for your own messages, monotonic reads for a conversation, and eventual consistency for “last seen” presence.

Quorums: tuning consistency with W, R, and N

Leaderless and some replicated systems let you tune consistency with three numbers:

• N — how many replicas hold each piece of data.
• W — how many replicas must acknowledge a write for it to count as successful.
• R — how many replicas you read from and compare.

The key rule:

If W + R > N, the set of replicas you wrote to and the set you read from are guaranteed to overlap — so at least one replica in your read has the latest write. You get a strong read.

Worked example: N = 3, W = 2, R = 2. Then W + R = 4 > 3. A write succeeds once 2 of 3 replicas ack it; a read collects responses from 2 of 3 and takes the newest. Because any 2-of-3 write set and any 2-of-3 read set share at least one replica, the read always sees the latest value.

Tuning the knobs changes the trade-off:

Sloppy quorums with hinted handoff relax this during failures: if the “right” replicas are unreachable, writes land on other reachable nodes temporarily and are handed back later — favouring availability over strict overlap.

Resolving conflicts

Once more than one node can accept writes (multi-leader or leaderless), two writes can race. You need a rule to decide the winner:

• Last-write-wins (LWW) — attach a timestamp; the latest wins. Simple, but silently discards one of the writes and is sensitive to clock skew.
• Vector clocks — track causality so the system can tell a true conflict from a stale copy, flagging genuine conflicts for the application to resolve.
• CRDTs (conflict-free replicated data types) — data structures (counters, sets) that merge automatically without conflict, used for collaborative editing and shopping carts.

Two background mechanisms keep replicas honest: read-repair (when a read notices one replica is stale, it writes the fresh value back) and anti-entropy (a background process that continuously compares and reconciles replicas).

CAP and PACELC, in plain terms

The CAP theorem says that when a network partition splits your replicas, you must choose:

• CP (consistency) — refuse reads/writes that can’t be made consistent, so some requests fail, but you never return wrong data. (A bank ledger.)
• AP (availability) — keep serving every request, accepting that some replicas return stale data until the partition heals. (A social feed.)

There is no “CA” system in the real world, because partitions will happen — the only question is how you behave when they do.

PACELC completes the picture: during a Partition, choose Availability or Consistency; Else (normal operation), choose Latency or Consistency. Even with no partition, stronger consistency costs round-trips. This is why “strong everywhere” is the wrong default — it makes every read slower for data that didn’t need it.

In the interview: classify each data type up front. “Account balances need strong consistency, so I’ll read from the leader or a quorum. The activity feed can be eventually consistent, so I’ll serve it from read replicas and accept a second of lag.” That one sentence demonstrates more than any diagram.

Key Takeaways

1. Replicate for availability, read scaling, durability, or geo-locality — and say which one you’re after.
2. Single-leader is the default. Writes to the leader, reads from anywhere. Synchronous replication is safe but slow; asynchronous is fast but can lose the last few writes on failover.
3. Consistency is a spectrum, not a switch: strong → read-your-writes → monotonic reads → eventual. Pick per data type.
4. W + R > N gives strong reads in quorum systems. Tune W and R to trade write latency, read latency, and freshness.
5. Multi-leader and leaderless buy availability by paying for conflict resolution (LWW, vector clocks, CRDTs, read-repair).
6. CAP: during a partition, choose consistency or availability. PACELC: even without one, choose latency or consistency. State your choice explicitly and your interviewer will know you’ve done this before.