Scaling Writes

In the previous lesson, reads scaled by making copies: a cache, a replica, a CDN edge — each copy answers reads independently. Writes do not get that luxury. A write has to be applied everywhere the data lives, so adding copies makes writes harder, not easier: now the write must reach the primary and propagate to every replica. This is why, in almost every system design, the write path is where the real scaling challenge lives.

This lesson is the toolbox for scaling writes. The through-line is a single uncomfortable fact: you cannot make writes faster by duplicating data, so you have to divide it instead.

In the interview: when write volume is high, the first thing to reach for is sharding, and the first question the interviewer will ask is “what is your shard key?” Have an answer ready, and be able to estimate the write QPS per shard after splitting.

The single-primary ceiling

Start with the simplest architecture: one primary database taking all writes. You can make it bigger — more CPU, more RAM, faster NVMe disks. This is vertical scaling, and it works right up until it doesn’t. A single well-tuned relational primary can often handle on the order of thousands to low tens of thousands of writes per second, depending on the workload. Beyond that, you hit hard physical limits: disk write bandwidth, the cost of maintaining indexes, lock contention, and the simple fact that one machine has a finite ceiling.

Read replicas do not help here — every write still goes to the one primary before fanning out to followers. So once sustained write throughput approaches the primary’s ceiling, there is only one real answer: stop putting all the data on one machine.

Sharding: divide the data across primaries

Sharding (horizontal partitioning) splits the dataset across multiple independent database nodes, each of which is the primary for its slice of the data. If you split across N shards, each shard receives roughly 1/N of the writes, and you add write capacity simply by adding shards. This is the fundamental, and essentially the only, way to scale writes past a single machine.

A routing layer sits in front of the shards. For each write, it computes which shard owns the key — typically hash(shard_key) % N — and sends the write there. Each shard can also have its own read replicas, so sharding and the read-scaling techniques from the previous lesson compose cleanly.

Choosing the shard key

The shard key is the most consequential decision in a sharded system, and interviewers will push on it. A good shard key spreads both data and traffic evenly across shards and keeps related data that is queried together on the same shard. Common choices: user_id for a social app (a user’s data lives on one shard), tenant_id for a B2B SaaS, or a composite key.

Hash sharding vs range sharding

Hash sharding:  shard = hash(key) % N
  + Even distribution, no natural hotspots
  − No efficient range scans (keys are scattered)
  − Resharding moves almost everything (unless consistent hashing)

Range sharding:  shard = lookup(key) in a range map   (A–H → s0, I–P → s1 ...)
  + Efficient range scans ("all orders from last week")
  − Hotspots: sequential or trendy keys pile onto one shard
  + Resharding can split a hot range without touching others

• Hash sharding distributes keys evenly and avoids hotspots, at the cost of losing range queries — “give me all events between two timestamps” becomes a scatter-gather across every shard.
• Range sharding keeps adjacent keys together so range scans are efficient, but risks hotspots: if you range-shard by timestamp, all new writes land on the newest shard while the others sit idle.

Pick based on your dominant access pattern. If you mostly look up by exact key, hash. If you frequently scan ranges, range — but then watch for hotspots.

The resharding problem

When N shards fill up and you need to add more, naive hash(key) % N is a nightmare: changing N remaps almost every key, forcing a massive data migration. Consistent hashing solves this — it maps keys and nodes onto a ring so that adding or removing a shard moves only ~1/N of the keys instead of nearly all of them. (We go deep on consistent hashing in the distributed-cache and key-value-store design lessons.) Many systems also pre-split into many more logical shards than physical nodes, so growing capacity means moving whole shards rather than rehashing keys.

Hot partitions

Even with a good hash, one key can be disproportionately popular — a celebrity account, a viral product, a trending hashtag — and overload the single shard that owns it. This is the hot-partition or celebrity problem, and hashing alone cannot fix it because the hot key always lands on one shard. The fixes (replicating the hot key, caching it, giving it dedicated capacity, or splitting it) get a full treatment in the Scaling Pitfalls lesson; for now, just flag that you know a single hot key can defeat sharding.

Buffering writes with a log

Not all write pressure is sustained — much of it is bursty. A flash sale, a viral post, a notification fan-out, or a batch job can produce a spike far above the steady-state rate. If you point that spike directly at the database, it falls over. The fix is to put a durable buffer in front of the database.

An append-only log (Kafka) or a message queue absorbs the burst: producers append writes at whatever rate they arrive, and a pool of workers drains the log into the database at a rate the database can actually sustain. The log flattens the spike into a steady stream. This pattern buys you several things at once:

• Spike absorption — the log holds the backlog so the database is never overwhelmed; it just processes a little behind during the peak.
• Batching — workers can group many small writes into fewer, larger, sequential writes. Sequential writes are dramatically faster than scattered random writes, so batching can multiply effective throughput.
• Decoupling and durability — if a worker or the database is briefly down, writes sit safely in the log and are processed when it recovers; nothing is lost.

The trade-off is that writes become asynchronous: the client is told “accepted” when the write hits the log, not when it is durably in the database. For most high-volume writes (events, metrics, feed fan-out, analytics) that is exactly the right trade. For writes that must be read back immediately and consistently, keep them synchronous.

Write-optimized storage

Sometimes the bottleneck is the storage engine itself. The two dominant families behave very differently under heavy writes:

• B-tree engines (traditional relational databases) update data in place. Every write may touch a random location on disk and rebalance the tree, which is read-friendly but makes heavy random writes expensive.
• LSM-tree engines (Cassandra, RocksDB, ScyllaDB, LevelDB) buffer writes in memory and flush them to disk as large, sequential, immutable files, compacting in the background. Because writes are always appended sequentially, LSM engines absorb very high write throughput — at the cost of more work at read time and background compaction.

If a design is dominated by writes (time-series data, event logs, a write-heavy messaging store), reaching for an LSM-based store like Cassandra is a strong, defensible choice. Say why: “it’s write-heavy, so I want an LSM engine that turns writes into sequential appends.”

High-contention counters

A special case worth knowing: a single counter that everyone increments (likes on a viral post, views on a trending video) serializes all those writes onto one row and becomes a hotspot. Two standard fixes:

• Sharded counters — split the counter into K sub-counters on different shards; increment a random one and sum them on read. This spreads the write load K-fold.
• Write-back / aggregation — buffer increments in memory (or a cache) and flush the aggregated total to the database periodically, turning thousands of writes into one.

The consistency cost

Everything in this lesson distributes writes across more machines — more shards, more async buffers, more replicas receiving propagated writes. Every one of those steps weakens consistency. Once a write might be in the log but not yet in the database, or on the primary but not yet on a replica, or on shard A while a related write is on shard B, you can no longer assume every reader sees the latest, fully-consistent state. Cross-shard transactions are especially painful — there is no cheap global lock across independent databases.

That trade-off — throughput and availability versus consistency — is the heart of distributed systems, and it is the subject of the next lesson on replication and consistency. For now, the lesson is: scaling writes is not free. Every technique here buys write capacity by giving up some guarantee you had on a single machine, and a strong design names which guarantees it is trading away and why that is acceptable for this workload.

Key Takeaways

1. You cannot scale writes by copying — you must divide. Adding replicas makes writes harder, not easier. The real lever is sharding the data across independent primaries so each takes 1/N of the writes.

2. The shard key is the critical decision. It must spread data and traffic evenly and keep co-queried data together. Be ready to name it and estimate per-shard write QPS.

3. Hash sharding for even load, range sharding for range scans. Hash avoids hotspots but loses range queries; range enables scans but risks piling trendy keys onto one shard. Use consistent hashing to make resharding cheap.

4. Buffer bursty writes behind a log. An append-only log (Kafka) absorbs spikes, lets workers batch small writes into large sequential ones, and decouples producers from the database — at the cost of asynchronous, eventually-durable writes.

5. Match the storage engine to the workload. LSM-tree stores (Cassandra, RocksDB) turn writes into sequential appends and absorb far higher write throughput than in-place B-trees. For single hot counters, use sharded counters or write-back aggregation.

6. Distributing writes costs consistency. Every write-scaling technique trades a single-machine guarantee for throughput or availability. Name the guarantee you are giving up — it leads directly into replication and consistency.