Scaling Pitfalls: Hotspots, Stampedes and Failure

A design that looks correct on the whiteboard can still collapse the first time real traffic hits it. The gap between “it works” and “it works at scale” is a short list of failure modes that recur in almost every system. Senior engineers don’t just draw the happy path — they point at their own design and say “here’s where this breaks, and here’s how I’d stop it.”

This lesson is that list. Learn to spot these five pitfalls and to propose their standard mitigations, and you’ll handle the “what happens when this gets popular?” follow-up that interviewers love.

Hot keys: the celebrity problem

You’ve sharded your data by key, so load spreads evenly across nodes — in theory. In practice, access is almost never uniform. One key gets disproportionately more traffic than all the others: a celebrity with 100 million followers, a flash-sale product, a viral video, a single trending hashtag.

Because that key lives on one shard, that shard melts while the others sit idle. Consistent hashing doesn’t save you — it distributes keys evenly, not load, and all the load for this key maps to one place.

Mitigations, usually in combination:

• Replicate the hot key across many nodes and spread reads over the copies. Reads scale with the number of replicas.
• Cache it aggressively at the edge (CDN) and in an in-memory cache, so most requests never reach the shard at all.
• Give it dedicated capacity — detect hot keys at runtime and move them to their own well-provisioned nodes.
• Split the key. For a hot counter (likes on a viral post), keep N sub-counters on different shards and sum them on read. The write load fans out N-ways; the read pays a cheap fan-in.

Worked example — the split counter. A viral post gets 50,000 likes per second. A single likes:post123 counter row can’t take 50K writes/sec; the shard holding it saturates. Split it into 100 sub-counters on 100 shards: each takes ~500 writes/sec, which is trivial. The displayed total is SUM(likes:post123:0 .. likes:post123:99) — one cheap fan-in read, often itself cached for a second. The write load fanned out 100×; the read cost barely moved.

Interview tip: When you shard anything, immediately say “the risk here is a hot key — a celebrity user — so I’ll replicate and cache the hot entries.” Unprompted, this is a strong signal.

Cache stampede (thundering herd)

Caching is the highest-leverage optimization in system design — and its most famous failure. A popular key is cached with a TTL. The TTL expires. In the microseconds before anything repopulates the cache, thousands of concurrent requests all miss at once and stampede straight to the database. The database, which was comfortably serving one query’s worth of load through the cache, suddenly takes thousands of identical queries and falls over. The outage then prevents the cache from being repopulated, so it keeps happening.

The mitigations stack well together:

• Single-flight (request coalescing). When the key is missing, let exactly one request rebuild it (hold a short lock); every other request waits for that result or briefly serves stale. The database sees one query instead of thousands.
• Early / probabilistic recomputation. Refresh the entry slightly before it expires, so it’s never actually absent during high traffic.
• Serve stale while revalidating. Return the expired value immediately and refresh in the background. Users get a slightly old answer instead of an outage.
• Jittered TTLs. If you populate many keys at once (a deploy, a cache warm), don’t give them all the same expiry — add random jitter so they don’t all expire in the same instant and stampede together.

Cascading failures and retry storms

Systems are chains of dependencies. When one link slows down, naive retry logic turns a small problem into a total outage.

The pattern: service B gets slow. Service A’s calls to B start timing out. A retries — now B gets double the load it was already struggling with. More calls time out, more retries pile on, and the load amplifies until B is completely down, taking A with it, and often A’s callers too. This is a retry storm, and it’s how a minor blip becomes a full outage.

The standard defences:

• Timeouts everywhere. A call with no timeout can hang forever and tie up a thread. Every network call gets a bounded timeout.
• Exponential backoff with jitter. Don’t retry immediately or on a fixed schedule — wait longer each time, with randomness, so all clients don’t retry in lockstep.
• Circuit breakers. After a dependency fails repeatedly, stop calling it for a cooldown period and fail fast (or return a fallback). This gives the struggling service room to recover instead of being pounded.
• Load shedding. When overloaded, deliberately reject a fraction of requests early (return 429/503) so the system stays up for the rest, rather than collapsing for everyone.
• Bulkheads. Isolate resources (separate thread pools / connection pools per dependency) so one slow dependency can’t consume every thread and sink unrelated features.

Without protection:  B slow → A retries → B slower → more retries → outage
With protection:     B slow → timeout → circuit opens → A fails fast / falls back → B recovers

Single points of failure

A single point of failure (SPOF) is any component with no redundancy, whose failure takes down the system. The classic interview question is simply: “What happens if this box dies?” Ask it of every component in your diagram.

• One load balancer → run at least two with automatic failover (and resilient DNS).
• A single database leader → keep hot standby replicas with automatic promotion on failover.
• One message broker node → run a replicated, partitioned cluster.
• A single coordinator or config service → use a replicated consensus system (e.g. a Raft/ZooKeeper quorum), not one node.

The method is mechanical: walk the request path, point at each box, and ask what fails if it disappears. Where the answer is “everything,” add redundancy. Where a dependency is unavoidable, make sure the system degrades rather than dies — which is the next point.

Back-pressure and graceful degradation

When a downstream component can’t keep up, a naive system just buffers the backlog — an ever-growing queue, a swelling thread pool — until it runs out of memory and crashes hard. The better behaviour is back-pressure: signal upstream to slow down or stop, using bounded queues that reject (or block) when full, so load is refused at the edge instead of silently piling up inside.

Pair this with graceful degradation — when you can’t serve the full experience, serve a reduced one that’s still up:

• Can’t reach the personalization service? Serve a generic, non-personalized feed.
• Recommendations down? Show popular items instead of nothing.
• Search index lagging? Return slightly stale results with a note, not an error page.

A system that gets worse under stress but stays available beats one that’s perfect until it abruptly falls over. Cold starts are a specific case: a freshly deployed service with an empty cache can thundering-herd its own dependencies, so warm caches gradually and ramp traffic rather than sending full load at a cold node.

Interview tip: After you’ve drawn your design, proactively walk through its failure modes: “The hot key here is a celebrity, which I’d replicate and cache. If the cache stampedes I’d use single-flight. The database leader is a SPOF, so I’ll run standby replicas with automatic failover, and if the recommendation service is down the feed degrades to popular items.” Volunteering this is one of the clearest signals of a senior engineer.

The pitfall cheat sheet

Keep this table in your head. When you present a design, scan it against each row:

Notice that the mitigations repeat: redundancy, caching, bounding, and failing fast solve most of these. That’s the toolkit.

Key Takeaways

1. Hot keys overload one shard no matter how good your hashing is. Replicate, cache, split, or dedicate capacity to the celebrity.
2. Cache stampedes hit the database when a popular key expires. Use single-flight, early recomputation, serve-stale, and jittered TTLs.
3. Retry storms cascade. Protect every dependency with timeouts, backoff + jitter, circuit breakers, load shedding, and bulkheads.
4. Hunt single points of failure by asking “what breaks if this box dies?” of every component, and add redundancy.
5. Apply back-pressure with bounded queues instead of buffering without limit, and degrade gracefully so the system stays up under stress.
6. Name your own design’s failure modes unprompted. Spotting where your system breaks — and how you’d stop it — is exactly what distinguishes a senior answer.