Scaling Infrastructure: The Edge, Horizontal Scale, and the Math Behind Them

Application-level fixes (kill the N+1, add the index) get you a long way, but eventually you need the infrastructure to carry more load without falling over. Infrastructure performance comes down to two moves — do less work (cache it, push it to the edge) and spread what’s left (scale out, offload) — plus the math to know when scaling will actually help. As always, measure first.

The scaling toolkit, in one picture

The four moves, in the order that pays off:

Cache at the edge (CDN). The fastest request is the one that never reaches your servers. Serve static assets and cacheable responses from a CDN POP near the user — it cuts latency (no transcontinental round trip) and load (your origin never sees those requests).
Scale horizontally behind a load balancer. More cheap boxes beat one huge one: it’s cheaper past a point, and it gives you redundancy. Autoscale on real signals — p95 latency or queue depth, not just CPU, because a service can be latency-dead at 40% CPU (blocked on I/O or a lock).
Offload. Push reads to replicas (most apps are read-heavy) and slow work to a queue (email, thumbnails, exports — anything the user doesn’t need synchronously). Both take load off the hot path.
Right-size before you scale. A mis-sized box (wrong instance type, undersized connection pool, default heap) wastes money whether you scale it up or out. Fix the unit before you multiply it.

Vertical vs horizontal: vertical (a bigger box) is simplest and sometimes correct — it has no distributed-systems tax — but it has a ceiling and a single point of failure. Horizontal scales further and survives failures, at the cost of needing statelessness, a load balancer, and the math below. Reach for vertical first for small systems; horizontal when you outgrow one box or need redundancy.

The scaling math they don’t teach you

Scaling out is not free speed. Three numbers decide whether adding infrastructure helps.

$The scaling math they don't teach you: tail amplification — one request fans out to N services, each slow just 1% of the time, so fan-out 1 is 1% of requests slow, fan-out 10 is 9.6%, fan-out 100 is 63.4%; scale wide and the tail becomes the norm. Cache hit-ratio with hit 1 ms and miss 50 ms: 50% hit gives 25.5 ms average and origin sees 50%, 90% gives 5.9 ms and 10%, 95% gives 3.5 ms and 5%, 99% gives 1.5 ms and 1% — the last few percent do the most, for latency and for shielding the origin. Amdahl's law: if 90% of the work parallelises, 10× is the most you can ever get; the serial 10% caps you. Find the serial part first; scaling the parallel part past it just burns money.$

Cache hit-ratio — the cheapest win, and the last few percent matter most

With a 1 ms hit and a 50 ms miss:

hit 50%  →  avg 25.5 ms,  origin sees 50% of traffic
hit 90%  →  avg  5.9 ms,  origin sees 10%
hit 95%  →  avg  3.5 ms,  origin sees  5%
hit 99%  →  avg  1.5 ms,  origin sees  1%

Notice the top end: going 90% → 99% halves average latency again and cuts origin load 10×. The last few percent of hit-rate are the most valuable — both for user latency and for shielding your origin from load. Chasing 95→99% is usually worth more than any amount of scaling the origin.

Tail amplification — scale makes the tail the norm

When a request fans out to many services, their slow tails compound. Each service slow just 1% of the time (its p99):

fan-out   1  →   1.0% of requests slow
fan-out  10  →   9.6% slow
fan-out 100  →  63.4% slow

At 100-way fan-out, 63% of requests wait on someone’s bad p99. This is the defining problem of microservice and distributed systems. The cures are structural: cut fan-out (don’t call 100 services when 10 will do), set tight timeouts with fallbacks, hedge (send a duplicate request after a short delay and take the first response), and above all make the tail fast, not just the median.

Amdahl’s law — one serial bottleneck caps everything

speedup ≤ 1 / (1 − p). If 90% of the work parallelises, 10× is the ceiling no matter how many servers you add — the serial 10% (a global lock, a single DB writer, a shared counter, a leader node) decides the limit. So find the serial part first. Adding boxes to scale the parallel part once the serial part dominates is pure waste. This is why “we added more servers and it didn’t get faster” happens — the bottleneck was never the thing you scaled.

Where things actually break

Autoscaling on the wrong signal. CPU looks fine while latency dies (I/O- or lock-bound). Scale on p95 latency and queue depth.
Scaling the app, not the database. You can add app servers in minutes; the single primary database they all hammer is the real ceiling. Replicas, caching and sharding are the harder, more important work — see distributed systems in practice.
Cold caches and thundering herds. When a popular cache entry expires, thousands of requests stampede the origin at once. Use request coalescing / “single-flight” and staggered TTLs.
No backpressure. Under overload, a system without limits accepts work it can’t finish and collapses. Shed load (reject early), queue with bounds, and fail fast.

Takeaways

Do less work: cache at the edge so most requests never reach you — and chase the top of the hit-ratio curve (90→99% is where the big wins are).
Spread what’s left: scale horizontally behind a load balancer, autoscale on p95/queue depth, offload reads to replicas and slow work to queues.
Right-size before you multiply a mis-configured unit.
Tail amplification: at scale a “rare” slow call becomes the norm — cut fan-out, set timeouts, hedge, make the tail fast.
Amdahl’s law: one serial bottleneck caps the whole system — find and fix it before adding hardware.

Part of a series: Performance, Measured (the method), backend performance (the app), and AI application performance. Go deeper in distributed systems in practice and the Kubernetes course.