Low-Latency Implementation — Low-Latency Trading Systems & Algo Trading — From the Order Book to a Live Retail Algo

“Low latency” sounds like it should be about exotic tricks, but most of it is the opposite: it’s about removing the things that cause delay and, just as importantly, jitter (unpredictable variation). A hot path that’s fast on average but occasionally stalls for a millisecond is often worse than one that’s a bit slower but consistent, because that one stall is the tick you missed. This lesson covers the concrete techniques — ring buffers, lock-free design, and the Python-vs-C++ decision — all grounded in measurements I ran so you can see the real numbers, not hand-waving.

The Ring Buffer: Lock-Free Handoff

The most important data structure in a low-latency system is the humble ring buffer (circular buffer). It’s how you pass data between threads — say, from the thread reading the market-data feed to the thread running your strategy — without a lock and without allocating memory.

The ring buffer, a lock-free single-producer single-consumer queue: the workhorse that moves ticks between threads without a lock. A fixed-size array of slots arranged in a circle; some slots filled, some empty. Producer writes at head, advancing to slot 7; consumer reads at tail, advancing from slot 2. Why it's fast: fixed memory allocated once so zero garbage on the hot path; one producer one consumer so no lock needed, just atomic head/tail indices; cache-friendly sequential access. Market-data thread writes ticks, strategy thread reads them, handoff with no lock and no allocation

The idea: a fixed-size array treated as a circle. One thread (the producer) writes new items at the head index and advances it; another thread (the consumer) reads items at the tail index and advances it. When an index reaches the end of the array, it wraps around to the start. For the common single-producer, single-consumer (SPSC) case, you need no lock at all — just the two indices, updated atomically — because only one thread ever writes the head and only one ever writes the tail.

Why this is the workhorse of low-latency systems:

Fixed memory, allocated once. The array is allocated at startup and reused forever, so there’s zero allocation on the hot path — which means zero garbage to collect (critical for Python, below) and no malloc latency.
No lock needed (SPSC). With one producer and one consumer, atomic index updates are enough; threads never block each other. (For multiple producers/consumers you need a bit more care, but the SPSC case covers the common “feed thread → strategy thread” handoff.)
Cache-friendly. Sequential access through a contiguous array plays well with CPU caches, unlike pointer-chasing structures.

In a trading system, the market-data thread produces ticks into the ring buffer and the strategy thread consumes them — a clean, fast, predictable handoff. This single pattern underlies most of what makes a hot path quick.

Why Thread Locks Hurt

The instinct when sharing data between threads is to reach for a lock (mutex). On a hot path, that instinct is often wrong — and I measured why, in C++ with real threads (Python’s GIL hides the effect, which is itself telling).

The measurement: a counter incremented millions of times, protected first by a mutex and then by a lock-free atomic, across 1, 2, 4 and 8 threads:

One thread: the mutex is cheap (~21 ns/op) — no contention, nobody to fight.
Under contention (2–8 threads): the mutex jumps to ~55–60 ns/op — roughly 3–4× slower — because threads now block waiting for each other, and the OS may get involved. The atomic stays low (~6–22 ns/op) throughout.

Two lessons from this. First, the cost of a lock isn’t the lock itself — it’s contention: the more threads fight over one lock, the more each pays, in both average time and unpredictable jitter (a thread might wait a little or a lot). Second, that jitter is poison on a hot path: you can’t have your strategy thread randomly stalling because it’s waiting for a lock the logging thread holds.

The fixes, in order of preference:

Don’t share — give each thread its own data; pass messages via ring buffers (above) instead of sharing mutable state.
Shard — if you must share, split the state so different threads touch different shards and rarely collide (e.g. per-instrument state).
Go lock-free — use atomics and lock-free structures for the genuinely shared bits.
If you must lock, hold it briefly — never do slow work (I/O, logging, allocation) while holding a hot-path lock.

This is also why matching engines (Lesson 4) are single-threaded per instrument: no sharing, no locks, no races, deterministic order.

Python’s Hidden Jitter: the Garbage Collector

Even single-threaded, Python has a latency problem that matters here: the garbage collector and memory allocation cause pauses. I measured the per-iteration latency of a Python loop that allocates objects:

The median was tiny — around 250 ns.
But the tail was ugly: the 99.9th percentile spiked to ~18–30 microseconds, and the maximum to the millisecond range.

That shape — fast median, occasional big spikes — is exactly what you don’t want on a hot path. If one tick in a thousand takes 100× longer to process because the GC kicked in, that’s the tick where you missed the quote. The median doesn’t trade; the tail does.

Mitigations if you must keep hot logic in Python: pre-allocate and reuse objects (don’t create garbage in the loop), disable the GC in the critical section (it helped the tail in my measurement) and collect manually at safe moments, and avoid allocation-heavy patterns. But the honest conclusion is that Python’s jitter is hard to fully tame — which leads to the central decision of this lesson.

Python vs C++: the ~257× Gap, and the Split

I ran the same tick-processing loop (a running EMA over a simulated price walk) in both languages:

C++: 5.25 ns per tick (~190 million ticks/sec).
Python: 1,351 ns per tick (~0.74 million ticks/sec).

That’s roughly 257× faster in C++ — and C++ also gives you the predictable latency (no GC pauses) that the hot path needs. So does everyone rewrite everything in C++? No — and understanding why not is the key architectural insight.

Python vs C++ on the hot path: same tick-processing loop measured, why the engine is C++. Per-tick cost, lower is better. C++ 5.25 ns/tick (190 M ticks/sec), a small green bar. Python 1351 ns/tick (0.74 M ticks/sec), a long red bar. Approximately 257x slower in pure Python. So how do real retail systems use both? C++ for the HOT path, the tick-to-order loop where nanoseconds and predictable latency (no GC pauses) matter: order book, matching/signals, risk checks, the ring buffers between them. Python for everything ELSE: research, backtesting, config, the web service, dashboards, glue, where developer speed matters more than microseconds. Bridge them with a ring buffer, shared memory or a small C++ extension; Python orchestrates, C++ executes; you don't rewrite the world in C++, you rewrite the 5% on the critical path

The standard, sane split:

C++ for the hot path — the latency-critical tick-to-order loop where nanoseconds and determinism matter: order-book maintenance, signal computation, pre-trade risk checks, and the ring buffers between them.
Python for everything else — research, backtesting, configuration, the web/API service, dashboards, glue code. Here, developer speed matters far more than microseconds, and Python’s productivity wins easily.
Bridge them — pass data between the Python and C++ parts via a ring buffer / shared memory, or wrap the C++ hot path as a Python extension so Python orchestrates and C++ executes.

The wisdom is: you don’t rewrite the world in C++; you rewrite the ~5% that’s genuinely on the critical path. Most of your system — and all of your research — stays in friendly, fast-to-write Python; only the hot loop drops to C++. This is exactly the architecture we assemble in Lesson 8. And for a retail algo in the milliseconds game (Lesson 1), you may not need C++ at all — but knowing where the line is, and why, is what separates someone who understands latency from someone who cargo-cults it.

Pitfalls to Avoid

Allocating on the hot path. Every allocation is potential GC/malloc latency and jitter. Pre-allocate and reuse (ring buffers).
Sharing mutable state behind a lock on the hot path. Contention adds latency and jitter. Don’t share; shard; or go lock-free.
Holding a lock while doing slow work. Logging or I/O under a hot-path lock stalls every waiting thread. Keep critical sections tiny.
Ignoring the tail latency. The median doesn’t trade; the p99.9 does. Measure and tame the tail, especially in Python (GC).
Rewriting everything in C++. Wasteful and slow to build. Put only the ~5% critical loop in C++; keep the rest in Python.
Cargo-culting low-latency tricks you don’t need. A retail millisecond strategy doesn’t need FPGA-grade tuning. Optimise where it actually matters.

Quick Recap

The ring buffer (fixed-size, SPSC, atomic indices) passes data between threads with no lock, no allocation — the workhorse of a hot path.
Thread locks hurt under contention: measured ~55–60 ns/op vs an atomic’s ~6–22 ns, plus jitter. Prefer not-sharing, sharding, or lock-free.
Python’s GC causes tail-latency spikes (median ~250 ns, p99.9 tens of µs) — bad on a hot path. Pre-allocate, or keep the hot path out of Python.
The same loop was ~257× faster in C++ (5.25 ns vs 1351 ns/tick), with better determinism — so use C++ for the hot path, Python for everything else, bridged by a ring buffer.
Optimise only the ~5% that’s critical. Educational, not financial advice.

Next lesson: zooming out from the code to the whole path — the tick-to-trade latency budget, where the microseconds actually go, and what co-location buys the professionals.