Pipelining & Transactions

A single Redis instance does ~100,000 operations per second (Lesson 1) — but that number hides something important: at that rate, Redis is mostly waiting on the network, not doing work. Which means the single biggest thing you can do for throughput isn’t tuning Redis — it’s changing how your client talks to it. This lesson is pipelining (the measured 16–25× lever) and the atomicity tools: transactions, optimistic locking, and Lua scripts.

Pipelining: batch the round trips

Normally, a client sends one command, waits for the reply, then sends the next. Each command is a full network round trip, and at sub-millisecond Redis execution times, that round trip — not Redis — dominates. Pipelining breaks the lockstep: the client sends many commands back-to-back without waiting for each reply, then reads all the replies together. Dozens of commands now ride in one (or a few) round trips.

The effect is dramatic. I measured GET throughput on one connection at different pipeline depths:

• No pipeline (P=1): ~93,000 ops/sec (p50 ~0.26 ms).
• Pipeline P=16: 1,497,006 ops/sec — a 16× increase (p50 ~0.43 ms).
• Pipeline P=64: 2,369,820 ops/sec — a 25× increase (p50 ~1.15 ms).

Sixteen to twenty-five times the throughput, from the same Redis, just by not waiting for each reply. Notice the trade-off is tiny: latency per command barely rose (0.26 → 1.15 ms even at P=64), because you’re amortising the round trip, not slowing Redis. Whenever you issue several commands at once — populating a cache, writing a batch, reading many keys — pipeline them. Most client libraries make this a pipeline() call. (The related MGET/MSET do this for multi-key get/set in a single command.) Pipelining is the highest-leverage, lowest-effort Redis optimisation there is.

One caution: pipelining is about throughput, not atomicity — the commands in a pipeline still interleave with other clients’ commands, and you don’t see replies until the end (so you can’t branch on an earlier reply mid-pipeline). For atomicity, you need the next tools.

Transactions: MULTI / EXEC

Redis transactions group commands so they execute atomically and in isolation. You queue commands between MULTI and EXEC:

MULTI
INCR orders:count
LPUSH orders:recent "order:7"
EXEC

Between MULTI and EXEC, the commands are queued, not run. EXEC runs them all as a single unit — no other client’s command interleaves, and they all execute in order. (Note: Redis transactions don’t roll back on a logic error the way a SQL transaction does — if one command fails, the others still run; the guarantee is atomic isolation, not rollback. Validate inputs before queueing.)

This gives you “these commands happen together, with nothing in between” — useful when a logical operation spans several commands and you can’t let another client see a half-done state.

Optimistic locking: WATCH

What about check-then-act — “read a value, and write only if it hasn’t changed”? That’s a classic race. Redis solves it with WATCH (optimistic locking):

WATCH balance:42
val = GET balance:42          # read and decide in the client
MULTI
SET balance:42 <new value>
EXEC                          # aborts (returns nil) if balance:42 changed since WATCH

WATCH marks a key; if that key is modified by anyone between the WATCH and the EXEC, the EXEC aborts (does nothing) and you retry the whole read-decide-write. It’s optimistic: you assume no conflict and only pay (a retry) when there is one. This is the Redis way to do conditional updates safely without holding a lock — great for low-contention check-then-set.

Lua scripts: the most powerful atomicity

For genuinely complex atomic logic — multiple steps with branching that must all happen as one indivisible operation — the strongest tool is a Lua script run with EVAL. Redis executes the whole script server-side, atomically, as a single command (the single thread runs it start to finish with nothing interleaving):

-- atomic "decrement stock only if available", in one round trip and one atomic op
if tonumber(redis.call('GET', KEYS[1])) > 0 then
  return redis.call('DECR', KEYS[1])
else
  return -1
end

A Lua script combines the best of everything: it’s atomic (no interleaving), it’s one round trip (the logic runs on the server, not the client), and it can do arbitrary conditional multi-step logic. It’s the go-to for rate limiters, conditional updates, custom atomic operations, and distributed-lock release (Lesson 10). The rule of thumb: when you need several steps to be one atomic operation, write a Lua script. (Keep scripts small and fast — they block the single thread while running.)

So: pipeline for throughput (16–25×), MULTI/EXEC for atomic grouping, WATCH for optimistic check-then-set, and Lua for complex atomic logic. Next we look at Redis as a message broker: pub/sub and streams.

Pitfalls to Avoid

1. Issuing many commands one-by-one. At ~100k ops/sec you’re mostly paying network latency. Not pipelining leaves 16–25× throughput on the table. Batch commands (pipeline, or MGET/MSET).
2. Confusing pipelining with atomicity. A pipeline is about round trips, not isolation — other clients’ commands still interleave, and you can’t branch on mid-pipeline replies. Use MULTI/EXEC or Lua for atomicity.
3. Expecting SQL-style rollback from MULTI/EXEC. Redis transactions don’t roll back on a command error — the rest still execute. Validate before queueing; use Lua for conditional logic.
4. Doing check-then-set without WATCH (or Lua). Reading a value and writing back based on it is a race if another client changes it in between. Use WATCH (optimistic) or a Lua script (atomic).
5. Huge pipelines or slow Lua scripts. A giant pipeline buffers lots of replies in memory, and a long Lua script blocks the single thread (Lesson 12), stalling all other clients. Keep batches reasonable and scripts short.
6. Using Lua for everything. Lua is powerful but harder to debug and blocks the thread. Reach for it when you genuinely need multi-step atomicity, not for simple operations a single command handles.

Quick Recap

• Redis spends most of its time on network round trips, so pipelining (send many commands without waiting for each reply) is the biggest throughput lever. Measured (GET): ~93k → 1.50M ops/sec at P=16 (16×) → 2.37M at P=64 (25×), latency barely rising. Batch whenever you issue several commands.
• MULTI/EXEC run a group atomically and in isolation (no interleaving) — but no rollback on errors.
• WATCH adds optimistic locking for check-then-set: EXEC aborts if the watched key changed, so you retry.
• Lua (EVAL) runs multi-step logic server-side as one atomic unit — the most powerful tool for complex atomic operations (and lock release, Lesson 10). Keep scripts small.

Next lesson: Redis as a message broker — pub/sub and streams (fire-and-forget vs durable).