menu_bookGyanByte
arrow_backBACK TO REDIS — THE IN-MEMORY DATA-STRUCTURE STORE, END TO END
Lesson 06Redis — The In-Memory Data-Structure Store, End to End4 min read

Pub/Sub & Streams

October 30, 2026

TL;DR

Redis has two messaging tools. PUB/SUB: PUBLISH to a channel, SUBSCRIBE to receive — messages are pushed to whoever is listening RIGHT NOW and are NOT stored, so an offline subscriber misses them (no replay, no ack). Great for live fan-out where loss is fine: notifications, chat, broadcasting cache-invalidation signals. STREAMS: XADD appends entries that ARE stored, with consumer groups, offsets and explicit ACKs — a durable, replayable, at-least-once log, essentially a mini-Kafka inside Redis. Use streams when messages must not be lost (durable queues, event streams). vs Kafka: Redis Streams is great for modest volumes and when you already run Redis (lower latency, one less system), but for a high-volume, many-consumer, long-retention event backbone, Kafka is built for that scale. Redis Streams is the convenient middle; Kafka is the heavy-duty log.

Redis isn’t only a store — it can move messages between parts of your system, and it offers two quite different tools for it. Choosing the wrong one is a common mistake: using fire-and-forget pub/sub for messages you can’t afford to lose, or reaching for heavyweight streams when a simple broadcast would do. This lesson is both, and when each fits — including how Redis Streams stacks up against Kafka.

Two tools: pub/sub and streams

Pub/Sub and Streams. Pub/Sub (fire-and-forget): PUBLISH channel msg to SUBSCRIBE, pushed to whoever is listening NOW, NOT stored — an offline subscriber misses it, no replay, no ack; great for live fan-out: notifications, chat, invalidation signals. Streams (durable log): XADD appends, entries are STORED, consumer groups, offsets, ACKs, replay history, at-least-once — a mini-Kafka inside Redis, for durable queues and event streams where you can't lose messages. Choosing, and how this compares to Kafka: use pub/sub when losing a message is fine and you want simple live fan-out; if a subscriber isn't connected it simply misses the message, no storage, no acknowledgement, no replay. Use streams when messages must not be lost — XADD stores entries, consumer groups split the work with offsets and explicit ACKs, you can replay — Kafka-like semantics inside Redis. vs Kafka: Redis Streams is great for modest volumes and when you already run Redis (lower latency, one less system), but for a high-volume many-consumer long-retention event backbone Kafka is built for that scale; Redis Streams is the convenient middle, Kafka is the heavy-duty log.

Pub/Sub (fire-and-forget). The publish/subscribe model: a publisher sends a message to a channel (PUBLISH news "..."), and every client currently subscribed to that channel (SUBSCRIBE news) receives it immediately. The defining property: messages are not stored. They’re pushed to whoever is listening at that instant and then gone. If a subscriber isn’t connected, or connects a second later, it misses the message entirely — there’s no history, no replay, no acknowledgement. This makes pub/sub:

  • Simple and low-latency — a pure live broadcast.
  • Lossy by design — fine when missing a message is acceptable.

Use pub/sub for live fan-out where loss is tolerable: pushing a notification to currently-online users, a chat room, live dashboard updates, or broadcasting a cache-invalidation signal (“invalidate product:99”) to all app servers. If a server was briefly disconnected and missed one invalidation, the TTL (Lesson 4) covers it. The signal is ephemeral; losing one is no catastrophe.

Streams (durable log). Redis Streams (added in Redis 5) are a completely different beast: an append-only log of entries that are stored. XADD stream * field value appends an entry with a time-based id; consumers read with XREAD, and — crucially — consumer groups (XREADGROUP) let multiple consumers share the work with offsets and explicit acknowledgements (XACK). This gives you:

  • Durability — entries persist (within the stream’s length/retention), so consumers can read history and replay.
  • At-least-once delivery — a consumer must ACK an entry; un-acked entries (from a crashed consumer) can be re-delivered via the pending-entries list.
  • Consumer groups — split a stream’s work across consumers, like Kafka.

If this sounds like a mini-Kafka inside Redis, that’s exactly right. Use Streams when messages must not be lost: durable work queues, event streams, task processing where a crashed worker’s tasks must be retried, not dropped.

Choosing between them

The decision is simply can you afford to lose a message?

  • Loss is fine, want simple live fan-out → pub/sub. Notifications to online users, invalidation broadcasts, chat. No storage, no acks, lowest latency.
  • Loss is not fine, need durability/replay/acks → streams. Durable queues and event logs where every message must be processed, with consumer groups to scale and ACKs to guarantee it.

A common mistake is using pub/sub for a job queue — then wondering why tasks vanish when a worker was briefly down. For anything that must be processed, use streams (or a proper queue/Kafka). For ephemeral broadcasts, pub/sub is perfect and simpler.

Redis Streams vs Kafka

Since Streams are Kafka-like, when do you use which? (Kafka has its own course.)

  • Redis Streams shines when: you already run Redis (one less system to operate), volumes are modest to moderate, you want low latency, and your retention needs are limited (streams live in RAM, so retention is bounded by memory — you cap length with MAXLEN). It’s the convenient choice for a durable queue or event stream inside an app that already uses Redis.
  • Kafka is built for the heavy-duty event backbone: very high throughput, many consumers, long on-disk retention (days/weeks), replay across huge history, and a rich ecosystem (Connect, Streams, Schema Registry). When the event stream is a core, high-volume, many-system backbone, Kafka is designed for exactly that scale and durability; Redis Streams would strain under it (it’s memory-bound).

The honest framing: Redis Streams is the convenient middle — durable messaging without standing up Kafka — and Kafka is the heavy-duty log for when you’ve outgrown that. Start with Redis Streams if you already run Redis and your scale is modest; move to Kafka when volume, retention or the number of consumers genuinely demands it.

Next we address the elephant in the room for anything durable: persistence — does Redis survive a restart?

Pitfalls to Avoid

  1. Using pub/sub for messages you can’t lose. Pub/sub doesn’t store or ack — an offline or briefly-disconnected subscriber misses messages forever. For durable delivery, use streams (or a real queue/Kafka).
  2. Using pub/sub as a job queue. Tasks vanish if no worker is listening at that instant. Use streams with consumer groups and ACKs so a crashed worker’s tasks are re-delivered.
  3. Unbounded streams. Streams live in RAM and grow with every XADD. Without MAXLEN (or XTRIM), they fill memory (Lesson 3). Cap stream length to what you need.
  4. Not ACKing stream entries. With consumer groups, un-acked entries accumulate in the pending list and get re-delivered forever. XACK after processing; monitor pending entries.
  5. Reaching for Kafka when Redis Streams would do (or vice versa). For modest volume with Redis already in place, Streams is simpler. For a high-volume, long-retention, many-consumer backbone, Kafka is built for it. Match the tool to the scale.
  6. Assuming pub/sub scales to many channels/subscribers for free. Pub/sub delivery is work on the single thread; very high fan-out or message rates can load Redis. For heavy messaging, streams or Kafka are better suited.

Quick Recap

  • Pub/Sub: PUBLISH/SUBSCRIBE, messages pushed to whoever is listening now, not stored — lossy, simple, low-latency. Use for live fan-out where loss is fine: notifications, chat, cache-invalidation signals.
  • Streams: XADD appends stored entries; consumer groups, offsets, ACKs → durable, replayable, at-least-once — a mini-Kafka in Redis. Use when messages must not be lost: durable queues, event streams.
  • Choose by: can you lose a message? Loss OK → pub/sub; must not lose → streams.
  • vs Kafka: Redis Streams is the convenient middle (modest volume, Redis already running, memory-bound retention); Kafka is the heavy-duty log (very high volume, long on-disk retention, many consumers). Cap streams with MAXLEN.

Next lesson: does any of this survive a restart? — persistence and durability (RDB vs AOF).

redispub-substreamsmessagingconsumer-groups
arrow_back PreviousPipelining & TransactionsNext arrow_forwardPersistence & Durability