Consumers & Consumer Groups — Apache Kafka — The Distributed Log, End to End

Producers append; consumers read. But the clever part of Kafka consumption isn’t reading a single stream — it’s how a consumer group lets many consumers share a topic’s work and scale out, while different groups each get the whole stream independently. Understanding consumer groups — and the hard ceiling the partition count puts on them — is essential to running Kafka at scale.

A consumer group is a set of consumers that cooperate to read a topic. Kafka divides the work by assigning each partition to exactly one consumer in the group:

With 12 partitions and 3 consumers in a group, each consumer handles 4 partitions. Add consumers and the partitions redistribute — the work scales out. Each consumer independently tracks the offset of the partitions it owns, so it knows where to resume. This is how Kafka scales consumption: add consumers to a group, and throughput grows.

Measured: the partition count caps parallelism

But there’s a hard ceiling, and it’s the single most important operational fact about consumers. Because each partition goes to exactly one consumer in a group, you can’t have more active consumers than partitions. I measured it on a 12-partition topic:

1 → 12 consumers: linear scaling. 1 consumer handles all 12 partitions; 3 handle 4 each; 12 handle 1 each — the maximum useful parallelism.
16 consumers: 4 sit idle. The 13th, 14th, 15th and 16th consumers get no partition at all — they’re assigned nothing and do nothing. Adding them is pure waste.

So the partition count is your maximum consumer parallelism. If you need 20 consumers’ worth of throughput, you need at least 20 partitions. And since adding partitions later is disruptive (it remaps keys, Lesson 2), you choose the partition count with headroom for growth up front. This is why “how many partitions?” is a question you answer by thinking about peak consumer parallelism, not just producer throughput.

(The flip side: if one partition is a hot partition from a skewed key — Lesson 3 — its single consumer is overloaded while others idle, and adding consumers can’t help because the work is stuck in one partition. Skew and the parallelism cap are two sides of the same coin.)

Committing offsets: where delivery semantics begin

A consumer periodically commits its offset — records “I’ve processed up to here” — so that after a restart or a rebalance it resumes from the right place. When you commit relative to when you process is exactly what determines your delivery semantics (full treatment in Lesson 5):

Commit after processing → at-least-once. If you crash between processing and committing, you reprocess those messages on restart (duplicates possible, no loss). This is the common, safe default.
Commit before processing → at-most-once. If you crash after committing but before processing, those messages are skipped (loss possible, no duplicates). Rarely what you want.

The practical rule: process, then commit — and make your processing idempotent so the occasional reprocess is harmless (Lesson 5). Auto-commit (enable.auto.commit) is convenient but commits on a timer regardless of processing, so for correctness-sensitive work, commit manually after you’ve done the work.

Rebalancing: the cost of elasticity

When a consumer joins, leaves, or crashes, the group must rebalance — reassign partitions among the remaining consumers. Rebalancing is what makes consumer groups elastic and fault-tolerant (a crashed consumer’s partitions get picked up by others automatically), but it has a cost: during a rebalance, consumption pauses while partitions are reassigned.

So frequent rebalances hurt throughput. They’re triggered by consumers joining/leaving, but also by consumers being too slow (missing heartbeats or max.poll.interval.ms, so the group thinks they died). To keep rebalances rare and cheap:

Tune session.timeout.ms and max.poll.interval.ms so a briefly-slow consumer isn’t wrongly evicted.
Prefer cooperative (incremental) rebalancing (the modern default), which reassigns only the partitions that need to move instead of stopping the whole group — far less disruptive than the old “stop-the-world” rebalance.
Keep per-message processing fast, or increase the poll interval, so consumers always heartbeat in time.

Groups are independent: that’s the fan-out

The last essential idea: different consumer groups are completely independent. Each group gets the entire topic and tracks its own offsets. So:

The analytics group reads all of orders at its pace; the fraud group also reads all of orders at its pace; the search-indexer group too. None affects the others.
Within a group, work is shared (each partition to one member). Across groups, the stream is fanned out (each group gets everything).

This is how Kafka is simultaneously a queue (share work within a group) and pub/sub (fan out across groups) — a single model that does both, which is exactly the flexibility the distributed log gives you. One topic, many groups, each consuming independently, each scaling within itself up to the partition count.

Pitfalls to Avoid

More consumers than partitions. Measured, extra consumers beyond the partition count sit idle. The partition count caps parallelism — add partitions (with care) if you need more consumers, don’t just add consumers.
Too few partitions for future growth. Since re-partitioning is disruptive (remaps keys), a too-low partition count locks in a low parallelism ceiling. Size with headroom.
Committing before processing. That’s at-most-once — a crash loses the uncommitted-but-unprocessed messages. Process first, then commit (Lesson 5), and make processing idempotent.
Relying on auto-commit for correctness. It commits on a timer regardless of whether processing succeeded, risking both loss and duplicates. Commit manually after processing for anything important.
Frequent rebalances from slow consumers. A consumer that exceeds max.poll.interval.ms is kicked out, triggering a rebalance and a pause. Keep processing fast or raise the interval; use cooperative rebalancing.
Assuming groups share data. Different groups each get the full stream. If you expected two groups to split the work, you’ll double-process — put consumers in the same group to share.

Quick Recap

A consumer group shares a topic: each partition → exactly one consumer in the group, processed in parallel, each tracking its offset. Add consumers to scale.
Measured: parallelism is capped by the partition count — 12 partitions feed up to 12 consumers; a 16-consumer group leaves 4 idle. Size partitions with headroom.
Commit after processing for at-least-once (safe default, make processing idempotent); committing before is at-most-once (lossy).
Rebalancing (on join/leave/crash/slowness) reassigns partitions and pauses consumption — keep it rare: tune timeouts, use cooperative rebalancing, keep processing fast.
Different groups are independent (each gets the whole stream) — Kafka is a queue within a group and pub/sub across groups.

Next lesson: the guarantees underneath all of this — delivery semantics: at-least-once by default, and how exactly-once actually works (with a measured dedup).