Delivery Semantics

“Does Kafka deliver each message exactly once?” is the question everyone asks, and the honest answer — “by default, at least once; exactly-once is possible but scoped” — is worth understanding precisely, because getting it wrong means either lost data or double-charged customers. This lesson is the three delivery semantics, what Kafka actually gives you, and how to build correct systems on top.

The three guarantees

• At-most-once — each message is delivered zero or one times. You get this by committing the offset before processing (or acks=0): if you crash in between, the message is skipped and lost. No duplicates, but data loss — rarely acceptable for anything that matters.
• At-least-once — each message is delivered one or more times. You get this by acks=all and committing after processing: nothing is ever lost, but a crash or retry between processing and committing means you reprocess — duplicates are possible. This is the practical default for correctness-sensitive systems, because “never lose data” is usually the priority.
• Exactly-once — each message takes effect once: no loss and no duplicates. The ideal, achievable within Kafka with the right machinery (below), at some throughput cost.

Measured: at-least-once means duplicates happen

Why is at-least-once the default, and why must you design for duplicates? Because to guarantee no loss, the system retries on uncertainty — and retries produce duplicates. I measured the effect: a stream with a 5% duplicate delivery rate arrived as 1,049,817 messages (4.7% of them duplicates). Deduplicating by a stable event id recovered exactly 1,000,000 unique messages — and without that dedup step, 49,817 messages would have been double-processed, inflating every count and double-applying every effect.

So the robust pattern is: accept at-least-once, and make your consumers idempotent so that reprocessing a message is harmless:

• Dedup on a stable event id — give every event a unique id and drop ones you’ve already seen (a set, a cache, or a uniqueness constraint).
• Upsert by a business key — write operations keyed by order_id so applying the same event twice is a no-op (INSERT becomes an idempotent UPSERT).
• Make effects naturally idempotent — “set status = PAID” is idempotent; “increment balance” is not (so dedup before incrementing).

This is the same idempotency principle as the idempotency article and the data engineering course: at-least-once delivery + an idempotent consumer = exactly-once effect, which is what you actually care about. Build this way and duplicates — which will happen — simply don’t matter.

How Kafka’s exactly-once really works

Kafka does offer exactly-once, and it’s genuinely useful — but it’s important to understand its scope. It’s built from two pieces:

• The idempotent producer (Lesson 3, now the default): stops producer retries from writing duplicates to the broker. Each producer session has sequence numbers, so the broker discards a re-sent message. This removes broker-side duplicates.
• Transactions: let a producer write to multiple partitions/topics atomically — all the writes commit together or none do. Combined with consumers set to isolation.level=read_committed (which skip aborted transactions), this enables atomic read-process-write: consume from topic A, process, and produce to topic B, where the consume-offset-commit and the produce happen in one transaction. If anything fails, the whole thing rolls back and is retried, with no partial or duplicated output.

This gives true exactly-once within Kafka — for Kafka-to-Kafka pipelines. Kafka Streams turns it on with a single setting (processing.guarantee=exactly_once_v2), which is why stream processing can be exactly-once almost for free.

The crucial limitation: exactly-once is scoped to Kafka. It does not magically extend to external systems. If your consumer reads from Kafka and writes to an external database or calls an external API, Kafka’s transactions don’t cover that write — you’re back to needing an idempotent write to the external system (dedup/upsert) to avoid double effects. So “exactly-once” is not a switch that makes your whole pipeline perfect; it’s a strong guarantee inside Kafka that you extend outward with idempotent sinks.

The practical stance

Putting it together, the design that actually works:

1. Default to at-least-once — acks=all, commit after processing — so you never lose data.
2. Make every consumer idempotent — dedup on an event id or upsert by a business key — so the duplicates that at-least-once inevitably produces are harmless.
3. Use Kafka exactly-once (transactions) for Kafka-to-Kafka pipelines where you need it — especially via Kafka Streams, where it’s one setting.
4. For external sinks, rely on idempotent writes — Kafka’s transactions stop at Kafka’s boundary.

Design this way and you get exactly-once effect end to end, which is the real goal. Chasing a magic “exactly-once delivery” setting and assuming it covers everything is how teams get surprised by double-processing at the edges. Next we go underneath delivery to the thing that makes “no loss” possible at all: replication.

Pitfalls to Avoid

1. Assuming Kafka is exactly-once by default. It’s at-least-once — duplicates happen (measured 4.7%). If you don’t dedup, you double-process. Design for it.
2. Not making consumers idempotent. At-least-once + a non-idempotent consumer = double effects (double charges, inflated counts). Dedup on an event id or upsert by business key.
3. Thinking exactly-once covers external systems. Kafka’s transactions are Kafka-to-Kafka only. A write to an external DB/API still needs an idempotent write — exactly-once stops at Kafka’s boundary.
4. Committing before processing to “avoid duplicates.” That’s at-most-once — you trade duplicates for lost data, which is almost always worse. Prefer at-least-once + idempotence.
5. Turning on transactions everywhere for safety. Exactly-once transactions cost throughput and add complexity; use them where the pipeline genuinely needs atomic read-process-write, not as a blanket default.
6. No stable event id. Dedup needs a stable, unique id per logical event. If producers don’t attach one, consumers can’t deduplicate reliably — design the id in from the start.

Quick Recap

• Three semantics: at-most-once (lossy, no dups), at-least-once (no loss, dups possible — the default), exactly-once (no loss, no dups — within Kafka, at a cost).
• Measured: a 5% duplicate stream (1,049,817 messages, 4.7% dups) deduped by a stable event id to exactly 1,000,000; without dedup, 49,817 double-processed.
• The robust design: at-least-once + idempotent consumers (dedup on event id / upsert by business key) = exactly-once effect, which is what matters.
• Kafka’s exactly-once = idempotent producer + transactions (atomic read-process-write), great for Kafka-to-Kafka (one setting in Streams) — but it doesn’t extend to external systems, which still need idempotent writes.

Next lesson: what makes “no loss” possible at all — replication, ISR and durability (with the measured acks/min.insync.replicas table).