Messaging and Delivery Guarantees

Every company I’ve worked with has had at least one “duplicate” incident from a message queue. Duplicate SMS (“Your order is out for delivery” twice), duplicate emails, a refund processed twice, cashback credited twice. Each time, someone says “but we’re using Kafka, it’s supposed to be reliable”, or “but the queue is configured for exactly-once”.

The queue was reliable. It did exactly what it promised. The problem was a misunderstanding of what it promised. This lesson clears that up, with a real test you can run in a minute.

Why Use Messaging At All?

Quick reminder of why we put a queue or log between services:

• Decoupling: the order service publishes “order placed” and doesn’t need to know who cares: restaurant, notifications, analytics, loyalty points.
• Buffering: if the notification service is slow or down for 10 minutes, messages wait in the queue instead of failing the order.
• Load levelling: a sale spike of 10x orders can be processed at a steady rate.
• Replay: with a log like Kafka, a new service can read history from the beginning.

The System Design Masterclass covers queues vs logs and when to use which. Here, we focus on one question: how many times does each message get processed?

The Core Problem: Two Steps, One Crash

A consumer has to do two things for every message:

1. Process it (credit the wallet, send the SMS, update a row).
2. Record that it’s done: acknowledge it (RabbitMQ, SQS, Redis Streams) or commit the offset (Kafka).

These are two separate actions, usually on two separate systems. A crash, a deploy, a network blip or a rebalance can happen between them. Which order you do them in decides what can go wrong:

• At-most-once: acknowledge first, then process. If you crash after the ack, the message is gone. No duplicates, but losses. Fine for live location pings, metrics, or debug logs, where a lost message doesn’t matter.
• At-least-once: process first, then acknowledge. If you crash after processing but before the ack, the message is delivered again, and processed again. No losses, but duplicates. This is the default almost everywhere, and the right default.
• Exactly-once: what everyone wants. As a delivery guarantee across a network, it’s impossible, for the same reason as Lesson 2’s “timeout means unknown”: the consumer can never be sure its acknowledgement arrived. What you can build is exactly-once effect: at-least-once delivery plus processing that’s safe to repeat.

Real Test: ₹5,500 of Extra Cashback

I built a small version of the cashback flow with Redis Streams (which has consumer groups and acknowledgements, like Kafka or SQS) and SQLite as the wallet database:

• The stream had 5,000 events, each “credit ₹100 cashback to user u1”.
• The consumer read events, updated the wallet balance, committed, then sent XACK.
• To simulate crashes, deploys and OOM kills, the consumer “crashed” 1% of the time after committing the wallet update but before XACK.
• After each crash, the consumer restarted and, as it should, first re-read its own pending (delivered but unacknowledged) messages.

dedupe=False: crashes=55  expected balance=500000  actual=505500
dedupe=True:  crashes=51  expected balance=500000  actual=500000

Without deduplication, 55 crashes meant 55 events processed twice: ₹5,500 credited for nothing. Scale that to a real cashback campaign with lakhs of events and a bad deploy day, and it’s real money.

With deduplication, the balance was exactly right, even though crashes still happened and messages were still redelivered.

The consumer code with deduplication:

import sqlite3

def handle(db: sqlite3.Connection, event_id: str, user: str, amount: int) -> None:
    try:
        db.execute("INSERT INTO processed(event_id) VALUES (?)", (event_id,))
    except sqlite3.IntegrityError:
        db.rollback()            # already processed: skip, but still ack
        return
    db.execute("UPDATE wallet SET balance = balance + ? WHERE user = ?", (amount, user))
    db.commit()                  # dedupe record + side effect commit TOGETHER

The key detail, again: the dedupe record and the side effect are committed in the same transaction. If you record “processed” in Redis and update the wallet in Postgres, a crash between them breaks the guarantee in one direction or the other.

When the side effect isn’t a database

If processing means calling an external API (send SMS, call the payment gateway), you can’t put it in your database transaction. Options:

• Pass the event ID as the idempotency key to the external API, if it supports one. Then duplicates are harmless on their side.
• Record intent first: insert “SMS for event X: pending” in your DB, send, mark “sent”. On redelivery, check the record. A crash after sending but before marking “sent” can still cause one duplicate, so this narrows the window but doesn’t close it. For SMS, that’s usually acceptable; for payments, use the gateway’s idempotency.
• Accept rare duplicates where they’re harmless, and make sure they’re rare.

What Kafka’s “Exactly-Once” Really Covers

Kafka has a feature called exactly-once semantics (EOS): idempotent producers plus transactions. It’s real and useful, but its scope is narrower than the name suggests.

What it does:

• Idempotent producer (enable.idempotence=true, the default since Kafka 3.0): if the producer retries a send because it didn’t get an acknowledgement, the broker detects the duplicate and stores the message only once, in order.
• Transactions: a stream processor can read from topic A, write results to topic B, and commit its consumer offset on A, all atomically. Either all of it happens or none of it. Kafka Streams uses this with processing.guarantee=exactly_once_v2.

What it doesn’t do:

• It doesn’t cover side effects outside Kafka. The moment your consumer writes to Postgres, calls an API, or sends an email, you’re back to at-least-once for that side effect, and you need the dedupe pattern from above.

So the practical rule stays the same: assume every message can be delivered more than once, and make every handler idempotent.

Ordering: Per Partition, Not Per Topic

The second big source of messaging bugs is ordering. Kafka (and similar logs like Kinesis, Pulsar, and Azure Event Hubs) only guarantees order within a partition:

• Messages with the same key go to the same partition, and are stored and delivered in order.
• Messages in different partitions have no ordering relative to each other.
• Within a consumer group, each partition is read by exactly one consumer at a time. So the number of partitions is the maximum parallelism.

So, if the order of events for one order matters (created → paid → shipped), use order_id as the message key. Then all events for that order go to one partition, in order. Events for different orders can be processed in parallel.

Ways I’ve seen ordering break anyway:

• No key, or the wrong key. With no key, messages are spread across partitions, and “paid” can be processed before “created”.
• Parallel processing inside a consumer. A consumer that hands messages to a thread pool for speed gives up ordering. If message 1 fails and is retried, message 2 is already done.
• Changing the partition count. hash(key) % partitions changes (the same problem as Lesson 7), so new events for an order may land in a different partition than old ones, and be processed out of order during the transition.
• Retries to the end of the queue. Many SQS/RabbitMQ setups put a failed message back at the end. Now it’s behind newer messages for the same entity.

And the defensive habit that saves you anyway: make handlers tolerate out-of-order events. Include a version number or sequence in each event, and ignore events older than what you’ve already applied (“I’m already at status SHIPPED, version 3; ignore this PAID event with version 2”).

Poison Messages and Dead-Letter Queues

Sometimes a message can never be processed: malformed JSON, a reference to a deleted record, a bug triggered by one specific input. With at-least-once delivery, the consumer fails, doesn’t acknowledge, gets the message again, fails again… forever. In Kafka, this also blocks the whole partition behind it, because the consumer can’t move past it.

The fix is a dead-letter queue (DLQ):

1. Retry a few times with backoff (the failure might be temporary, like a dependency being down).
2. After N attempts, publish the message to a separate DLQ topic, with the error and attempt count, and acknowledge the original.
3. Alert on DLQ size, and have a way to inspect, fix and replay DLQ messages.

Be careful to separate the two kinds of failure: if the database is down, every message will fail, and you don’t want to dump your entire stream into the DLQ. Pause consumption (or apply a circuit breaker, Lesson 11) for infrastructure failures; use the DLQ for message-specific failures.

Consumer Lag: The Metric to Watch

Consumer lag is how far behind the consumer is: for Kafka, the difference between the latest offset and the consumer’s committed offset, per partition. It tells you whether consumers are keeping up.

• Steady and small: healthy.
• Growing steadily: consumers are too slow for the incoming rate. Add consumers (up to the partition count), or make processing faster.
• Growing on one partition only: a hot key or a stuck message on that partition.
• Lag in time (“the oldest unprocessed message is 12 minutes old”) is often more useful to alert on than lag in message count, because it’s what users feel (“my order confirmation SMS came 12 minutes late”).

Pitfalls to Avoid

1. Believing the queue gives exactly-once processing. It gives at-least-once. Duplicates will happen on every crash, deploy and rebalance.
2. Dedupe records stored separately from the side effect. A crash in between breaks it.
3. Acknowledging before processing for data that matters. That’s at-most-once: you’ll lose messages.
4. No message key, or a key that doesn’t match the entity whose order matters.
5. Parallel processing within a partition without thinking about ordering.
6. Increasing Kafka partitions casually on a topic where per-key order matters.
7. No dead-letter queue. One bad message blocks a partition or loops forever.
8. Dumping everything into the DLQ when a dependency is down.
9. Not monitoring consumer lag, especially lag in time.

Quick Recap

• Processing and acknowledging are two steps; the order decides at-most-once vs at-least-once.
• Exactly-once delivery isn’t possible over a network; exactly-once effect = at-least-once + idempotent handlers.
• Real test: a 1% crash rate before acknowledging gave ₹5,500 of extra cashback over 5,000 events; dedupe in the same transaction made it exact.
• Kafka’s exactly-once covers Kafka-to-Kafka processing, not your database or external APIs.
• Order exists only within a partition: key by the entity whose order matters, and make handlers tolerate out-of-order events.
• Use dead-letter queues for poison messages, and watch consumer lag in time.

Next lesson: the related, harder problem of keeping data consistent across services: two-phase commit, sagas, and the transactional outbox, with a real test of what goes wrong when you write to a database and a queue separately.