Idempotency in Distributed Systems — Why Your Retry Just Charged the Customer Twice

Here is a bug that has cost real companies real money: a customer taps “Pay ₹500”, sees a spinner, then an error, taps again — and gets charged ₹1000. Nobody wrote a bug that charges twice. The code charged once, correctly, each time it ran. The problem is that it ran twice, because a retry happened, and the server treated the retry as a brand-new payment. This is the problem idempotency solves, and it’s one of those topics that sounds academic until it quietly corrupts your data or double-charges your users. Let’s make it concrete.

The Bug: Retries Are Normal, and They Duplicate

In a distributed system, retries aren’t an edge case — they’re constant. A client times out and retries. A load balancer re-sends. A mobile network drops the response. A message queue redelivers. Your API will receive the same logical request more than once, and if it treats each copy as new, you get duplicates.

Walk through the sequence. The client sends POST /pay ₹500. The server charges the card — charge #1 succeeds — and sends back a success response. But that response is lost (a timeout, a dropped connection, a dead Wi-Fi moment). From the client’s point of view, the request failed: it never got an answer. So the client does the sensible, safe-looking thing and retries. The server, having no memory that it already did this, charges again — charge #2 — and the customer is out ₹1000 for one purchase.

Notice there’s no villain here. The client behaved correctly (retrying a request with no response is standard). The server behaved “correctly” too (it charged a card when asked). The failure is architectural: the system had no way to recognise that the second request was the same operation as the first. That recognition is exactly what idempotency provides.

The word itself: an operation is idempotent if doing it many times has the same effect as doing it once. Turning off a light switch is idempotent (off is off, however many times you press it); adding ₹500 to a balance is not (each press adds more). The goal of this article is to make the second kind behave like the first.

The Fix: Idempotency Keys

The standard solution is an idempotency key: a unique identifier the client attaches to a logical operation, and crucially reuses on every retry of that same operation. The server uses the key to recognise duplicates.

The flow:

1. The client generates a unique key (a UUID) for the operation — once — and sends it, e.g. in an Idempotency-Key: abc123 header. If the request times out and the client retries, it sends the same key abc123 again. (This is the critical detail: the key identifies the operation, not the attempt.)
2. The server, before doing the work, checks a dedup store (a database table, Redis, etc.): have I seen key abc123 before?
   • No → do the work (charge the card), and store the result under abc123.
   • Yes → skip the work entirely and return the stored result.
3. Either way, the client gets the same success response — but the card is charged exactly once.

The subtle but essential requirement is that the check-and-store must be atomic. If two copies of the request arrive at the same instant (which happens), you can’t have both check “seen abc123? no” and then both proceed. Enforce it at the database level — a unique constraint on the key, or an atomic “insert if not exists” — so that exactly one request wins the race and does the work, and the other sees the key already present. A naive “read, then write” in application code has a race window that will, under load, let duplicates through.

A few practical notes on keys:

• The client generates the key, not the server — because the whole point is that the same key survives across retries, and only the client knows two attempts are the same operation.
• Store the response, not just the key. When a duplicate arrives, you want to return the original result (the same payment id, the same success), so the client’s retry gets a sensible answer rather than an error.
• Give keys a TTL. You don’t need to remember every key forever — a retention window (hours to days, matching how long retries could plausibly arrive) keeps the dedup store from growing without bound.
• Scope the key correctly. A key should identify one logical operation for one user; don’t let two different operations accidentally share a key, or you’ll suppress a legitimate second action.

Delivery Semantics: Why “Exactly-Once” Is a Myth (Sort Of)

Idempotency is really a response to a deep truth about distributed systems, and it’s worth understanding the vocabulary, because you’ll meet it constantly in message queues (Kafka, SQS, RabbitMQ) and APIs.

There are three “delivery semantics”:

• At-most-once — send it, never retry. Fast and simple, but if anything fails, the message is lost. Acceptable for, say, non-critical metrics; unacceptable for a payment.
• At-least-once — keep retrying until you get an acknowledgement. You never lose a message, but you may deliver duplicates (because a lost ack looks identical to a lost message, so you retry something that actually succeeded). This is the common, sensible default for important work.
• Exactly-once — never lost, never duplicated. What everyone wants.

Here’s the key insight: true exactly-once delivery is essentially impossible in a distributed system, because you fundamentally cannot distinguish “the message was lost” from “the message arrived but the acknowledgement was lost.” Faced with silence, you must either retry (risking a duplicate) or not (risking a loss). There’s no third option at the delivery layer.

So the industry doesn’t chase exactly-once delivery. It aims for exactly-once effect: accept that delivery is at-least-once (messages may arrive more than once), and make the processing idempotent so that a duplicate changes nothing. At-least-once delivery + an idempotent handler = the effect of exactly-once, which is what you actually cared about. When a vendor advertises “exactly-once,” this combination — dedup plus idempotent processing — is almost always what’s happening under the hood. Understanding this saves you from chasing a guarantee that doesn’t exist and instead building the one that does.

Which Operations Are Already Safe?

Good news: many operations are naturally idempotent, and you only need to do extra work for the ones that aren’t. HTTP method semantics are a useful guide (and a reason to design APIs along them).

• GET — a read changes nothing, so it’s trivially idempotent. Retry freely.
• PUT — sets a resource to a specific value (“set status to paid”). Doing it twice leaves the same state, so it’s idempotent. (This is why PUT is preferred over POST when you can express the operation as “make it be this.“)
• DELETE — after the first delete the resource is gone; deleting again leaves it gone. Idempotent (return the same “deleted / not found” either way).
• POST — the troublemaker. A POST /orders creates a new order each time; POST /pay or POST /transfer moves money again each time. These are not idempotent, and they’re exactly the operations where a duplicate hurts most.

So the rule of thumb: lean on PUT/DELETE semantics where you can, and for the genuinely non-idempotent operations (creates, payments, transfers), add protection:

• An idempotency key (above) — the general solution, ideal for client-driven operations like payments. (This is exactly what Stripe and other payment APIs expose, for this exact reason.)
• A natural uniqueness constraint — sometimes the operation is naturally unique on some business key. “At most one payment per order_id” can be enforced with a unique index on order_id in the payments table; the second insert fails cleanly, and you treat that as “already paid.” When you have a natural key, this is simpler and more robust than a separate dedup store, because the database enforces it for you.

Where Idempotency Shows Up

This isn’t just a payments concern — it’s everywhere once you look:

• Message consumers. Any queue with at-least-once delivery (most of them) will redeliver. Your consumer must be idempotent — process a message, record that you’ve processed its id, and skip duplicates — or every redelivery double-counts.
• Webhooks. Providers retry webhooks until you return 200, so you’ll receive duplicates. Dedup on the event id.
• Background jobs and retries. A job runner that retries a failed job must not double-apply the parts that did succeed before the failure. Make job steps idempotent, or checkpoint progress.
• Distributed transactions / sagas. Each step in a saga may be retried, so each step (and its compensation) should be idempotent.

The unifying mental model: in a distributed system, assume everything can happen more than once, and design so that it doesn’t matter. That assumption, baked into your handlers from the start, is far cheaper than discovering duplicates in production via an angry customer.

Pitfalls to Avoid

1. Treating a retry as a new request. Retries are normal and constant; an API that doesn’t expect them will duplicate. Design for at-least-once from the start.
2. The server generating the idempotency key. Then retries get different keys and the dedup fails. The client generates one key per operation and reuses it across retries.
3. A non-atomic check-and-store. “Read, then write” has a race window that lets concurrent duplicates through. Enforce uniqueness atomically in the database.
4. Storing only the key, not the result. A duplicate should return the original response (same id, same success), not an error. Store the response too.
5. Chasing exactly-once delivery. It doesn’t exist. Aim for exactly-once effect: at-least-once delivery + idempotent processing.
6. Forgetting consumers and webhooks. Idempotency isn’t only an API concern; queue consumers and webhook handlers get duplicates too. Dedup on message/event ids.
7. Keys that never expire. Give the dedup store a TTL matched to your retry window, or it grows forever.

Quick Recap

• A retry is not a new request, but naive servers treat it as one — turning one payment into two charges.
• Fix it with an idempotency key: the client sends a unique key per operation and reuses it on retries; the server does the work once, atomically stores the result, and returns the stored result for duplicates.
• You can’t have exactly-once delivery (a lost message looks like a lost ack), so aim for exactly-once effect = at-least-once delivery + idempotent handler.
• GET/PUT/DELETE are idempotent by nature; POST (create, pay, transfer) is not — protect it with an idempotency key or a natural uniqueness constraint in the DB.
• Idempotency applies everywhere duplicates can occur: APIs, queue consumers, webhooks, jobs, sagas. Assume everything happens more than once, and make it not matter.

If you’re building payment or ordering systems, idempotency isn’t optional — it’s the difference between a reliable system and one that quietly double-charges under load. Build it in before you need it; you’ll never see the bug you prevented.