Idempotency, Retries and Webhooks — Never Charge Twice

Here’s a bug I helped debug at a mid-sized ed-tech company. Some students were getting two enrolment confirmation emails, and a few were getting enrolled in the same batch twice, taking two seats. Not many, maybe 15 a week out of 8,000 payments. It took us a while to find the cause.

The gateway sent the payment.captured webhook. Our handler did a lot of work inline: marked the order paid, created the enrolment, called the email service, updated the CRM. On busy days, this took more than the gateway’s timeout. The gateway didn’t get a 200 in time, so it sent the webhook again. Our handler ran a second time. And because the “is it already paid?” check and the “mark it paid” update were two separate steps, both runs passed the check before either one finished.

Nothing in that code was obviously wrong when you read it line by line. That’s what makes duplicate bugs so painful. This lesson is about building payment code where duplicates simply don’t matter.

Duplicates Are Normal, Not Rare

First, accept this: in payments, you will see the same request or event more than once. It’s not a bug in the gateway, it’s how reliable systems work. When a sender isn’t sure the receiver got a message, it sends it again. Better twice than never.

The goal is idempotency: doing something twice has the same effect as doing it once. Calling “mark order 812 paid by payment C” ten times should leave exactly one paid order, one enrolment, one email.

Let’s handle each case.

Idempotency Keys for Your Own APIs

Any API in your system that creates a money movement (create a refund, create a payout, charge a saved card, top up a wallet) should accept an idempotency key.

The idea:

• The client generates a unique key for each logical action. For example, when an ops person clicks “Refund”, the frontend generates a UUID. If the request times out and the frontend retries, it sends the same key.
• The server stores the key with the result. If it sees the key again, it returns the stored result instead of doing the work again.

Here’s a working version with Postgres. The table:

CREATE TABLE idempotency_keys (
    scope          TEXT NOT NULL,          -- e.g. user id or service name
    key            TEXT NOT NULL,
    request_hash   TEXT NOT NULL,          -- detects the same key reused with a different body
    status         TEXT NOT NULL,          -- 'in_progress' | 'completed'
    response_code  INT,
    response_body  JSONB,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (scope, key)
);

And the logic:

import hashlib
import json

class KeyConflict(Exception): ...
class KeyInProgress(Exception): ...

def run_idempotent(conn, scope: str, key: str, request: dict, work):
    req_hash = hashlib.sha256(json.dumps(request, sort_keys=True).encode()).hexdigest()

    with conn.transaction():
        row = conn.fetchrow("""
            INSERT INTO idempotency_keys (scope, key, request_hash, status)
            VALUES ($1, $2, $3, 'in_progress')
            ON CONFLICT (scope, key) DO NOTHING
            RETURNING key
        """, scope, key, req_hash)

    if row is None:
        # Someone used this key before
        existing = conn.fetchrow("SELECT * FROM idempotency_keys WHERE scope=$1 AND key=$2",
                                 scope, key)
        if existing["request_hash"] != req_hash:
            raise KeyConflict("Idempotency key reused with a different request")
        if existing["status"] == "in_progress":
            raise KeyInProgress("Same request is still running, retry shortly")
        return existing["response_code"], existing["response_body"]

    # We own this key. Do the work.
    code, body = work()

    conn.execute("""
        UPDATE idempotency_keys SET status='completed', response_code=$3, response_body=$4
        WHERE scope=$1 AND key=$2
    """, scope, key, code, json.dumps(body))
    return code, body

Why this works under concurrency: the INSERT ... ON CONFLICT DO NOTHING relies on the primary key. If two requests with the same key arrive at the same time, the database lets exactly one insert succeed. The other sees row is None and backs off.

What if the server crashes after the work but before saving the response? Then the key is stuck in_progress. Add an expiry: if a key has been in_progress for more than a few minutes, a cleanup job checks whether the work actually happened (look up the refund by your reference) and completes or clears the key. The work itself should also be idempotent where possible, which brings us to the next point.

Keys for internal jobs

For background jobs (like the “refund extra payment” job from Lesson 2), there’s no client generating a UUID. Derive the key from business data instead, so the same business action always produces the same key:

key = f"refund:extra_capture:{payment_id}"            # one refund per extra payment, ever
key = f"payout:{seller_id}:{settlement_date}"         # one payout per seller per day

This is the same idea we used in the AI agents course for tool calls: identity comes from the action, not from who happened to call it.

Passing idempotency to the gateway

When you call the gateway to create a refund or payout, check if the gateway supports an idempotency key or a unique reference field on that API. Many payout APIs require a unique reference_id and reject duplicates. If the gateway supports it, always send it.

If it doesn’t, and your request times out, don’t blindly retry. First check whether the refund was created (fetch refunds for the payment, or search by your reference in the notes), and only create it if it doesn’t exist.

Deduplicating Webhooks

Gateways deliver webhooks at least once. You will get duplicates. You might also get them out of order (a payment.failed after a payment.captured for a different attempt, or a refund event before you’ve processed the capture).

The pattern from Lesson 4, in detail:

CREATE TABLE webhook_events (
    gateway        TEXT NOT NULL,
    event_id       TEXT NOT NULL,
    event_type     TEXT NOT NULL,
    payload        JSONB NOT NULL,
    received_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at   TIMESTAMPTZ,
    attempts       INT NOT NULL DEFAULT 0,
    last_error     TEXT,
    PRIMARY KEY (gateway, event_id)
);

def receive_webhook(gateway: str, event_id: str, event_type: str, payload: dict) -> bool:
    row = conn.fetchrow("""
        INSERT INTO webhook_events (gateway, event_id, event_type, payload)
        VALUES ($1, $2, $3, $4)
        ON CONFLICT (gateway, event_id) DO NOTHING
        RETURNING event_id
    """, gateway, event_id, event_type, json.dumps(payload))
    return row is not None          # True only the first time

If your gateway doesn’t send a unique event id, build one from fields that identify the event uniquely, for example f"{event_type}:{payment_id}:{status}".

Storing every webhook also gives you a full history for debugging and audits. When a customer says “I paid at 9:42 PM”, you can see exactly what the gateway told you and when.

Out-of-order events are handled by the state machine in Lesson 6: you only apply a transition if it’s allowed from the current state, and you treat the gateway’s own status (fetched fresh) as the truth.

The Race: Webhook and Browser Callback Together

This is the bug from my ed-tech story. Two processes both run “check if paid, then mark paid”, and both pass the check.

The wrong way:

# DON'T DO THIS
order = db.get_order(order_id)
if order.status != "paid":                 # both requests see 'pending_payment'
    db.update_order(order_id, status="paid")
    create_enrolment(order)                # runs twice
    send_email(order)                      # runs twice

The right way is to make the check and the update one atomic operation, using a conditional UPDATE:

def mark_order_paid_once(order_id: str, payment_id: str) -> str:
    with conn.transaction():
        row = conn.fetchrow("""
            UPDATE orders
               SET status = 'paid', paid_payment_id = $2, paid_at = now()
             WHERE id = $1
               AND status IN ('pending_payment', 'payment_pending', 'failed_final')
            RETURNING id
        """, order_id, payment_id)

        if row is None:
            current = conn.fetchrow("SELECT status, paid_payment_id FROM orders WHERE id=$1",
                                    order_id)
            if current["paid_payment_id"] == payment_id:
                return "already_paid_same_payment"         # duplicate, do nothing
            if current["status"] == "paid":
                schedule_extra_payment_refund(order_id, payment_id)   # Lesson 2 case
                return "extra_payment"
            return "not_payable"                            # e.g. cancelled, needs review

        # We won the race. Record follow-up work in the SAME transaction (outbox below).
        conn.execute("""
            INSERT INTO outbox (event_type, aggregate_id, payload)
            VALUES ('order.paid', $1, $2)
        """, order_id, json.dumps({"order_id": order_id, "payment_id": payment_id}))
    return "marked_paid"

Only one transaction can change the row from pending_payment to paid. The other gets zero rows back and goes to the “already paid” branch. No locks held across network calls, no race.

Notice we don’t send the email or create the enrolment inside this function. We write an outbox row instead. Here’s why.

The Outbox Pattern

After an order is paid, many things need to happen: send a receipt, create the enrolment or shipment, write ledger entries, update analytics. If you do these by calling other services or publishing to a queue after your database commit, you have a “dual write” problem.

If the process crashes after the commit but before publishing, the order is paid but nothing else happens: no email, no shipment. If you publish first and the commit fails, you send a receipt for an order that isn’t paid. Both are real incidents I’ve seen.

The outbox pattern fixes it:

1. In the same transaction as the state change, insert a row into an outbox table.
2. A separate relay process reads unpublished outbox rows and publishes them to your queue (or calls the services), marking them published.
3. If the relay crashes, it just picks up where it left off. Events may be published twice, so consumers must dedupe (by outbox row id or event id).

A simple relay with Postgres:

CREATE TABLE outbox (
    id            BIGSERIAL PRIMARY KEY,
    event_type    TEXT NOT NULL,
    aggregate_id  TEXT NOT NULL,
    payload       JSONB NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at  TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

def relay_once(batch_size: int = 100) -> int:
    with conn.transaction():
        rows = conn.fetch("""
            SELECT id, event_type, payload FROM outbox
             WHERE published_at IS NULL
             ORDER BY id
             LIMIT $1
             FOR UPDATE SKIP LOCKED
        """, batch_size)
        for r in rows:
            queue.publish(topic=r["event_type"], key=str(r["id"]), value=r["payload"])
        if rows:
            conn.execute("UPDATE outbox SET published_at = now() WHERE id = ANY($1)",
                         [r["id"] for r in rows])
    return len(rows)

FOR UPDATE SKIP LOCKED lets you run several relay workers without them picking the same rows. For higher volumes, teams use change data capture (CDC) tools like Debezium to stream outbox rows to Kafka, but the polling version works fine for most products.

On the consumer side, each consumer remembers what it has processed:

def on_order_paid(event_id: str, payload: dict):
    if not processed_events.insert_if_new(consumer="email", event_id=event_id):
        return                                   # seen before
    send_receipt_email(payload["order_id"])

For the ledger consumer (Lesson 7), the dedupe is even simpler: every ledger entry has a unique source_event_id, so a duplicate insert just fails the unique constraint.

Retries: When and How

Retries are necessary, but in payments a wrong retry can mean a double charge. Rules I follow:

1. Know which operations are safe to retry.

• Reads (GET status, fetch payment) are always safe.
• Writes are safe only if they’re idempotent: you’re sending an idempotency key, or the downstream rejects duplicates by reference id.
• A write without idempotency that timed out is not safe to retry. You don’t know if it happened. Check first.

2. Use exponential backoff with jitter.

import random
import time

def with_retries(fn, *, attempts: int = 5, base: float = 0.5, cap: float = 20.0,
                 retry_on=(TimeoutError, ConnectionError)):
    for i in range(attempts):
        try:
            return fn()
        except retry_on:
            if i == attempts - 1:
                raise
            sleep = min(cap, base * 2 ** i)
            time.sleep(random.uniform(0, sleep))      # "full jitter"

Jitter spreads retries out, so a gateway blip doesn’t cause thousands of your workers to retry at exactly the same moment and knock it over again.

3. Retry in one layer only. If your HTTP client retries 3 times, and your job retries 3 times, and the queue redelivers 3 times, that’s up to 27 attempts. Decide where retries live (usually the job/queue level for background work) and turn them off elsewhere.

4. Have a dead-letter path. After the last retry, put the job somewhere a human will look, with an alert. A refund that silently failed after 5 retries is a customer who will call you angry in a week.

Pitfalls to Avoid

1. Assuming duplicates are rare. They’re normal. Design for them from day one.
2. Check-then-update in two steps. Use a conditional UPDATE ... WHERE status IN (...) or a row lock.
3. No idempotency keys on money-moving APIs. Refunds, payouts, charges: all need them.
4. Random keys for background jobs. Derive keys from business data so the same action always gets the same key.
5. Blindly retrying a timed-out write. Check whether it happened first.
6. Doing side effects inside the webhook request. Store the event, return 200, process from a queue.
7. Dual writes (update DB, then publish). Use an outbox in the same transaction.
8. Consumers that don’t dedupe. The relay can publish twice; consumers must handle it.
9. Retries at every layer. Multiply out of control. Pick one layer.
10. No dead-letter queue and alert. Failed money operations must reach a human.

Quick Recap

• Duplicates are normal: double clicks, client retries, webhook redelivery, races, queue redelivery.
• Idempotency keys with a unique constraint and stored response for money-moving APIs.
• Dedupe webhooks by event id with insert-if-new.
• Mark paid with one atomic conditional update; extra captures go to the refund path.
• Outbox pattern so state changes and their events commit together; consumers dedupe.
• Retry only safe operations, with backoff and jitter, in one layer, with a dead-letter path.

Next lesson: we’ve been using statuses like payment_pending and failed_final loosely. Let’s design proper state machines for payments and orders, and handle “pending” in a way that keeps customers calm.