Timeouts, Retries and Idempotency

Here’s a bug I’ve seen in some form at three different companies.

A cashback service listens for “payment successful” events and credits ₹50 to the user’s wallet. The wallet service is sometimes slow. So the cashback service has a timeout of 2 seconds and, being careful, retries three times on timeout. One evening the wallet database has a slow few minutes. Calls take 2.5 seconds. Every call “times out”, gets retried, also “times out”, gets retried again. Each of those calls actually succeeded, just slowly. Some users got ₹200 instead of ₹50. Finance found out at month end.

The retry logic was written with good intentions. The missing piece was idempotency. This lesson is about getting timeouts, retries and idempotency right together, because any one of them alone makes things worse.

A Timeout Means “Unknown” (Real Test)

I said in Lesson 1 that a timeout doesn’t mean failure. Let me show you with a real database.

I took the 3-node etcd cluster from the lab setup and paused two of the three nodes with kill -STOP, so the leader couldn’t reach a majority. Then I sent a write to the leader with a 3-second timeout:

write on leader, 1 of 3 up:  FAIL: TimeoutError  (3.00 s)
... resume the two paused nodes ...
after resume read:           Mumbai

The client was told the write failed. But the leader had already written it to its log. When the other nodes came back, the entry reached a majority, got committed, and became the real value. If the client code had shown “update failed, please try again” to the user, and the user had changed their mind and not retried, the database would still contain the “failed” update.

This isn’t a quirk of etcd. Postgres behaves the same way if the connection drops after you send COMMIT but before the reply arrives. Payment gateways, Kafka producers and cloud APIs all have this window. Any time a request has left your machine, the work may happen whether or not you hear back.

So what do you do with an unknown outcome? There are only three honest options:

1. Retry, safely. Make the operation idempotent, so doing it twice has the same effect as doing it once.
2. Ask. Call a status API (“what happened to request X?”) and act on the answer.
3. Give up and reconcile later. Mark it as “pending”, and have a background job fix it up by comparing records (very common in payments).

What you must never do is assume: neither “it failed” nor “it worked”.

Choosing Timeouts

Many outages start with a missing timeout. The default for many HTTP clients is “wait forever” or something huge like 60-120 seconds. One slow dependency then holds every thread in your service, and you’re down too.

Rules of thumb that have worked for me:

• Every network call gets an explicit timeout. HTTP clients, database drivers, Redis clients, gRPC calls. Check the defaults of your libraries; you’ll be surprised.
• Base it on the dependency’s real latency. Look at its p99.9 over a normal week, and set the timeout a bit above it. If p99.9 is 300 ms, a timeout of 500 ms-1 s is reasonable. A timeout of 30 s is not a timeout, it’s a hope.
• Separate connect and read timeouts. Connecting should be fast (100-500 ms within a data centre). A connect timeout of 10 s means you wait 10 s for a dead pod.
• Use a deadline for the whole request, and pass it down. If the user-facing API has 3 seconds in total, and 2.5 seconds are already used, there’s no point calling the next service with its own fresh 2-second timeout. gRPC does this for you (deadline propagation); with HTTP, pass a header like X-Request-Deadline and have each service respect it.

import time, requests

def call_with_deadline(url, payload, deadline):
    remaining = deadline - time.monotonic()
    if remaining <= 0.05:                     # not enough time left to be useful
        raise TimeoutError("deadline exceeded before calling " + url)
    return requests.post(url, json=payload, timeout=(0.3, remaining))  # (connect, read)

deadline = time.monotonic() + 3.0             # the whole API call gets 3 seconds

Notice time.monotonic(), not time.time(). The wall clock can jump (we’ll see why in the next lesson); the monotonic clock only moves forward, so it’s the right one for measuring durations.

When to Retry (and When Not To)

Retries are useful because many failures are temporary: a dropped connection, a pod being replaced, a leader election (which, as we’ll measure in Lesson 6, takes about a second in etcd). But retrying the wrong thing makes it worse.

Retry:

• Connection errors, connection resets, DNS hiccups.
• Timeouts, if the operation is idempotent.
• HTTP 502, 503, 504, and 429 (respecting Retry-After).
• Database errors that say so: serialization failures (40001), deadlocks (40P01), “leader changed” errors.

Don’t retry:

• 400, 401, 403, 404, 422: the request is wrong, and sending it again won’t fix it.
• Non-idempotent operations without an idempotency key.
• When the deadline has passed. Nobody is waiting for the answer any more.
• Most 500s from your own code: if it crashed on this input, it’ll crash again.

Backoff and Jitter

If a service is struggling and every client retries immediately, the retries arrive exactly when the service is weakest. Worse, if a thousand clients all failed at the same moment (because the service paused), a fixed retry delay makes them all come back at the same moment again. This is the “thundering herd”.

The fix is two things together:

• Exponential backoff: wait longer after each failure (for example, up to 1 s, then 2 s, then 4 s).
• Full jitter: pick a random wait between zero and that limit, so clients spread out.

In my simulation of 1,000 clients, the fixed delay sends all 1,000 retries in the same 0.2-second window, three times over. With backoff and full jitter, the busiest 0.2 seconds sees 290, and the rest are spread across several seconds.

A small, correct implementation:

import random, time

RETRYABLE = (ConnectionError, TimeoutError)

def with_retries(fn, attempts=3, base=0.5, cap=8.0, deadline=None):
    for attempt in range(attempts + 1):
        try:
            return fn()
        except RETRYABLE:
            if attempt == attempts:
                raise
            sleep = random.uniform(0, min(cap, base * 2 ** (attempt + 1)))   # full jitter
            if deadline is not None and time.monotonic() + sleep >= deadline:
                raise
            time.sleep(sleep)

Most good libraries already do this (tenacity in Python, resilience4j in Java, the AWS SDKs, gRPC retry policies). Use them; just check their defaults.

One important warning, which we’ll come back to in Lesson 11 with a full simulation: jitter doesn’t reduce the total number of retries. Three retries still means up to 4x the load on a struggling service. In that simulation, backoff with jitter alone did not save a service from collapse. What saved it was limiting retries (a retry budget) and having the server drop requests the client had already given up on.

Retry amplification across layers

Retries multiply across layers. Suppose:

mobile app  -> retries 3 times
API gateway -> retries 3 times
order svc   -> retries 3 times when calling payment svc

One user tap can become 4 × 4 × 4 = 64 calls to the payment service when it’s already slow. The usual rule: retry at one layer only, usually the one closest to the failure that knows whether the operation is safe to repeat. Other layers should fail fast and pass the error up.

Idempotency: Making Retries Safe

An operation is idempotent if doing it many times has the same effect as doing it once.

Some operations are naturally idempotent:

• SET status = 'shipped' WHERE order_id = 42 (but not SET stock = stock - 1)
• PUT /users/42/address with the full address
• DELETE /cart/items/7
• An upsert keyed by a unique ID: INSERT ... ON CONFLICT (event_id) DO NOTHING

Others are not: “charge ₹499”, “credit ₹50 cashback”, “send OTP”, “increment counter”, “create order”. For these, you need an idempotency key.

The idea:

1. The client generates a unique key (a UUID) once per user intent, for example when the user taps “Pay”. Every retry of that same intent sends the same key.
2. The server stores the key with the result of the operation.
3. If the same key arrives again, the server doesn’t redo the work. It returns the saved result.

Here’s how I implement it with Postgres. The table:

CREATE TABLE idempotency_keys (
    key           text PRIMARY KEY,
    request_hash  text NOT NULL,
    status        text NOT NULL,          -- 'processing' | 'done'
    response      jsonb,
    created_at    timestamptz NOT NULL DEFAULT now()
);

And the handler logic:

import hashlib, json

def handle_payment(db, idem_key, body):
    req_hash = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO idempotency_keys (key, request_hash, status) VALUES (%s, %s, 'processing') "
            "ON CONFLICT (key) DO NOTHING RETURNING key", (idem_key, req_hash)).fetchone()
        if not inserted:
            row = db.execute("SELECT request_hash, status, response FROM idempotency_keys "
                             "WHERE key = %s", (idem_key,)).fetchone()
            if row.request_hash != req_hash:
                return 422, {"error": "idempotency key reused with a different request"}
            if row.status == "processing":
                return 409, {"error": "request still in progress, retry later"}
            return 200, row.response                      # replay the saved answer
        payment = create_payment(db, body)                 # the real work, same transaction
        response = {"payment_id": payment.id, "status": payment.status}
        db.execute("UPDATE idempotency_keys SET status = 'done', response = %s WHERE key = %s",
                   (json.dumps(response), idem_key))
    return 201, response

The details that matter:

• Same transaction. The key and the business change (the payment row, the wallet credit) are committed together. If the process crashes halfway, both roll back, and a retry starts fresh. If you store the key in Redis and the payment in Postgres, a crash between the two leaves them disagreeing.
• The unique constraint does the locking. Two retries arriving at the same moment both try to insert; the database lets only one win. No need for a separate lock.
• Check the request hash. If the same key comes with a different amount, it’s a client bug. Refuse it loudly.
• Keep keys long enough. Longer than the longest time a client might retry. Stripe keeps them for 24 hours; for internal event processing, I usually keep a few days.
• The key must come from the client. If the server generates the key after receiving the request, a retry gets a new key, and you’ve protected nothing.
• Don’t build the key from the request content. Two genuine ₹500 mobile recharges a minute apart are two different intents, even if the bodies are identical.

When the work involves another system

What if “the work” is calling an external payment gateway, which can’t be inside your database transaction? Then you split it into steps and use the gateway’s own idempotency support:

1. Insert the key and a payment row with status initiated, commit.
2. Call the gateway, passing your payment ID as their idempotency key or reference ID (most Indian payment gateways accept a merchant reference for exactly this purpose).
3. Update the row with the result, commit.

If you crash after step 2 but before step 3, the payment is stuck in initiated. A retry (or a background job) calls the gateway again with the same reference, and the gateway returns the existing result instead of charging twice.

The status-check pattern

For money movement, many systems don’t retry the action at all after a timeout. They ask. UPI is the classic example: when a payment request times out, the PSP doesn’t send a new debit request. It calls the status API for that transaction ID, again and again, until the answer is definite: success, failure, or “deemed” status after a cut-off. Then a daily reconciliation file from the bank settles anything still unclear.

This pattern works whenever the downstream system can look up a request by an ID that you chose:

timeout on "debit txn T123"
  -> GET status of T123
       SUCCESS -> continue
       FAILED  -> safe to retry with a NEW txn id, or fail the order
       PENDING / UNKNOWN -> wait, ask again with backoff; don't retry the debit

Pitfalls to Avoid

1. No timeout, or a huge default timeout. One slow dependency will hang your whole service.
2. Treating a timeout as failure. Showing “payment failed” when the payment may have succeeded.
3. Retrying non-idempotent operations without an idempotency key. That’s how users get charged twice.
4. Retrying at every layer. 3 retries at 3 layers = up to 64 calls per user action.
5. Retrying non-retryable errors like 400s and validation errors.
6. Fixed retry delays without jitter. Synchronised retries hit the server in waves.
7. Storing idempotency keys separately from the business data. A crash in between breaks the guarantee.
8. Server-generated or content-derived idempotency keys. Neither identifies the user’s intent.
9. Measuring timeouts with the wall clock. Use a monotonic clock.

Quick Recap

• After a timeout, the result is unknown. Real test: an etcd write that “failed” with a timeout was committed later.
• Every network call needs a timeout based on the dependency’s real tail latency, and a deadline should flow down the call chain.
• Retry only retryable errors, at one layer, with exponential backoff and full jitter.
• Make retried operations idempotent: client-generated keys, stored in the same transaction as the change, replaying the saved response.
• For money, ask for the status rather than retrying the action, and reconcile.

Next lesson: time. Why you can’t trust timestamps from different machines to order events, how that silently loses data, and what Lamport and vector clocks do about it.