Distributed Transactions

In a single database, a transaction is a gift: BEGIN, do several writes, COMMIT, and either all of them happen or none of them do. The database guarantees atomicity for you. The moment your operation spans two databases or two services — charge a card and reserve inventory and create an order, each owned by a different service — that gift is gone. There is no BEGIN that wraps three independent systems. This lesson is about getting correctness back without it.

The honest framing for an interview: distributed transactions are hard, and the senior move is usually to avoid needing one — but when you cannot, you reach for a Saga, not a global lock.

In the interview: if a design has one logical operation touching several services, do not hand-wave “it’s transactional.” Name the approach: a Saga with compensating actions, made reliable with the Outbox pattern and idempotent consumers. Reserve two-phase commit for the rare case where you genuinely control all participants and volume is low.

The problem: no shared transaction

Consider an e-commerce checkout that must: (1) create an order, (2) charge the customer, (3) reserve inventory. In a monolith with one database, this is a single transaction — trivially atomic. Split those into an Order service, a Payment service, and an Inventory service, each with its own database, and you have three separate commits that can each succeed or fail independently.

The failure modes are the whole problem. The payment succeeds but inventory is out of stock. The order is created but the charge times out. A service crashes after step two and never reaches step three. Without coordination you get partial failure: money taken with no order, inventory reserved for an order that never got paid, or an order that exists but was never charged. Every one of these is a real bug that a user or an accountant will notice.

Two-phase commit (2PC)

The textbook way to make a multi-participant commit atomic is two-phase commit. A coordinator drives two rounds across all participants.

1. Prepare phase. The coordinator asks every participant “can you commit this?” Each does the work tentatively, locks the relevant rows, writes to its log, and votes yes or no — promising that if it voted yes, it will be able to commit.
2. Commit phase. If every participant voted yes, the coordinator tells all of them to commit. If any voted no (or timed out), it tells all of them to abort.

This is correct — it genuinely gives you atomicity across participants. But it has serious problems at scale:

• It blocks. Participants hold locks from the time they vote yes until they hear the final decision. Under contention, this serializes work and kills throughput.
• The coordinator is a single point of failure. If it crashes after participants have voted yes but before sending the decision, those participants are stuck holding locks, unsure whether to commit or abort. This is the infamous “blocking” window.
• It couples availability. The transaction can only commit if every participant and the coordinator are up at the same time. More participants means lower combined availability.

For these reasons, 2PC is largely avoided across independent services in high-scale systems. It shows up inside a single distributed database (which controls all participants and can bound the blocking window), but reaching for 2PC across your own microservices in an interview is usually the wrong answer.

The Saga pattern

The practical alternative is the Saga: instead of one atomic transaction, run a sequence of local transactions, each committing independently in its own service, and define a compensating action for each step that undoes its effect if a later step fails.

For the checkout: create the order (compensation: cancel it), charge the card (compensation: refund), reserve inventory (compensation: release it), ship. Each step commits locally and emits an event that triggers the next. If the inventory reservation fails, the saga runs the compensations in reverse — refund the charge, cancel the order — leaving the system in a consistent state.

The key mental shift: a saga does not give you atomicity (there are brief windows where the order exists but is not yet paid). It gives you eventual consistency with a guarantee that the system ends in a valid state — either fully completed or fully compensated. For most business workflows that is exactly the right trade, because the compensations map to real business operations you need anyway (refunds, cancellations).

Orchestration vs choreography

Sagas come in two coordination styles:

• Orchestration — a central orchestrator (often a workflow engine like Temporal or AWS Step Functions) explicitly invokes each step, tracks progress, and triggers compensations on failure. Easy to reason about and to observe; the orchestrator is a component you must run.
• Choreography — there is no central brain. Each service listens for events and reacts by doing its step and emitting the next event. More decoupled, but the overall flow is implicit and emergent, which makes it harder to follow and debug as steps multiply.

A good rule: choreography for two or three simple steps, orchestration once the workflow has branches, retries, timeouts, or more than a handful of steps.

The dual-write problem and the Outbox pattern

Sagas depend on a subtle guarantee that is easy to get wrong: each step must commit its local change and publish its event atomically. Doing them as two separate operations is the dual-write problem:

# BROKEN: two independent operations that can half-fail
db.commit(order)            # succeeds
broker.publish(order_event) # crashes here → event lost, saga stalls

If the process dies between the two lines, the database has the order but the broker never got the event — the saga silently stalls, and the order sits in limbo forever. Writing the event first has the mirror problem: an event for a change that never committed.

The fix is the Outbox pattern. Write the event into an outbox table in the same database transaction as the business change:

BEGIN
  INSERT INTO orders (...)            -- the business write
  INSERT INTO outbox (event, ...)     -- the event, same transaction
COMMIT

Because both inserts are in one local transaction, they are atomic — you can never have one without the other. A separate relay process (polling the outbox table, or tailing the database’s change log via Change Data Capture) reads new outbox rows and publishes them to the broker, marking them sent. If the relay crashes mid-publish, it simply retries on restart, which is why consumers must be idempotent (next section). The outbox turns an impossible atomic dual-write into a reliable “commit locally, publish eventually.”

Idempotency and exactly-once effects

Message delivery in real systems is at-least-once: the broker, the outbox relay, and retrying clients can all deliver the same event more than once. True exactly-once delivery is effectively impossible across a network. What you can achieve is exactly-once effects — processing a duplicate has no additional impact — by making consumers idempotent.

The standard technique is an idempotency key: a unique id carried on each event. Before acting, the consumer checks whether it has already processed that id (a processed_events table, or a dedup set); if so, it skips. For operations that are naturally idempotent (setting a status, upserting a row) you may not even need the key. For dangerous ones (charging a card, decrementing stock) the key is mandatory.

def handle(event):
    if seen(event.id):        # already processed this exact event
        return                # safe no-op on a duplicate
    apply(event)
    mark_seen(event.id)       # ideally in the same transaction as apply()

With idempotent consumers, at-least-once delivery is safe: duplicates are harmless, and the system converges to the correct result. This is the combination interviewers want to hear — at-least-once delivery plus idempotent consumers equals effectively exactly-once.

Putting it together

A robust multi-service workflow at scale almost always looks the same: a Saga (orchestrated once it gets complex) drives a sequence of local transactions; each step uses the Outbox pattern so its commit and its event are atomic; and every consumer is idempotent so retries and duplicates are safe. Two-phase commit stays on the shelf unless you control all participants and can tolerate its blocking. Say that out loud and you have demonstrated real distributed-systems judgment.

Key Takeaways

1. Spanning services means losing the single transaction. A logical operation across multiple databases has no shared COMMIT; partial failure (money taken, no order) is the core risk you must design against.

2. Avoid two-phase commit across services. It is correct but blocks on locks, makes the coordinator a single point of failure, and couples the availability of every participant. It belongs inside a distributed database, not across your microservices.

3. Use a Saga. Run a sequence of local transactions, each with a compensating action that undoes it if a later step fails. You trade atomicity for eventual consistency that always ends in a valid state — fully done or fully compensated.

4. Orchestration vs choreography. A central orchestrator (e.g. Temporal) for complex, branching workflows; event-driven choreography for a couple of simple steps.

5. Solve the dual-write problem with the Outbox pattern. Write the event to an outbox table in the same transaction as the business change, then relay it to the broker — so the commit and the event are atomic.

6. Make consumers idempotent. At-least-once delivery plus an idempotency key (dedup on a unique event id) gives exactly-once effects, which is the achievable and correct goal.