Transactions Across Services: 2PC, Sagas and the Outbox

Here’s a bug I helped track down in an e-commerce backend. Customers would occasionally complain that they’d paid, got an order confirmation in the app, and then… nothing. The order never reached the warehouse.

The order service did two things when an order was placed: insert a row into its Postgres database, then publish an order_placed event to Kafka for the warehouse service. In between those two lines of code, a few times a day, something happened: a pod got killed during a deploy, a Kafka broker was briefly unreachable and the retry gave up, or the process ran out of memory. The order was in the database. The event was never sent. The warehouse never heard about it.

The fix was not “retry harder”. The fix was to stop pretending that two separate writes could behave like one. That’s what this lesson is about.

The Dual-Write Problem

Any time one operation needs to update two systems that don’t share a transaction, you have a dual write:

• Insert into the database, then publish to a message broker.
• Update the database, then update a search index (Elasticsearch).
• Update the database, then invalidate a cache.
• Charge the payment gateway, then update your order table.

Whatever order you do them in, a failure between the two leaves them disagreeing. I measured this with a real setup: SQLite as the database, Redis Streams as the message broker, 10,000 orders, and a simulated crash injected with 2% probability between the two steps.

DB first, then publish:  orders=10000 events=9819  orders_without_event=181 events_without_order=0   duplicate_events=0
publish first, then DB:  orders=9802  events=10000 orders_without_event=0   events_without_order=198 duplicate_events=0
transactional outbox:    orders=9792  events=9989  orders_without_event=0   events_without_order=0   duplicate_events=197

• Commit first, then publish: 181 orders that nobody downstream knows about. Paid orders that never get cooked.
• Publish first, then commit: 198 events for orders that don’t exist (the DB insert failed or rolled back after the event went out). The restaurant cooks food for an order that was never placed.
• Transactional outbox: every order has exactly its event, nothing lost or orphaned. The price: 197 events published twice, which consumers handle with deduplication (Lesson 9).

(The order and event counts differ between runs because crashes in the outbox case happen before commit, so those orders simply never exist. What matters is that the database and the events always agree.)

Two things are worth noticing. First, 2% is a lot; real crash rates are much lower. But even at 0.01%, a service handling 10 lakh orders a day loses 100 events a day, and nothing ever fixes them. Second, “retry the publish” doesn’t help in the first case: the crash kills the process, and with it the retry.

The Transactional Outbox

The idea is simple: don’t do two writes. Do one write to the database, and let the database transaction cover the event too.

1. In the same transaction as the business change, insert a row into an outbox table describing the event.
2. A separate relay process reads unsent rows from the outbox, publishes them to the broker, and marks them as sent.
3. If the relay crashes after publishing but before marking a row as sent, it publishes it again later. That’s the source of the duplicates in my test, and it’s fine, because consumers deduplicate by event_id.

Since the order row and the outbox row commit together, there are only two possible states: both exist, or neither does. The event is guaranteed to be published eventually, as long as the relay keeps running.

The table and the write:

CREATE TABLE outbox (
    id           bigserial PRIMARY KEY,
    event_id     uuid NOT NULL UNIQUE,
    aggregate_id text NOT NULL,            -- e.g. the order id, used as the message key
    event_type   text NOT NULL,
    payload      jsonb NOT NULL,
    created_at   timestamptz NOT NULL DEFAULT now(),
    sent_at      timestamptz
);

BEGIN;
INSERT INTO orders (id, customer_id, amount_paise, status) VALUES (81, 42, 49900, 'PLACED');
INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), '81', 'order_placed', '{"order_id": 81, "amount_paise": 49900}');
COMMIT;

Two ways to build the relay

Polling: a small worker that loops:

SELECT id, event_id, aggregate_id, event_type, payload
FROM outbox
WHERE sent_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;          -- several relay instances can run without stepping on each other
-- publish each row to Kafka with key = aggregate_id, wait for the broker ack
-- UPDATE outbox SET sent_at = now() WHERE id = ANY(...); COMMIT;

Simple and easy to understand. Costs a query every few hundred milliseconds, and adds that much delay.

Change data capture (CDC): a tool like Debezium reads the database’s write-ahead log (Postgres) or binlog (MySQL), sees new outbox rows, and publishes them to Kafka. No polling load on the database, very low delay, and events come out in commit order. Debezium even has a built-in “outbox event router” for exactly this pattern. The cost is running Kafka Connect and Debezium, and learning their failure modes (like a CDC replication slot holding WAL when the connector is down, which can fill the database’s disk).

Whichever you choose:

• Key the published message by the aggregate ID (order_id) so events for one order stay in order (Lesson 9).
• Delete or archive sent rows after a few days, or the outbox grows forever.
• Alert on the age of the oldest unsent row. If the relay is stuck, events silently stop flowing, and nothing else will tell you.

The inbox: the same idea on the consumer side

The consumer-side counterpart is sometimes called the inbox pattern: store the event_id of every processed event in the same transaction as the side effects. That’s exactly the dedupe table from Lesson 9’s cashback test. Outbox on the producer, inbox on the consumer, and you get exactly-once effect across services.

Two-Phase Commit (2PC)

Isn’t there a way to make two systems commit atomically? Yes: two-phase commit. A coordinator talks to all participants:

1. Prepare: “Can you commit this transaction?” Each participant does all the work, writes it durably, locks what it needs, and replies “yes” (promising it will commit if asked) or “no”.
2. Commit: if everyone said yes, the coordinator tells everyone to commit. If anyone said no, it tells everyone to abort.

It works, and it’s atomic. The problem is what happens when the coordinator fails after the prepare phase. Each participant has promised to commit if told to, so it can’t abort on its own. But it doesn’t know whether the others said yes, so it can’t commit on its own either. It must wait, holding its locks, until the coordinator comes back. Every other transaction that touches those rows waits too.

That’s why 2PC is common inside a database system, where the coordinator itself is replicated with consensus and recovers quickly (Spanner, CockroachDB, and the internals of Kafka transactions all use forms of it). Between independent services owned by different teams, over HTTP, with no shared recovery process, it’s rarely used. Many services and brokers simply don’t support taking part in it.

Sagas: Local Transactions With Compensations

For business processes that span services (place order → reserve stock → charge payment → assign rider), the usual approach is a saga: a sequence of local transactions, each in its own service, where each step has a compensating action that semantically undoes it if a later step fails.

The System Design Masterclass lesson on sagas covers orchestration vs choreography in detail. Here are the distributed-systems points that usually get missed:

• Compensations are not rollbacks. A refund is a new transaction. The customer may see a debit and a credit on their bank statement. Some things can’t be compensated at all (an SMS already sent, food already cooked), so put those steps last, after everything that can fail.
• There is no isolation. Between steps, other requests can see the half-done state: an order that’s PENDING, stock that’s reserved but not yet paid for. Design explicit intermediate statuses, and make sure the rest of the system handles them (“Payment processing…” in the app, not “Order failed”).
• Every step and every compensation must be idempotent. Each step is triggered by a message or a call that will be retried (Lessons 2 and 9). A “refund” compensation that runs twice refunds twice.
• The saga’s own progress must be durable. An orchestrator that keeps saga state only in memory loses track on restart. Store it in a database (or use a workflow engine like Temporal, which is built for exactly this).
• Each step’s “publish the next event” is a dual write. Use the outbox for it.

Reconciliation: The Last Line of Defence

Even with outboxes, idempotency and sagas, some things slip through: a bug, a manual database fix, a partner system that lost a callback. Every serious payment and commerce system therefore runs reconciliation: periodic jobs that compare two sources of truth and flag or fix differences.

• Orders in PAID state vs successful payments in the gateway’s settlement report.
• Payments marked PENDING for more than 30 minutes: ask the gateway for their status.
• Orders in the order DB vs orders the restaurant service knows about.
• Wallet balances vs the sum of the wallet ledger entries.

It’s not glamorous, but in my experience reconciliation finds real money every month. Design for it from day one: keep IDs that let you match records across systems (your order ID in the gateway’s reference field, for example), and store enough history to compare.

Pitfalls to Avoid

1. Writing to the database and publishing to a broker as two separate steps. Real test: 181 lost events or 198 orphan events per 10,000 at a 2% crash rate.
2. “Fixing” dual writes with retries. A crashed process doesn’t retry.
3. An outbox without monitoring. A stuck relay silently stops all events.
4. Forgetting that the outbox produces duplicates. Consumers must dedupe (the inbox).
5. Using 2PC between microservices. A coordinator failure blocks everyone holding locks.
6. Sagas without idempotent steps and compensations.
7. Irreversible steps early in a saga. Put SMS, emails and anything you can’t undo at the end.
8. No reconciliation. Something will slip through; make sure you find it before customers do.

Quick Recap

• Writing to two systems without a shared transaction is a dual write, and no ordering of the two writes is safe.
• Real test at a 2% crash rate: commit-then-publish lost 181 events; publish-then-commit created 198 orphans; the outbox lost nothing and produced 197 duplicates.
• The transactional outbox puts the event in the same database transaction; a polling or CDC relay publishes it at-least-once.
• 2PC is atomic but blocks on coordinator failure; it belongs inside databases, not between services.
• Sagas chain local transactions with compensations; every step must be idempotent and durable.
• Reconciliation catches what everything else misses.

Next lesson: what happens under load. Queueing, retry storms, and how one slow dependency takes down a whole system, with a simulation where retries turned a 5-second slowdown into a permanent outage.