Lesson 16 · Distributed Systems / Failure Models

Why the order was saved but the email never arrived

Tied to Mission: Backend Depth — Lesson 14 made single retries safe; Lesson 15 made reads tolerate lag. This is where those meet: how to make a database write and a message to another system atomic, when no single transaction can span them.

You order something. The database row says the order exists — you can see it in the UI. But the confirmation email never arrives, and the analytics team's "orders" chart never moves. The order service wrote to its database and then… something between the write and the publish didn't happen. The order exists; the event doesn't. Two systems, one logical operation, and no atomicity between them.

The dual-write problem: two systems, one operation, no shared transaction

A "create order" operation touches more than the database: it should also emit an event so the email service, the analytics pipeline, and the inventory service can react. That's a write to your DB and a write to a message broker — two different systems. The naive code does them in sequence:

with db.transaction():
    insert_order(...)      # 1. write to the database
publish("order_created")   # 2. write to the broker

What happens if the process crashes between step 1 and step 2? The order exists, the event never fires — the email is never sent. What happens if you reverse the order and crash between those steps? The event fires for an order that doesn't exist — the email service confirms a phantom order, analytics counts a sale that never happened. Neither order is safe. The crash window just moves from "lost event" to "phantom event."

And you can't fix it by "wrapping it in a transaction," because a transaction belongs to one database. Your broker is a different system — it has no idea your DB transaction is open. Two-phase commit (2PC) exists to make heterogeneous systems commit together, but it's rarely the answer: it holds locks on every participant until the coordinator decides, and if the coordinator crashes mid-protocol the participants are stuck. Production systems mostly avoid it for exactly this reason. Lesson 2's move is the template here: the fix lives in the data layer, not in more careful app code.

The outbox pattern: make the event a row in the same transaction

The trick — this is the transactional outbox: write the event to a table in your own database, inside the same transaction as the business write. Now the database transaction — the atomicity you already trust — covers both the order and its event:

with db.transaction():
    insert_order(...)
    insert_outbox(event_type='order_created', payload=...)  # same tx

The crash window is gone: if the process dies before COMMIT, both the order and the event row vanish together (rollback). If it dies after COMMIT, both are durable — and the relay, a separate process doing a separate job, will find the event row and publish it. The database transaction is now the atomicity boundary for both systems' writes.

The relay does three things, in order:

Read unpublished rows, oldest first (order by the outbox id — publishing oldest-first preserves the order the writers created).
Publish each row to the broker.
Mark the row published (a flag, or delete it) so it isn't republished.

Three failure modes matter, and each has a name you already know:

Crash after publish, before mark → duplicate publish. The relay restarts, reads the row again, publishes again. This is at-least-once delivery — which is why Lesson 14 is a prerequisite: consumers must deduplicate (idempotency keys, or a "processed events" table), and then the end-to-end guarantee is effectively-once: no event lost, no event doubled.
Crash mid-publish → partially published. Same answer: the relay re-reads and re-publishes; consumer-side dedup absorbs the duplicates.
Writers outpace the relay → the outbox table grows. Lesson 13's shape again: an unbounded queue that delays failure rather than preventing it. Mitigations: publish in batches, alert on table size, and delete (or archive) rows once published.

Relay implementations

The relay is usually one of two things: a polling loop in your own service ("every 100ms, select unpublished rows, publish, mark") — simple, portable, adds a small publish delay — or change data capture (CDC, e.g. Debezium), which reads the database's own write-ahead log and turns every committed change into an event. CDC isn't a second reader competing with your app; it reads the same WAL the database itself uses to be durable (the machinery from Lessons 8 and 11). The outbox table makes CDC trivial to set up, because every event is just a normal committed row. One subtlety worth knowing: the outbox table is an ordinary table, so it's replicated and laggy like any other (Lesson 15) — reading it from a replica means your relay publishes with lag.

When the process spans services: sagas

The outbox solves "one database + one event." But a real order flow touches the order service, payment service, inventory service, shipping service — each with its own database, possibly owned by a different team. There is no transaction that spans them, and 2PC won't save you. This is what sagas are for.

A saga breaks the business process into steps, each with its own local transaction (committed for real), and gives every step with side effects a compensating transaction that undoes it. If step N fails, you run the compensations for steps N−1 down to 1, in reverse. The payment step's compensation is a refund. The inventory step's compensation is returning the stock. The order step's compensation is cancelling the order.

Two shapes exist:

Choreography: each service, after its step succeeds, publishes an event that triggers the next service. No central coordinator; the flow is implicit in the events. Simple when the chain is short; hard to see the whole flow in one place, and a failure means events have to flow backward too.
Orchestration: a coordinator — the saga state machine — calls each service in turn, knows what step it's on, and runs the compensation logic itself. More explicit: the whole process lives in one place, at the cost of a coordinator that must itself be reliable and is the natural place for Lesson 13's backpressure to bite.

The part that makes sagas hard is the failure path, and every piece of it reuses a previous lesson:

Steps must be retryable and idempotent. A step can succeed and the response be lost — at-least-once again (Lessons 7 and 14). The saga needs an idempotency key per attempt so a retried step doesn't double-charge or double-ship.
Compensation is a distributed call too. It can fail, and it must be retried with backoff (Lesson 7) until it succeeds — a saga that can't undo is a saga that can't fail cleanly. Compensation must also be idempotent (refunding twice).
A saga is eventually consistent. Between steps, other readers can observe the in-progress state: order exists, payment not yet taken. That's a deliberate trade: you give up "all-or-nothing at one instant" and get "converges, or rolls back cleanly." Design the read path for it (the payment is pending, not missing).

Saga vs. transaction

A saga is sometimes described as "a transaction without atomicity." More precisely: a normal transaction guarantees atomicity by not committing until everything is ready. A saga achieves its guarantee by running the undo when something goes wrong. That's why it works across systems and long-running flows — no locks are held across services, no coordinator freezes participants — and why the compensation path has to be designed first, not bolted on after.

Putting it together

Outbox answers "how do I make a write and an event atomic?" — one database transaction + a relay + idempotent consumers.
Saga answers "how do I make a multi-service process converge?" — retryable idempotent steps + compensations.
Both assume at-least-once delivery and both lean on idempotency (Lesson 14) to turn it into effectively-once. Both are eventually consistent and have to say so on their read paths.

And the two compose: each saga step's "step succeeded" event is written to that service's outbox in the same transaction as the step's own write. The outbox is where sagas' events come from.

Check yourself

Your order service runs this code: (1) insert the order row, (2) publish "order_created" to Kafka. The process is killed between the two lines. Which statement is true?

Right — the write committed, the publish never ran, and nothing will ever run it. Writing the event row inside the same transaction as the order makes the crash window disappear: the transaction either commits both or neither, and the relay handles publishing afterwards. Not quite — the transaction already committed (the order is visible), so there's nothing to roll back, and the broker can't publish an event nobody sent it. This is the dual-write problem: two systems with no shared atomicity. Re-read the outbox section.

Try it for real

Hands-on

Build a 20-line outbox with SQLite — no server needed. Create two tables, orders and outbox. First reproduce the bug: insert into orders, commit, and don't write the outbox row (simulate the crash by skipping the publish). The order exists; there is no event — a permanent state. Now redo it correctly: BEGIN; INSERT INTO orders; INSERT INTO outbox; COMMIT;. Write a relay loop that selects unpublished rows, prints them (the "publish"), and marks them done — then kill the relay between select and mark and restart it: the row gets published twice. That's at-least-once in miniature; add a consumer-side processed_events table with a unique constraint on the event id (Lesson 14's move) and the duplicate disappears. For the saga half: simulate a two-step flow (charge + ship) where step 2 fails, and write the compensation (refund) that runs in reverse — then make the compensation fail once, and see why it has to be retried with backoff (Lesson 7).

Primary sources

Designing Data-Intensive Applications, Chapter 11, "Stream Processing" — the dual-write problem and the outbox pattern (see "Keeping Systems in Sync"), plus Chapter 9 on why distributed transactions are hard. Chris Richardson's saga pattern is the canonical practitioner treatment, including choreography vs. orchestration and the compensation table. Pat Helland's "Life beyond Distributed Transactions: An Apostate's Opinion" is the original argument for why sagas and entities beat 2PC at scale. For the relay in production form: Debezium's outbox pattern documentation.

The outbox produces events; the next lesson is about the system those events land in — the log, and why "ordering within a partition" is the idea everything else builds on — Lesson 17.

← Lesson 15 Reference: Failure Models Cheat Sheet Lesson 17 →