Tied to Mission: Backend Depth — Lesson 16's outbox produces events; this lesson is the system they land in. The log is also the unifying abstraction underneath Postgres durability, replication, and the outbox itself — one idea, four appearances so far in this curriculum.
A consumer of your "user_updated" events crashes halfway through processing one. When it restarts, it doesn't resume where it stopped — it starts again from an earlier point, re-processing events you already handled. And a co-worker says, cheerfully, "just rewind the stream and rebuild the cache, the broker kept everything." It did. The broker isn't a queue that deletes messages once delivered. It's a log — and that distinction explains nearly everything about how message systems behave.
A log is the simplest possible data structure: an append-only sequence of records, each with a monotonically increasing offset (its position). New records go at the end. Nothing is ever modified in place. That's it. The interesting part is everything you can do with that shape:
The mental model flip: a queue is something you drain; a log is something you read. Retention (how long records are kept) is a storage policy, not a delivery mechanism.
A single log has a total order — every record has a position, and positions don't reorder. That's exactly the guarantee a saga or an outbox consumer usually needs: process "order_created" before "payment_succeeded" for the same order. But one log on one node has the capacity of one node (Lesson 18's problem, next lesson). The answer is partitions: split the log into several partitions, each an independent log with its own offsets.
The tradeoff is precise and worth stating exactly: within a partition, order is preserved; across partitions, there is no order at all. So the trick is deciding what must stay ordered. The standard rule: partition by key — hash of the entity id — so all events for one user land in the same partition, and per-user ordering is preserved while the cluster gets to spread users across partitions. Events for different users are independent anyway; nobody cares whether user A's update precedes user B's. "Partition by key, order within a partition" is the same design Lesson 16's relay implies and Lesson 18 will treat properly.
One consumer reading a log is the slow path. To parallelize, you use a consumer group: a set of consumers that share the partitions among themselves. The rule that keeps everything consistent: each partition is read by exactly one member of the group at any time. Add more consumers and partitions get reassigned to spread the load; add a consumer beyond the number of partitions and it sits idle (Lesson 13's saturation shape, again). Within a partition, order is preserved because one consumer reads it in offset order. Across partitions, the group is parallel — but each partition is still in order.
Here is where Lesson 14 shows up, now on the consumer's side. The consumer keeps its offset — the position it has processed up to. Committing the offset is a write with exactly Lesson 14's ambiguity: you commit, the ack is lost, you retry, the broker already advanced the position. So the ordering of two events — "process the record" and "commit the offset" — picks your delivery guarantee:
Producers write faster than consumers process, and the gap grows: that's consumer lag — the distance between the latest offset and the consumer's position. It is Little's law with a different name (Lesson 13): an unbounded backlog converting a throughput gap into delayed, eventually-failing work. Lag is the single most important operational metric of a log-based system, and the fix is Lesson 13's toolbox: scale consumers, shed load, alert on the leading indicator instead of the crash.
Replay is a feature, but storing everything forever is a cost. Logs apply retention — keep records for a time window or a size cap, then discard the oldest. Two flavors: time/size-based (keep the last 7 days) and compaction (keep only the latest record per key — a running "current state" for event-sourced systems, and the storage shape that makes "rebuild the cache from the stream" cheap). The outbox table from Lesson 16 has the same decision: rows are deleted once published; the relay's position plays the role of the consumer's offset.
Jay Kreps's essay "The Log" made the claim this lesson has been circling: the log is the one structure distributed systems keep reinventing. Postgres durability is a log (WAL). Replication ships log entries to replicas (Lesson 15). The outbox is a log inside your transaction. The broker is a distributed log with partitions. Every one of them gets the same three properties — ordered, append-only, replayable — and pays the same price: consumers must manage their own position, and duplicates are possible unless processing is idempotent.
Build a mini log in ~40 lines of Python: an append-only file where each line is offset<TAB>event (the "broker"). Write a produce() that appends and returns the offset, and a Consumer class that keeps its offset in a separate file, read_next()s in order, and commit()s after processing. Now run the crash experiment: process a record, print it, but don't commit, then restart the consumer — it re-reads the record (at-least-once). Add a processed-ids set (Lesson 14's move) and the duplicate disappears. Then demonstrate replay: start a second consumer with offset 0 and read the whole history to "rebuild a cache" — note that this works only because nothing was deleted on delivery. If you want the partition piece, split the file by user_id % 3 into three logs and verify each user's events stay in order within their partition.
Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" — the canonical essay this lesson's framing comes from; it names the log as the shared structure under databases, replication, and streaming. Designing Data-Intensive Applications, Chapter 11, covers logs, partitions, offsets, and consumer groups in the context of stream processing. The Apache Kafka documentation is the primary source for the concrete system: partitions and ordering, consumer groups, offsets, and retention policies.
The log gets its capacity from partitions — which is the same trick a database uses when one node can't hold the data. Lesson 18: partitioning and sharding.