Tied to Mission: Backend Depth — resolves the CAP forward-pointer from Lesson 7 and adds the distributed-systems phase's core topic: what replication buys you, and the consistency price it charges.
You update your profile. The write goes to the primary database and succeeds. You refresh the page; the read is served by a replica — and you see your old profile. Not a bug in the app, not a lost write: the replica simply hasn't received the update yet. This is the everyday face of a topic that sounds exotic in interviews: replication, and the consistency anomalies that come with it.
Nobody adds replicas for fun. The reasons, in rough order of how often they actually drive the decision:
In leader-based (primary/secondary) replication, all writes go to one node — the leader. The leader records the change and ships it to the followers, which apply it. Followers can serve reads; only the leader accepts writes. This is the model behind Postgres streaming replication, MySQL, and most managed databases, and it's the right mental model to start from.
The subtle part is when the leader considers a write committed. Two modes:
| Mode | What committing means | The tradeoff |
|---|---|---|
| Synchronous | The leader waits for at least one replica to acknowledge the change before telling the client "committed." | Stronger durability: a committed write is confirmed safe on another machine. But every write now depends on that replica's health — one slow replica slows every write, and if no replica is available, writes stop entirely. |
| Asynchronous | The leader acknowledges the write immediately and ships the change to replicas in the background. | Writes are fast and don't depend on replica health. But a replica may lag behind, and here is the sharp edge: if the leader dies before shipping a write that it already acknowledged, that write is gone. The client was told "committed," and the data vanished with the leader. |
That last sentence is how "we lost data" incidents happen — the async durability window, realized by a leader crash. Most systems compromise with semi-synchronous replication: wait for one replica, keep the window tiny.
With asynchronous replication, followers trail the leader by some amount: replication lag. Lag is normal, usually milliseconds. The problem is that reading from a lagging replica can produce three classic anomalies, and knowing them by name is knowing when you've found them:
None of these means the database is broken. They're the predictable cost of spreading reads across nodes that receive updates at slightly different times. The design work is deciding which anomalies your product can tolerate, and where — usually the "my own write must be visible immediately" rule is worth a leader read; "monotonic reads for everyone forever" is worth a lot less.
When the leader dies, a replica is promoted to leader and clients switch to it. With async replication, the new leader is missing the writes the old leader acknowledged but never shipped — the durability window again, now with a face: failover can lose acknowledged writes. This is why synchronous replication exists at all: for data you can't afford to lose (orders, balances), paying write latency to guarantee at least one other copy holds the data is the honest trade.
Lesson 7 flagged the hard version: a network partition between leader and replica — they can't reach each other, both individually healthy. During a partition, a system that keeps serving reads from the isolated replica is serving possibly-stale data (favoring availability); a system that refuses reads until it can confirm with the leader is choosing consistency. That's the binary choice CAP describes.
What this lesson adds is the practical frame: replication lag is the everyday, tunable version of that same tradeoff. With no partition, you choose your lag budget — how much staleness you accept per read — by configuring sync vs async and routing reads. Partitions just remove the middle ground and force the binary choice. The vocabulary in the failure models cheat sheet (availability, consistency, partitions) is the shared language for both.
Leader-based replication handles the common case. Two harder designs exist, worth knowing as forward pointers rather than covering here: multi-leader (writes accepted at several nodes; conflicts must be resolved — "two people edited the same record in different DCs") and no-leader/quorum (writes go to several nodes and are considered committed when a majority acknowledges — the terrain of Raft and other consensus algorithms, which is MIT 6.824 territory).
If you have access to any database with a read replica (Postgres streaming replication, a managed DB), write a row, immediately read it back through the replica endpoint, and repeat until you catch it missing — then check the lag metric: for Postgres, SELECT * FROM pg_stat_replication shows sent_lsn/replay_lsn and the lag between them. If you don't have a replica handy, do the mental experiment: your application's read path — how many of its reads actually go to a replica, and which of the three anomalies would your users notice first if it lagged by a second? That answer is your staleness budget.
Designing Data-Intensive Applications, Chapter 5, "Replication" — the authoritative treatment of exactly this lesson's material: leader-based replication, sync vs async, and all three lag anomalies with their fixes. PostgreSQL's own Streaming Replication documentation is the primary source for how the concrete system behaves (including pg_stat_replication). For the consensus/quorum design beyond the forward pointer, MIT 6.824 remains the destination — see the failure models cheat sheet and RESOURCES.md.
This closes the second pass: MVCC and the planner (Postgres), TCP under saturation (network), virtual memory (OS), the event loop (Python), and backpressure, idempotency, and replication (distributed systems). Good moment to ask for a recap that crosses all 15 lessons, or to drill into anything that didn't land.