Lesson 15 · Distributed Systems / Failure Models

Why your own write is missing when you read from a replica

Tied to Mission: Backend Depth — resolves the CAP forward-pointer from Lesson 7 and adds the distributed-systems phase's core topic: what replication buys you, and the consistency price it charges.

You update your profile. The write goes to the primary database and succeeds. You refresh the page; the read is served by a replica — and you see your old profile. Not a bug in the app, not a lost write: the replica simply hasn't received the update yet. This is the everyday face of a topic that sounds exotic in interviews: replication, and the consistency anomalies that come with it.

Why replicate at all

Nobody adds replicas for fun. The reasons, in rough order of how often they actually drive the decision:

Leader-based replication: the workhorse design

In leader-based (primary/secondary) replication, all writes go to one node — the leader. The leader records the change and ships it to the followers, which apply it. Followers can serve reads; only the leader accepts writes. This is the model behind Postgres streaming replication, MySQL, and most managed databases, and it's the right mental model to start from.

The subtle part is when the leader considers a write committed. Two modes:

ModeWhat committing meansThe tradeoff
Synchronous The leader waits for at least one replica to acknowledge the change before telling the client "committed." Stronger durability: a committed write is confirmed safe on another machine. But every write now depends on that replica's health — one slow replica slows every write, and if no replica is available, writes stop entirely.
Asynchronous The leader acknowledges the write immediately and ships the change to replicas in the background. Writes are fast and don't depend on replica health. But a replica may lag behind, and here is the sharp edge: if the leader dies before shipping a write that it already acknowledged, that write is gone. The client was told "committed," and the data vanished with the leader.

That last sentence is how "we lost data" incidents happen — the async durability window, realized by a leader crash. Most systems compromise with semi-synchronous replication: wait for one replica, keep the window tiny.

Replication lag: not a bug, a cost — and its three anomalies

With asynchronous replication, followers trail the leader by some amount: replication lag. Lag is normal, usually milliseconds. The problem is that reading from a lagging replica can produce three classic anomalies, and knowing them by name is knowing when you've found them:

None of these means the database is broken. They're the predictable cost of spreading reads across nodes that receive updates at slightly different times. The design work is deciding which anomalies your product can tolerate, and where — usually the "my own write must be visible immediately" rule is worth a leader read; "monotonic reads for everyone forever" is worth a lot less.

Leader failure: the promotion and the lost tail

When the leader dies, a replica is promoted to leader and clients switch to it. With async replication, the new leader is missing the writes the old leader acknowledged but never shipped — the durability window again, now with a face: failover can lose acknowledged writes. This is why synchronous replication exists at all: for data you can't afford to lose (orders, balances), paying write latency to guarantee at least one other copy holds the data is the honest trade.

The CAP tie-in, resolved

Lesson 7 flagged the hard version: a network partition between leader and replica — they can't reach each other, both individually healthy. During a partition, a system that keeps serving reads from the isolated replica is serving possibly-stale data (favoring availability); a system that refuses reads until it can confirm with the leader is choosing consistency. That's the binary choice CAP describes.

What this lesson adds is the practical frame: replication lag is the everyday, tunable version of that same tradeoff. With no partition, you choose your lag budget — how much staleness you accept per read — by configuring sync vs async and routing reads. Partitions just remove the middle ground and force the binary choice. The vocabulary in the failure models cheat sheet (availability, consistency, partitions) is the shared language for both.

Where this goes next: multi-leader and quorum

Leader-based replication handles the common case. Two harder designs exist, worth knowing as forward pointers rather than covering here: multi-leader (writes accepted at several nodes; conflicts must be resolved — "two people edited the same record in different DCs") and no-leader/quorum (writes go to several nodes and are considered committed when a majority acknowledges — the terrain of Raft and other consensus algorithms, which is MIT 6.824 territory).

Check yourself

A user updates their profile (write goes to the primary) and immediately refreshes. The read is served by a replica that hasn't received the update yet, so they see the old profile. What is this, and what's the standard fix?
Right — lagging replicas are normal, and the anomaly is one of the three classic ones. The fix is routing: reads for a session that just wrote go to the leader until the replica catches up, giving read-your-writes semantics where they matter. Not quite — a lagging replica serving old data is not a fault; it's the cost of asynchronous replication. The anomaly has a name (read-your-writes violation) and a standard fix (route that session's reads to the leader). Re-read the lag anomalies section.

Try it for real

Hands-on

If you have access to any database with a read replica (Postgres streaming replication, a managed DB), write a row, immediately read it back through the replica endpoint, and repeat until you catch it missing — then check the lag metric: for Postgres, SELECT * FROM pg_stat_replication shows sent_lsn/replay_lsn and the lag between them. If you don't have a replica handy, do the mental experiment: your application's read path — how many of its reads actually go to a replica, and which of the three anomalies would your users notice first if it lagged by a second? That answer is your staleness budget.

Primary sources

Designing Data-Intensive Applications, Chapter 5, "Replication" — the authoritative treatment of exactly this lesson's material: leader-based replication, sync vs async, and all three lag anomalies with their fixes. PostgreSQL's own Streaming Replication documentation is the primary source for how the concrete system behaves (including pg_stat_replication). For the consensus/quorum design beyond the forward pointer, MIT 6.824 remains the destination — see the failure models cheat sheet and RESOURCES.md.

This closes the second pass: MVCC and the planner (Postgres), TCP under saturation (network), virtual memory (OS), the event loop (Python), and backpressure, idempotency, and replication (distributed systems). Good moment to ask for a recap that crosses all 15 lessons, or to drill into anything that didn't land.