Lesson 19 · Distributed Systems / Failure Models

Why the cluster briefly accepted writes from two leaders

Tied to Mission: Backend Depth — resolves the "no-leader/quorum" forward pointer from Lesson 15. Lesson 15 assumed a leader existed; this is the lesson about how a cluster can agree on who the leader is and which writes are real, even when nodes fail and the network partitions.

A 5-node cluster. The leader's network cable gets unplugged — a partition splits the cluster into a 3-node side and a 2-node side (the old leader among them). The 3-node side elects a new leader and keeps accepting writes, committed against its majority of 3. Meanwhile, the isolated old leader still thinks it's the leader, and clients that can still reach it happily send writes. Two leaders, both accepting writes, both "sure" they're in charge. When the cable goes back in and the cluster reunites, which writes survive?

The agreement problem: leader identity and the commit point

Lesson 15's leader-based replication quietly assumed three things: there is exactly one leader, everyone knows who it is, and a write is committed when the leader says so. Under failure, each of those assumptions becomes a question the nodes must answer together: Who gets to be the leader? When is a write durable enough that it must never be lost? If two nodes disagree on either answer, the data diverges — and the divergences don't merge on their own. This is the consensus problem, and it's the hardest thing distributed systems do.

Majority: the overlap that makes agreement possible

Everything hinges on one arithmetic fact: in a group of N nodes, any two subsets of size N/2 + 1 (a majority, or quorum) must overlap — they can't both be majorities without sharing at least one node. That overlap is what turns "agreeing despite failures" from impossible into possible: if a write is acknowledged by a majority, and a read is answered by a majority, then some node has seen both, so the read cannot miss the write.

Generalized: with N nodes, a write quorum of W nodes and a read quorum of R nodes, the rule is R + W > N — the read quorum and write quorum must intersect. Set both to N/2 + 1 and you get the majority case. The properties:

This is also the answer to Lesson 15's durability window: with quorum writes, there is no "leader says committed but the ack was lost" window. A write is durable the moment a majority holds it — not before, and the client is told exactly when that happened.

What "consistent" means: linearizability vs. sequential consistency

Quorum math guarantees intersection. What users actually experience is a consistency model, and the vocabulary matters in interviews:

Quorum reads and writes with R + W > N, when both wait for their full quorums, deliver linearizable behavior — the intersection guarantees a read sees the latest acknowledged write, and waiting for the quorum guarantees the "completed before started" ordering. Real quorum stores add versioning and read repair on top (Dynamo-style vector clocks) because "latest" needs a precise definition when two writes race. The practical interview takeaway: quorum math gives you the overlap; the consistency model is what the client observes; and R + W > N with full-quorum waits is the configuration that buys you linearizable reads.

Raft: elections, terms, and log replication

Majorities explain why agreement is possible. Raft is a concrete protocol that uses them, and it's the one worth knowing (etcd, Consul, and many databases use it). Three mechanisms:

Back to the scenario

3-node side: elects a leader (majority of 3), accepts writes, commits them against its 3-node quorum. 2-node side: the old leader is isolated, can reach only one other node — never a majority — so it can never commit; its "accepted" writes are uncommitted. When the partition heals, the old leader hears the higher term, steps down, and its uncommitted writes are discarded (its log is brought in line with the majority's). The 2-node side's writes are gone — and that's the correct behavior. The alternative, letting both sides keep their writes, is split brain: data diverges, and no future protocol can reconcile it. Consensus's job is to make sure only one side ever commits, so the question "which writes survived" has exactly one answer.

What you pay for it

You've met quorums before without the name: Lesson 17's Kafka partitions commit a record when the in-sync replicas have it, and min.insync.replicas is a write quorum with the same availability cliff. The outbox relay in Lesson 16 writes to a database whose durability comes from its own replication — in a quorum-replicated store, the "committed in the same transaction" guarantee rests on this machinery underneath.

Check yourself

A 5-node Raft cluster partitions into a 3-node side and a 2-node side. The 2-node side elects a leader (its old leader) and accepts a write from a client. When the partition heals, what happens to that write?
Right — a write is committed only when a majority acknowledges it, and 2 of 5 is never a majority. The client's write was accepted but never committed, so it's discarded when the cluster heals. Allowing both sides to keep writing would be split brain — unrecoverable divergence. This is the price of the availability cliff: correctness wins. Not quite — a leader can only commit with a majority, so the 2-node side's write was never committed, and uncommitted writes are discarded on heal (the old leader steps down to the higher term). If both sides kept their writes, you'd have split brain. Re-read the Raft section and the scenario callout.

Try it for real

Hands-on

Two short simulations in Python. 1) Majority in a partition: write a function elect(nodes, total) where each node's random timeout decides who wakes up first in its side, everyone in the side votes for that first candidate, and a candidate wins only with a majority of the total cluster. Run it on the split [0,1,2] vs [3,4] (total 5, majority 3): the 3-side elects every time, the 2-side never does. Now you've seen why the 2-side can't commit — it can't even elect. 2) Quorum intersection: with N=5, pick many random read quorums of size R and write quorums of size W. Verify that R + W > N (e.g., 3+3) always intersects, and that R + W ≤ N (e.g., 2+2) can be disjoint — meaning a read can miss the latest write. The first case is the entire basis for quorum reads being linearizable; the second is how stale reads sneak in when someone "optimizes" quorum sizes.

Primary sources

Ongaro & Osterblom, "In Search of an Understandable Consensus Algorithm" — the Raft paper, and raft.github.io for the animation and the "Secret Lives of Data" video that makes elections click. Designing Data-Intensive Applications, Chapter 9, "Consistency and Consensus" — linearizability vs. sequential consistency and the majorities argument, in the context that ties this lesson to Lessons 15 and 17. For the operational half — quorum reads, versioning, and read repair — the Dynamo paper is the reference. MIT 6.824 remains the destination if consensus needs to go past interview depth (it has Raft labs and the Raft lectures).

Consensus tells a cluster what's true. The next lesson is about how you find out what's true about a running system at all — metrics, logs, traces, and the shapes of symptoms — Lesson 20.