Lesson 1 ended with a horror story: a zombie leader, a skipped fencing step, split-brain. The root problem was that leadership was an assumption — configured by humans, revoked by hope. Consensus algorithms (Raft, Paxos, Zab) make leadership a provable fact: a node leads only if a majority of the cluster voted for it, and a write is committed only when a majority holds it. Majorities of the same cluster always intersect — so two simultaneous legitimate leaders are mathematically impossible.
This is the machinery inside etcd (which runs Kubernetes), CockroachDB, TiDB, MongoDB's replica-set elections, Kafka's KRaft, Consul, ZooKeeper, and the coordination layer of Aurora and Spanner. We'll use Raft, the one designed to be understandable. Two mechanisms, two labs: elections and log commitment.
Mechanism 1: elections with terms and votes
Every node is a follower, candidate, or leader. Time is divided into numbered terms. Followers expect heartbeats from the leader; when a follower's randomized election timeout expires with no heartbeat, it increments the term, becomes a candidate, and requests votes. Each node votes for at most one candidate per term. Majority of the full cluster (3 of 5) wins — and if no majority is reachable, nobody wins, on purpose.
Run the elections yourself
Five nodes, one leader. Click any node to crash/revive it. When the leader is dead, fire the election timeout and watch votes gather. Then get destructive: crash nodes until only two remain, and try to elect anyone.
Three details make this bulletproof. Randomized timeouts mean one node usually times out first and wins cleanly; ties just retry with new random timers. One vote per term means two candidates can't both reach a majority in the same term. And a candidate must have a log at least as up-to-date as each voter's — so a node missing committed entries can never win, which kills Lesson 1's "promoted the stale follower" data-loss bug by rule rather than by operator diligence.
Mechanism 2: majority commit — and why split-brain can't happen
The leader appends each write to its log and replicates it. The write is committed — acknowledged to the client, applied to the state machine — only once a majority of nodes hold it. Now the payoff. Walk through the exact scenario that destroyed Lesson 1's cluster:
The partition drill
Five nodes; n1 leads. A network partition splits {n1, n2} from {n3, n4, n5}. Step through what each side can and cannot do.
Ready
Healthy cluster: n1 leads term 1, all logs identical. Press Next step to cut the network.
The one-line proof: committing needs a majority; electing needs a majority; two majorities of five nodes always share at least one node — and that shared node will not vote for a stale candidate or acknowledge a deposed leader's entries (its term number gives the zombie away instantly). Split-brain isn't prevented by careful operations; it's prevented by arithmetic. The price: a minority partition is unavailable for writes — consensus deliberately chooses consistency over availability (the C side of CAP), and every write costs a majority round-trip.
Where this sits among the four architectures
Consensus is single-leader replication with the two hard problems — who leads, and what survives — solved by protocol instead of by operators. You'd choose it when correctness is non-negotiable: metadata stores (etcd under Kubernetes, ZooKeeper under Kafka), financial ledgers, distributed SQL (CockroachDB runs one Raft group per data range — thousands of tiny consensus clusters). You'd avoid it where its costs bite: majority round-trips on every write, minimum three nodes, and cross-region deployments where "majority" means intercontinental latency on every commit.
Check yourself
1. A 5-node etcd cluster loses 3 nodes. The 2 survivors are healthy and networked to each other. What can they serve?
Majority means majority of the configured cluster (3 of 5), not of survivors. Two nodes can't elect or commit — by design, because the three missing nodes might be alive on the far side of a partition doing the same math. This is also why clusters are sized 3/5/7: an even count adds failure surface without adding tolerable failures.
2. A deposed leader (crashed, now recovered) tries to replicate an entry to a follower. How is it stopped?
Terms are Raft's fencing tokens, built into every message. The zombie says "append this, term 3"; the follower answers "we're on term 5"; the zombie instantly demotes itself. Lesson 1's STONITH step — the one humans forget at 3 a.m. — is here an unforgettable property of the protocol.
3. Why does CockroachDB run thousands of small Raft groups instead of one big one?
A single Raft group funnels every write through one leader — fine for a config store like etcd, a bottleneck for a SQL database. Sharding the keyspace into ranges, each its own consensus group with its own leader, is replication composed with partitioning: the course's two big ideas working together.