Lesson 02: Quorum & Heartbeats

Leader Election

Since true deterministic consensus is impossible (as we learned from the Two Generals), distributed systems cheat. They use timeouts and probabilistic approaches to agree on a single source of truth: The Leader.

Instead of trying to get every single node to agree simultaneously on every single write, the cluster elects one node to be the dictator. All writes go to the Leader. But what happens if the Leader crashes?

Randomized Election Timeouts

When a cluster starts, every node is a Follower. Every Follower has an internal countdown timer called the Election Timeout. This timer is continuously reset by Heartbeat messages sent by the Leader.

If the Leader crashes, the heartbeats stop. The followers' timers start ticking down. The genius of modern consensus algorithms (like Raft) is that these timers are randomized (e.g., between 150ms and 300ms).

Because they are random, one node will always time out before the others. That node immediately declares itself a Candidate and requests votes from the rest of the cluster. Since it asked first, it gets the majority of votes, becomes the new Leader, and starts sending heartbeats to suppress the other nodes' timers.

Leader Election Simulator

Node B is currently the Leader, sending heartbeats to keep A and C's timers full. Click on Node B to "kill" it. Watch the timers drop. The first node to hit empty will trigger an election and become the new Leader.

A
Follower
B
Leader
C
Follower
Leader B is sending heartbeats...

Split Brain and Quorums

What if a network partition occurs, and half the cluster can't talk to the other half? Could both halves elect their own leader, resulting in a "Split Brain" where two leaders start writing conflicting data?

No, because to become a Leader, a node must receive votes from a Majority (Quorum) of the cluster. In a 3-node cluster, a majority is 2. If the network splits into a piece with 2 nodes and a piece with 1 node, only the 2-node piece can form a majority and elect a leader. The 1-node piece can never get 2 votes, so it halts. This mathematically guarantees there can only ever be one Leader.

Check yourself

1. How does Raft avoid "split votes" where multiple nodes timeout at the exact same time and request votes?

Raft uses randomized election timeouts (e.g. between 150-300ms). Because they are random, it is highly likely one node will always time out before the others and win the election.