This is the architecture behind PostgreSQL streaming replication, MySQL replicas, MongoDB replica sets, Redis, SQL Server Always On, RDS — the overwhelming majority of databases in production. The design is one sentence: exactly one node (the leader / primary) accepts writes; it streams its change log to followers (replicas), which replay it in order.
Its power is that same sentence. One writer means no write conflicts, ever — the leader serializes everything, and followers converge to identical state by replaying an identical log. All the interesting engineering lives in two questions: when do you tell the client "committed", and what happens when the leader dies.
Question 1: when is "committed" true?
The leader writes locally, then streams to followers. Does it wait for them before acking the client? Three answers exist, and they're a strict trade of latency against durability:
Three modes, one crash test
Commit rows in each mode and watch what the ack waits for. Then the real test: crash the primary immediately after a commit — the most up-to-date follower gets promoted, and the lab counts what survived. Note how semi-sync behaves when you also mark follower F1 as slow.
acked to client
0
commit latency
—
rows lost in failover
0
The vocabulary for what you just measured: RPO (recovery point objective — how much acknowledged data a failure may destroy) and commit latency. Async: lowest latency, RPO > 0. Fully sync: RPO = 0, but latency is set by your slowest follower, and one dead follower blocks all writes — which is why almost nobody runs it across more than one confirmer. Semi-sync is the production sweet spot: every commit exists on the leader plus at least one other node, and a slow follower doesn't matter as long as some follower is fast. PostgreSQL's synchronous_commit / synchronous_standby_names and MySQL semi-sync plugins are exactly this dial.
The async tax: reading the past
Followers running behind isn't just a crash-time concern — it changes what reads mean. Any read served by an async follower may return the past. The classic bug even has a name: the user who posts a comment, refreshes, and watches it vanish.
Race your own write
Post, then immediately read from the follower (lag here is an exaggerated 5s so you can race it). Then enable read-your-own-writes routing and repeat.
on primary
0
on follower
0
follower lag
0.0s
The consistency ladder for follower reads, strongest to cheapest: strong (read the leader), read-your-own-writes (route a user's reads to the leader briefly after they write — the toggle above), monotonic reads (pin each user to one follower so time never runs backwards for them), eventual (any follower, no promises about when). Most "the database lost my data!" tickets are eventual-consistency reads wearing a scary costume.
Question 2: the leader is dead. Now what?
Failover sounds simple — promote a follower — but every step hides a trap. Run one, then cause the disaster that keeps DBAs up at night:
Failover drill, then split-brain
Step through a clean failover. Then hit the red button: the old primary comes back from its network blip, still believing it's the leader — and some clients never got the memo.
Everything in that drill maps to real knobs. Detection is a heartbeat timeout — too short and a GC pause triggers needless failovers, too long and you're down for minutes. Choosing the promotee should be the most-up-to-date follower, or you maximize data loss. Fencing the old leader (a.k.a. STONITH — "shoot the other node in the head") is the step teams skip until the day they learn why it exists: without it, a zombie leader plus a stale client list equals two databases both accepting writes for the same data — split-brain, the one failure with no automatic recovery, only forensic reconciliation. Managed services (RDS Multi-AZ, MongoDB replica set elections) exist largely because they do these four steps correctly at 3 a.m. so you don't have to.
Check yourself
1. Semi-sync with 4 followers, "wait for any 1." One follower's datacenter link degrades to 2s latency. What happens to commit latency?
"Any 1 of N" means the slowest follower is irrelevant as long as one is fast — that's the whole reason semi-sync is deployable where full sync isn't. The degraded follower simply accumulates lag (and should page someone before it's chosen in a failover).
2. During failover you promote a follower that was 30 seconds behind, while a more current follower existed. What did you just do?
The new leader's history is now the truth; everything past its position is gone. Worse, the more-current follower can't cleanly follow a leader whose history diverged behind its own — it typically must be rebuilt. Promotion choice is a data-loss decision, which is why tooling like pg_auto_failover and orchestrators compare log positions first.
3. What single property makes split-brain uniquely bad compared to ordinary downtime?
Downtime loses availability; split-brain loses truth. Two histories both told clients "committed," and reconciling €80-after-withdrawal vs €150-after-deposit is a business decision, not a database operation. Hence fencing, and hence Lesson 4's consensus algorithms, which make the situation impossible by construction.