Lesson 05 · Under the hood

Mechanics: what actually travels the wire

Four lessons of architectures, one unexamined phrase: the leader "streams its changes." Streams what, exactly? The answer splits into three formats with very different failure modes — and then the cloud adds a fourth trick where almost nothing travels at all.

Statement, WAL, or logical?

Statement-based: ship the SQL text itself and re-execute it on the replica. WAL / physical: ship the write-ahead log — the byte-level disk changes ("page 4711, offset 32, these bytes"). Logical / row-based: ship the effect — "row with id=7: column paid_at changed from NULL to this value." The differences stop being abstract the moment your SQL contains anything nondeterministic:

Replicate NOW() and watch the truth fork

Run the statement below through each format and compare what lands on the replica. Then try the second statement to see each format's other weakness.

UPDATE orders SET paid_at = NOW(), seq = RANDOM() WHERE id = 7;
on the wire
primary row
replica row

Scorecard. Statement-based is compact but broken by nondeterminism (NOW(), RANDOM(), sequences, triggers) — MySQL kept it for years with band-aids, modern defaults have moved on. WAL shipping is exactly faithful and cheap for the primary (the WAL exists anyway — it's how the database survives its own crashes; replication just mails it out), but bytes-on-disk coupling means primary and replica must run the same version on the same architecture, complicating upgrades. Logical decouples: cross-version, cross-engine, filterable per table — the format of PostgreSQL logical replication and MySQL row-based binlog. Its cost: one statement touching 10M rows becomes 10M row events.

Logical's superpower: CDC

Because logical events say what changed in engine-neutral terms, anything can subscribe — not just replicas. That's change data capture: tools like Debezium tail the log and feed every insert/update/delete into Kafka, from which you hydrate search indexes, caches, warehouses, and audit trails. The pattern matters because it replaces two bad habits: dual-writes from the app ("write to Postgres and Typesense" — which drift the moment one write fails) and polling (SELECT ... WHERE updated_at > ?, which misses deletes entirely). The database's own log becomes the single ordered truth, and every downstream copy is just another replica in spirit.

The cloud's fourth trick: replicate almost nothing

Aurora's observation: if the WAL fully determines the database, why ship whole modified pages to replica servers at all? Detach storage into a shared, distributed layer — 6 copies across 3 AZs — and have compute nodes write only tiny log records into it. Writes need 4/6 storage copies (a quorum — Lesson 3's arithmetic, applied to disks); reads need 3/6. Read replicas share the same storage, so "replication lag" collapses to cache-invalidation time, typically ~10–20ms.

Operate the 6-copy quorum

Six storage copies, two per AZ. Click copies to fail them (or lose a whole AZ at once), then test writes (need 4/6) and reads (need 3/6).

AZ-a AZ-b AZ-c

The map: every service, classified

SystemArchitecture (Lessons 1–4)Wire formatDefining consequence
RDS Multi-AZSingle-leader, synchronous standbyPhysical/blockRPO 0 failover; standby serves no reads
RDS read replicasSingle-leader, asyncEngine WAL/binlogRead scale with lag; promotion loses the tail
AuroraSingle-leader compute + quorum storage (4/6)Log records to shared storage~10–20ms replica lag, seconds-fast failover
DynamoDBLeaderless quorum per partition; global tables = multi-leader LWWInternal item eventsNo failover events; LWW discards on conflict
Cassandra / ScyllaLeaderless, tunable W/R per queryRow mutationsConsistency is a per-request dial
MongoDB replica setSingle-leader with Raft-style electionsOplog (logical)Self-healing leadership, majority write concern available
etcd / Consul / ZooKeeperConsensus (Raft/Zab)Replicated log entriesLinearizable; minority partitions go read-only
CockroachDB / TiDB / SpannerConsensus per data range (thousands of groups)Raft/Paxos log entriesSQL with serializable guarantees, majority RTT per commit
KafkaSingle-leader per partition, ISR quorum acksThe log IS the dataReplication and storage are the same mechanism
Redis / ElastiCacheSingle-leader, asyncCommand streamFast, small loss window accepted by design
Course complete — the whole field in four questions. ① Who may accept a write? (one node / many designated / anyone / an elected majority-holder). ② When is "committed" true? (locally / on k confirmers / on a quorum / on a majority). ③ What travels? (statements / physical bytes / logical rows / log records to shared storage). ④ When copies disagree, who wins? (the leader's log / LWW / siblings / CRDTs / it's impossible by arithmetic). Classify any database — current or future — by answering these four, and you'll know its failure modes before its documentation admits them.

Check yourself

1. You need to upgrade PostgreSQL 14 → 16 with near-zero downtime. Which replication mechanic makes this possible, and why?

Physical WAL is bytes-on-disk, and disk layouts differ across major versions — a v16 server can't replay v14 WAL. Logical events ("row id=7 changed to X") are format-neutral, enabling the standard playbook: logical-replicate into the new version, catch up, flip traffic. (Statement-based would also cross versions but reintroduces every nondeterminism bug from the first lab.)

2. Why does Aurora tolerate an entire AZ failing (2 copies) plus one more copy, for reads?

Lesson 3's inequality wearing storage clothes: W=4, R=3, N=6 → W+R=7 > 6, so any read quorum intersects any write quorum. AZ down (−2) plus one more failure (−1) leaves exactly 3 — reads survive; writes (needing 4) pause until a copy is rebuilt, which storage does automatically.

3. Your team writes to PostgreSQL and then, in application code, also pushes the same change into a search index. What is the CDC argument against this?

The app writing to two systems is multi-master replication implemented by hope: no atomicity, no ordering guarantee, no replay after failure. CDC (Debezium-style log tailing) turns the search index into an honest replica of the database's own committed log — the exact architecture this course has been drawing for five lessons, applied one more time.