Lesson 04 · All or nothing, across machines

Distributed transactions: atomicity meets the network

Inside one database, atomicity is a solved problem — the WAL commits or it doesn't. But the interesting operations in a real company span systems: reserve inventory in one service, charge the card in another, book the shipment in a third. "All of these or none of these" now has to be built on top of Lesson 1's unreliable network — where any participant, and any message, can vanish at any point in the protocol. Two families of solution exist, and they occupy opposite corners of the trade-off space.

Family 1: two-phase commit — real atomicity, at a price

2PC adds a coordinator and splits commit in two. Phase 1 (prepare): every participant durably readies the transaction — locks held, changes staged — and votes YES or NO. Phase 2 (commit/abort): if every vote was YES, the coordinator orders commit; any NO, abort. A participant that voted YES has promised it can no longer fail to commit — that promise is the protocol's power and its trap:

Run 2PC — then crash the coordinator at the worst moment

Step through the happy path first. Then reset and use the crash button after prepare, before the decision arrives — and look at what the participants can and cannot do.

COORDINATOR idle INVENTORY svc stock: 5 · no locks PAYMENTS svc card ok · no locks
Ready
Order #42 needs: reserve 1 unit AND charge €50, atomically. Press Next step to begin phase 1.

The crash scenario has a name: the participants are in doubt. Having voted YES, they may neither commit (the decision might have been abort) nor abort (it might have been commit) — they hold their locks and wait for a coordinator that may be gone for minutes. Rows stay locked, connection pools drain, and the blockage spreads to every transaction that touches those rows. This blocking property is intrinsic to 2PC, not an implementation bug — and it's why 2PC (XA transactions, PostgreSQL PREPARE TRANSACTION) is used sparingly, inside single administrative domains, with fast coordinator recovery. (The deeper fix — make the decision itself replicated and majority-elected so no single coordinator's death can hide it — is Raft-style consensus from the replication course; Spanner runs 2PC over Paxos groups for exactly this reason.)

Family 2: sagas — give up atomic, keep the business correct

The saga pattern refuses to hold cross-service locks at all. Run the steps as ordinary local transactions, one after another; if step k fails, run compensating actions for steps k−1 … 1 in reverse. No coordinator lock-step, no in-doubt state, every service stays available — and in exchange, the intermediate states are visible to the world:

Book a trip; watch it unwind

Flight → hotel → rental car, as a saga. The car service will fail. Watch the compensations run in reverse — and watch what the customer's email inbox sees along the way.

1 · book FLIGHTpending 2 · book HOTELpending 3 · book CARpending
Ready
Each step is a normal local transaction in its own service, no cross-service locks. Advance.

What the inbox showed is the saga's honest cost: no isolation. Between step 2 and the compensation, the hotel booking really existed — other systems saw it, emails fired, loyalty points accrued. Compensations are new forward actions ("cancel"), not rollbacks — history isn't erased, it's answered. Designing a good saga is therefore a domain-modeling exercise: every step needs a defined compensation, compensations must be idempotent (Lesson 1 — they will be retried), and the business must tolerate the visible intermediate states — the way airlines tolerate "reserved, payment pending" as a first-class concept. Orchestration engines (Temporal, Step Functions — whose .waitForTaskToken patterns you've met) exist mostly to run these state machines durably.

2PCSaga
AtomicityReal — all-or-nothing, invisible intermediate statesEventual — intermediate states visible, compensated afterwards
Locks heldAcross the whole protocol, on every participantOnly inside each local step
Failure behaviorCoordinator loss ⇒ participants blocked in-doubtAny step's failure ⇒ compensations; nothing blocks
Best whenShort transactions, one admin domain, strong invariants (funds transfer within a bank)Long-lived, cross-organization workflows (travel, orders, onboarding)

Check yourself

1. A participant voted YES in phase 1 and then lost contact with the coordinator. Why can't it just time out and abort?

The YES vote was a binding promise precisely so the coordinator could decide unilaterally afterwards. Once the decision might exist, acting against it risks the split outcome 2PC exists to prevent. Hence in-doubt participants hold locks and wait — the blocking that defines the protocol's cost.

2. A saga's "cancel hotel booking" compensation gets delivered twice due to a retry. What property must it have, and from which lesson?

Sagas run on the same unreliable network as everything else: compensations time out and get retried, and "did my cancel go through?" has the same four indistinguishable answers. Every saga step and every compensation needs an idempotency key or naturally idempotent semantics. The course's lessons compose.

3. Transferring €100 between two accounts in the same bank's core ledger vs. an e-commerce order spanning inventory, payment, and shipping services. Which mechanism fits which?

The choice follows the invariant's tolerance for exposure. Accounting identities must hold at every observable instant → real atomicity, short and local. Commerce workflows have legitimate in-between states and span organizations that will never share a lock manager → saga. "Which states may the world see?" is the deciding question, not fashion.