messengers sent
0
Every hard thing in distributed systems descends from one physical fact: between two machines there is a network, and the network is allowed to do anything. Drop a message. Delay it ten seconds. Deliver it twice. Deliver it after you've given up. And crucially — when you send a request and hear nothing back, the network offers you no way to distinguish "the request was lost," "the server crashed," "the server is slow," and "the server did the work but the reply was lost." One symptom, four realities.
Before building anything on such a foundation, it's worth knowing what's provably impossible on it. The 1975 result that opens every textbook:
Two armies must attack a city simultaneously or lose. They coordinate by messengers who cross a valley where they may be captured. Can the generals reach certainty that both will attack? Try:
Send messengers back and forth. Each arrival helps — and creates a new uncertainty of its own. Follow the "who's sure of what" tracker and try to get both generals to certain.
The proof is the pattern you just felt: whatever the last successfully delivered message was, its sender can't know it arrived — so its sender's willingness to attack must not depend on it. Induct backwards and no finite number of messengers produces common knowledge. The engineering moral is liberating, not depressing: certainty over an unreliable channel is off the menu, so real systems buy probability instead — acknowledgements, retries, timeouts — and design so that the residual uncertainty is harmless rather than absent. Which brings us to the harm:
A client charges a card. The request times out. The client is now standing exactly where General A stood — and what it does next separates correct systems from incident reports:
Each attempt randomly: arrives & acks, arrives but the ack is lost, or never arrives. On timeout, retry — first naively, then with an idempotency key. Watch the server's ledger, not the client's beliefs.
What the toggle changes is profound. Without it, the server treats every arrival as new work: retries are dangerous, so timeouts become paralyzing. With it, the server keeps a small table — "key ord_7f3a: already processed, result was OK" — and duplicate arrivals return the stored result instead of re-executing. Retrying is now safe, which means the client can retry aggressively, which means the unreliable network stops being scary. The formula worth memorizing: exactly-once effect = at-least-once delivery + idempotent processing. Nobody delivers exactly once; correct systems process as if they did.
One loose end from the lab: how long should the client wait before retrying? There is no correct answer, only a trade. Too short, and you retry against a server that's merely slow — duplicating work and adding load exactly when the system is struggling (Lesson 6 shows this becoming a self-inflicted outage). Too long, and users stare at spinners while a genuinely dead server wastes their time. Real systems pick timeouts from observed latency percentiles (e.g., p99 × a small multiplier), adapt them continuously, and — because a timeout is a guess, not a fact — pair every retry with the idempotency machinery you just built. The three-legged stool of this lesson: timeouts to suspect, retries to persist, idempotency to make persistence safe.