Resilience: how systems eat themselves, and how to stop it
Every lesson so far studied failures that happen to a system. This one studies failures a system does to itself. The ingredients are always the same: a partial slowdown, plus a well-intentioned mechanism (retries, health checks, queues) that responds by adding load, closing a feedback loop that turns a blip into an outage. The biggest internet outages of the last decade are mostly this shape.
The retry storm: kindness as a weapon
Lesson 1 taught: on timeout, retry. Correct advice — for one client. Now put that advice in every client at once, aim it at a service that's slow rather than dead, and watch the arithmetic:
Melt a service with politeness
The pricing service handles 100 req/s at capacity 120. Degrade it, watch timeouts begin, and let every client "helpfully" retry ×3. Then arm the two defenses and run it again.
offered load vs capacity (120 rps)
base demand
100 rps
offered load
100 rps
capacity
120 rps
success rate
100%
The mechanism you watched: failures breed retries, retries raise offered load, higher load raises the failure rate, which breeds more retries — a positive feedback loop with the service's corpse at its fixed point. Notice also what happens when the degradation ends: the accumulated retry queue keeps hammering, so the service can't recover — a metastable failure, stable in its broken state. The defenses attack the loop's gain: exponential backoff spreads retries over time, jitter de-synchronizes the herd (a thousand clients backing off identically just schedules a thousand-client stampede for the same later instant), and a retry budget caps total retry traffic to a small percentage of base load, so retries can never dominate. First-attempt traffic gets priority; retries are a luxury the system funds only when healthy.
The circuit breaker: giving up as a service
Backoff slows the hammering; the circuit breaker stops it. Wrap each dependency in a tiny state machine with three states — and note that the pattern's real gift isn't protecting the dependency, it's failing fast: callers get an instant error (or a fallback) instead of burning a thread for 30 seconds on a doomed call. Threads not burned = your own service not dragged down = the cascade cut at one hop.
Operate the breaker
Send requests to a dependency you control. Break it, trip the breaker (3 consecutive failures), watch calls fail instantly, then let the recovery timer expire and probe through the half-open gate.
CLOSEDrequests flow · failures counted
OPENfail instantly · no calls made
HALF-OPENone probe allowed
consecutive failures
0 / 3
calls that reached dep
0
instant-failed (saved)
0
The rest of the toolkit, briefly:load shedding — when saturated, reject cheap and early (HTTP 503 at the front door) rather than queue; a request served in 30s is usually worth less than an honest instant no. Bounded queues — an unbounded queue converts overload into latency and memory pressure and postpones the reckoning; small queues fail fast and keep latency honest. Bulkheads — per-dependency thread/connection pools, so one slow dependency can drown only its own pool, not the process. Timeouts budgets — a caller 3 hops deep passing its remaining deadline downstream, so nobody works on requests the user already abandoned. Every one of these is the same idea in different clothes: turn implicit unbounded coupling into explicit bounded contracts.
Course complete
The six lessons compose into one worldview. The network gives you nothing for free (L1), not even a shared clock (L2). You split data to scale and inherit new balance problems (L3), you coordinate across machines and pay in locks or in visible intermediate states (L4), you choose what reads are allowed to say and pay for the strength you pick (L5), and you wire it all together with bounded, fail-fast contracts so the pieces can't take each other down (L6). Paired with the replication course — copies, quorums, consensus — you now have both halves of the discipline: how to have the data in many places, and how to survive the places disagreeing.
Check yourself
1. A dependency slows from 20ms to 2s. Your service has a fixed 200-thread pool and keeps calling it synchronously with a 5s timeout. What happens to YOUR service?
Little's law: concurrency = rate × duration; 100 × 2 = 200 = your entire pool. This is the cascade mechanism: slowness converts into resource exhaustion in every caller, hop by hop, until the edge. Bulkheads cap how many threads one dependency may consume; breakers stop the doomed calls; both exist because of this arithmetic.
2. Why is jitter added to exponential backoff, rather than backoff alone?
Deterministic backoff preserves the herd's synchronization perfectly — the outage that started at the same instant for everyone produces retry spikes at identical instants forever. Full jitter (retry at random(0, backoff)) is one line of code and flattens the waves; AWS's classic blog post measured it cutting total work dramatically.
3. Half-open allows exactly one probe request instead of reopening the floodgates. What failure mode is this protecting against?
A service crawling back at 10% capacity greeted by 100% of traffic dies again immediately — and every client's breaker slams open in unison, then retries in unison: a system-wide oscillation. The single probe is a minimal-cost hypothesis test; only proof of health earns reopening. Gradual ramp-ups (10% → 50% → 100%) extend the same idea.