Tied to Mission: Backend Depth — and a direct sequel to the baseline debugging assessment's resource-exhaustion scenario, now with a second service in the picture.
A local function call fails in a small number of well-understood ways: it throws, or it returns. A call to another service over the network can fail in a way a local call never does — it can go quiet. The request left. Something happened, or didn't. The response never came back, or came back too late to matter. You genuinely do not know whether the downstream service never received the request, received it and is still working on it, or received it, finished it, and the response was lost on the way back.
This is called partial failure, and it's the defining trait of distributed systems versus single-process code. If a local function throws, you know it didn't complete. If a network call times out, "did the write happen?" has three possible answers, not one, and from the caller's side alone you can't tell which. Retrying a failed read is usually safe. Retrying a failed write without knowing whether it already landed is how duplicate charges and double-sent messages happen — a separate problem (idempotency) that this lesson only flags, not solves.
Now put this in a system with real load. A downstream service is under heavy traffic and starts responding slowly — some requests time out. Nothing is fully down yet; it's degraded. Every caller notices the timeout and, reasonably, retries. But if every caller retries immediately, each failed request just became two requests. If the retry also times out and gets retried again, it becomes three, four, more — all landing on a service that was already too slow to keep up with the original load.
The baseline debugging scenario was one service running out of a resource (connections, threads, memory) under load, and the symptom compounding rather than leveling off. A retry storm is that same compounding shape, produced across a service boundary instead of within one process: the retries themselves are the added load, generated by callers trying to help.
The result: a service that was slow becomes a service that is fully down, because the very mechanism meant to paper over transient failures (retry) added the load that pushed it over the edge. This is retry amplification — and it can happen with no bug in either service's business logic. It's purely a property of how the calling side reacts to failure.
None of these is optional on its own — they solve different parts of the problem and are normally used together for any call that crosses a network boundary.
| Mitigation | What it does | Why it's needed |
|---|---|---|
| Bounded timeout | Give every outbound call a deliberate, finite time limit — never rely on the OS or library default, and never leave it unbounded. | Without one, a slow downstream response ties up the caller's own resources (threads, connections) waiting indefinitely, spreading the degradation upstream. |
| Exponential backoff + jitter | Each retry waits longer than the last (exponential backoff), and the exact wait is randomized within a range (jitter) rather than fixed. | Backoff alone still has every caller retry in lockstep at the same intervals, re-synchronizing into a new wave of load. Jitter spreads retries out in time so they don't all land on the downstream service at once. |
| Circuit breaker | Tracks recent failures for a dependency; once failures cross a threshold, it "opens" and fails calls immediately without attempting the network call at all, then periodically lets one call through ("half-open") to test for recovery. | Timeouts and backoff still keep hammering a dependency that is clearly not coming back soon. A circuit breaker stops that — fail fast, protect the struggling dependency, and give it room to recover. |
Find the retry logic your own service uses for an outbound HTTP call (or check the default retry behavior of an HTTP client library you rely on). Does it wait between attempts? Does the wait grow on each retry? Is there any randomization (jitter), or would every instance of your service retry at the exact same intervals if they all failed at the same moment?
Everything above assumes the downstream service is still reachable, just slow or overloaded. A harder case is a genuine network partition — the caller and the service can't reach each other at all, even though both are individually healthy. When that happens for an operation that needs both sides to agree (e.g. a write that must be confirmed by a replica), you have to choose: answer anyway using possibly-stale local data (favor availability), or refuse to answer until you can confirm with the other side (favor consistency). This is the shape the CAP theorem describes. It's a deep topic in its own right — flagged here as a forward pointer, not covered further in this lesson.
Martin Kleppmann, Designing Data-Intensive Applications — the reliability discussion in Chapter 1 frames why distributed failure modes need deliberate handling rather than being treated as rare edge cases, and Chapter 8, "The Trouble with Distributed Systems," covers partial failure and unreliable networks in depth. For going further on the theory underneath all of this — consensus, fault tolerance, how systems agree on truth when parts of them are failing — see MIT 6.824, Distributed Systems.
This closes the initial pass through the curriculum — indexing, isolation, caching, connections, OS scheduling, concurrency, and now distributed failure models. Good moment to ask for a recap across all of it, revisit any earlier lesson that didn't fully land, or ask follow-up questions on anything above directly.