Reference · Distributed Systems / Failure Models

Distributed Systems / Failure Models — Cheat Sheet

Pairs with Lesson 7: Retries and Partial Failure.

Glossary

TermMeaning
Partial failureA network call fails ambiguously — the caller can't tell whether the request was never received, was received and is still processing, or was processed and only the response was lost.
TimeoutA deliberate, bounded time limit a caller places on waiting for a response, after which it treats the call as failed even though the true outcome downstream is still unknown.
Retry storm / retry amplificationMany callers retrying a failing or slow dependency add load on top of the load that caused the failure in the first place, deepening the outage rather than riding it out.
Exponential backoffEach successive retry waits longer than the last (e.g. doubling), instead of retrying at a fixed interval.
JitterRandomizing the exact wait time within a range so retries from many callers don't stay synchronized and re-collide on the same instant.
Circuit breakerA guard around a dependency with three states — closed (calls go through normally), open (calls fail immediately, no network attempt, after failures cross a threshold), half-open (after a cooldown, a limited number of calls are let through to test whether the dependency has recovered).
CAP theoremUnder a network partition, an operation that needs agreement across nodes must choose between answering with possibly-stale data (favoring availability) or refusing to answer until agreement is possible (favoring consistency) — one line, not the full model.

Decision rule

The default policy for any network call

Every network call needs a bounded timeout and a backoff-plus-jitter retry policy. Every dependency you retry against eventually needs a circuit breaker, so a struggling dependency gets protected instead of buried.

Primary sources