Tied to Mission: Backend Depth — Lesson 7 flagged idempotency as the unsolved half of retrying writes. This is that lesson: the mechanism that makes retries safe.
Lesson 7 set up the problem: a client calls POST /charges, the request times out, and you genuinely don't know whether the server received it, processed it, and lost the response — or never got it at all. The response to a timeout is a retry. But if the first attempt did land, the retry is a second charge. The customer gets billed twice, and the payment service did exactly what it was told. The problem isn't the retry; it's that retrying an operation that isn't safe to repeat is a new operation.
An operation is idempotent if applying it N times has the same effect as applying it once. The HTTP verbs encode this by design: GET is naturally idempotent, PUT (full replacement) and DELETE are generally idempotent, and POST — "create a new thing" — is not. "Create a new charge" executed twice creates two charges. That's the entire problem in one sentence.
So there are two ways to make a retry safe: make the operation idempotent (so repeating it is harmless), or make the request recognizable (so the server can detect the repeat and not re-execute it). Production APIs use the second one, because you usually can't make "create a charge" idempotent by itself — every call is genuinely a new creation.
The standard design, used by Stripe and most payment systems:
Idempotency-Key). Retries of the same operation reuse the same key.Now the retry from Lesson 7 changes meaning: it's no longer "do this again, hoping it didn't happen" — it's "give me the outcome of the thing I already asked for." The ambiguous timeout resolves itself: if the first attempt landed, the stored result comes back; if it never did, the operation runs now. Either way the client gets exactly one charge and one answer.
The key must be unique per logical operation, not per client. Reusing one key for two different payments tells the server "these are the same operation" — the second payment silently returns the first one's result. A client that generates one key and reuses it across retries of different operations corrupts the semantics. The rule: new logical operation → new key; retry of the same operation → same key.
UPDATE ... WHERE balance >= x (Lesson 2's fix) is idempotent by construction, as is INSERT ... ON CONFLICT DO NOTHING — "add to the set" semantics. Choosing the right layer is a design decision: keys handle arbitrary workflows; data-layer idempotency handles specific writes without any client cooperation.Over an unreliable network, exactly-once delivery is unachievable — the sender can't distinguish "lost" from "processed but ack lost," so it must retry (at-least-once) or risk losing data. What the idempotency key buys is the next best thing: at-least-once delivery + deduplication = effectively-once. Nothing is lost, nothing is doubled. Every time someone claims a system does "exactly once," what they mean — if they're being careful — is this combination. The retry is the mechanism that guarantees no loss; the key is the mechanism that guarantees no duplication. (This is also why Lesson 7's lesson matters here: an idempotency key doesn't help if every caller retries immediately and synchronously — the dedup prevents double-execution, but backoff and jitter still prevent the storm.)
POST /payments and retries the identical request. The server processed the first attempt, but the response was lost. There is no idempotency mechanism anywhere. What happens?Go back to Lesson 7's homework — the retry logic your own service uses for outbound calls — and add the second question: is the operation being retried idempotent? If it's a read, yes. If it's a write, what makes repeating it safe? Then check your inbound API: does any POST endpoint accept an idempotency key, or would a duplicate request double-create? A quick way to feel the mechanism: implement a tiny key store in code — a dict mapping key → response, checked before executing — and verify that a "retry" with the same key returns the stored result without re-running the handler.
Stripe's documentation on idempotent requests is the canonical production description of the pattern — including key format, storage, and expiry. Brandur Leach's essay "Idempotency Keys" walks through a reference implementation and the edge cases (racing requests, key expiry) that this lesson compressed. For the "exactly-once is a lie, effectively-once is the truth" framing, see Designing Data-Intensive Applications, Chapter 11, which dissects exactly-once semantics in message systems.
Idempotency makes single retries safe. The last lesson of this pass zooms out to the whole system: replicas, lag, and the consistency tradeoffs that Lesson 7 only pointed at — Lesson 15.