Lesson 21 · Operations

Why every deploy causes a 5xx blip

Tied to Mission: Backend Depth — ties together the scheduler (Lesson 5), the event loop (Lesson 12), backpressure (Lesson 13), and replication (Lesson 15) into one story: what happens to a request when you deploy.

2am. The deploy tool rolls three instances, one at a time. Error rate blips for 90 seconds, then recovers. The logs show connection reset by peer — no application error, no exception. The code is fine. The deploy procedure isn't: the load balancer was still sending traffic to an instance that was already gone. This is the most common recurring incident in the industry, and it's fully explainable — the request's path to your code has its own lifecycle, and a deploy interrupts it mid-flight.

The path a request takes

Client → DNS → load balancer → instance. Two ideas make the whole path legible:

L4 vs L7 load balancing. An L4 load balancer works at the transport layer: it forwards TCP connections (or packets) to backends based on IP and port — fast, protocol-agnostic, can't see URLs or headers. An L7 load balancer terminates the connection, reads the HTTP request, and can route by path or host, run health checks, retry, and terminate TLS. Most backend request handling happens behind L7 (nginx, Envoy, cloud LBs); L4 is the layer underneath when raw throughput and protocol-agnosticism matter. When someone says "the load balancer," ask which layer — it determines what the thing can see and do.
Health checks: how the LB knows who's alive. Active checks probe an endpoint (/healthz) on a cadence; passive checks watch real traffic and remove instances that start erroring. A failed check removes the instance from rotation — but only on the next check cycle. The interval between "instance dies" and "LB notices" is a window where traffic still goes to a dead node. That window, plus how the instance handles the shutdown, is the entire deploy incident.

The deploy: mark, drain, kill

A graceful rollout of one instance is three steps, and skipping any of them produces the 5xx blip:

Mark unhealthy. The instance tells the LB to stop sending new requests (via a health-check endpoint flipping to failing, or the LB draining it). Traffic stops arriving — eventually, after the next check cycle.
Drain. The instance keeps processing the requests already in flight. The LB holds the connection open — the drain timeout — while in-flight work finishes. The drain window is a bounded queue (Lesson 13): if requests take longer than the drain timeout, the LB force-closes them. This is why drain timeouts, max request durations, and health-check cadence have to be tuned together.
Exit. Only now does the instance terminate, with nothing left in flight.

On the server side, the same sequence has a name: graceful shutdown. The process receives SIGTERM, stops accepting new work, finishes what's in flight, flushes, and exits. What kills in-flight requests is the opposite: handling SIGTERM by exiting immediately (or being SIGKILLed). Every in-flight request dies mid-response — the client sees a connection reset — exactly the error in the 2am alert. The mechanism is the same one from Lesson 12's event loop (drain the loop, then stop) and Lesson 5's threads (let each thread finish its current request), and the same principle as Lesson 16's relay: never kill work mid-flight if you can finish it first.

The restart is a mini-deploy

Every time you restart a process — a crash recovery, an OOM-kill bounce, a manual "just restart it" — the same three steps happen, whether you orchestrate them or not. An ungraceful restart is a mini-deploy that skips mark-and-drain: the LB still thinks the instance is healthy, in-flight requests die, and the error blip is indistinguishable from a deploy's. This is also why the baseline assessment's "restart fixes it" pattern deserves a second look: if a restart fixes something, the restart itself may have caused a blip — and the thing it fixed was an accumulator (Lesson 20's shape map).

Retries and hedging: the LB as the client's safety net

Because an L7 LB can see requests, it can also protect them — this is where earlier lessons show up as features of the path, not just failure modes:

Retries at the proxy. The LB can retry a failed request on another instance. That's Lesson 7's retry policy, now executed centrally — which means its rules apply: bounded retries, or the retry amplifies the very failure it's papering over (retry storms during a partial outage are the classic LB-induced outage).
Hedging. For the tail (Lesson 20), send the request to two instances and use the first response. Costs 2× load on the rare requests it's used for; buys back the slowest-request tail that fan-out multiplies (Dean & Barroso's answer).
Connection limits as saturation. The LB's per-instance connection limit is a bounded queue (Lesson 13): when it's full, new requests fail fast (or queue briefly) instead of stacking up on the instance — the difference between load shedding and memory death.

The pattern, named

A request's life has three phases — in flight, draining, gone — and every system that touches requests (LBs, orchestrators, deploy tools, the process itself) must agree on which phase each instance is in. The deploy incident is what happens when they disagree: the LB thinks an instance is serving, the instance thinks it's done. Lesson 15's replica removal had the same shape — the read path must stop using a replica before it's gone — and Lesson 13's queues have the same lesson at a lower level: the consumer must stop accepting work it can't finish. The fixes are all the same three words: mark, drain, kill.

Check yourself

Your instance gets SIGTERM and immediately exits. The LB health check runs every 5 seconds and last saw the instance healthy 1 second ago. What happens to the requests in flight?

Right — the health check's last pass is only a second old, so the LB keeps routing; the instance exits with requests mid-flight, and those clients see resets. Mark → drain → kill is what makes a deploy invisible: stop new traffic, finish in-flight, then exit. Not quite — the LB only knows what its last health check told it, and that was a pass. Immediate exit kills in-flight requests (connection reset). And retries only happen if the LB is configured for them, and even then only for requests it can safely re-send. Re-read the mark/drain/kill section.

Try it for real

Hands-on

Reproduce the blip in ~60 lines of Python, no infrastructure. 1) The server: a socket server that accepts a connection, sleeps 2s (simulating a slow request), and writes a response. Install a SIGTERM handler that exits immediately. Start it, send a request from another terminal, and SIGTERM it mid-request: the client sees the connection die — the reset. 2) The fix: change the handler to (a) stop accepting new connections, (b) wait for the in-flight socket to finish, (c) then exit — and verify the client now gets its response before the process exits. 3) The LB half: write a 15-line round-robin forwarder over two server instances with an active health check (try to connect every 100ms). Kill one instance and watch how many requests fail before the health check notices — that count is the "check interval" window, and shrinking it (or having the dying instance flip its own health to failing) is exactly the draining fix in miniature.

Primary sources

nginx's upstream module docs and Envoy's health checking docs are the concrete L7 primitives: active/passive checks, draining, retries. Google's SRE book, "Load Balancing at the Frontend", covers the request path at scale. Kubernetes' pod lifecycle documentation is the reference for how orchestrators implement mark/drain/kill (readiness probes, preStop hooks, terminationGracePeriodSeconds). For the tail and hedging, Dean & Barroso's "The Tail at Scale" again — this time for its hedging section.

That closes the third pass: outbox and sagas, the log, partitioning, consensus, observability, and the traffic path. The full curriculum now runs from a single index to two leaders and back. Ask for a recap that crosses all 21 lessons, or drill into anything that didn't land.

← Lesson 20 Reference: Observability Cheat Sheet