Why the p99 chart lies to you

Tied to Mission: Backend Depth — the mission's first success criterion is diagnosing a live issue from the shape of the symptom, and the baseline assessment's first evidence was "no specific tool/metric named." Every prior lesson taught a mechanism; this one teaches the instrument you use to see those mechanisms misbehave.

2am. The alert says response time is up. The dashboard shows the average is fine, CPU is flat, error rate is zero. The p99 line shows 5 seconds — but nobody trusts the p99, because the last engineer who "fixed" it averaged the per-instance p99s, and the chart has been wrong since. Even when the numbers are right, they don't tell you which service to blame. You have monitoring, but you can't see. Observability is the discipline of turning data into a diagnosis — and it starts with knowing which number means what, and which numbers lie.

The three pillars: metrics, logs, traces

Three data types, each answering a different question, each with a different cost:

• Metrics — aggregated numbers over time (request rate, p99 latency, queue depth). Cheap to store, always on, perfect for alarms and for seeing the shape of a problem. They tell you that something is wrong.
• Logs — discrete events with high-cardinality detail (a specific request ID, an error message). Expensive to search at scale, so they can't be always-on for everything. They tell you the details of one thing once you know where to look.
• Traces — the causal path of one request through every service and database call it touched, as spans linked by a trace ID. Sampled (recording every trace is too expensive), so they're statistical, not complete. They tell you where the time went for a request.

The drill-down is the workflow that uses all three: an alert on a metric fires → you find a representative failing request → its trace shows which hop is slow → the logs for that span give the error. Metrics point, traces locate, logs explain.

RED vs USE: which numbers to watch, and where

Two acronyms organize monitoring by what you're measuring, and they cover different layers:

• RED — for services: Rate (requests/sec), Errors (failed requests/sec or %), Duration (latency distribution). RED tells you whether the service is doing its job. This is what you alarm on for your own API.
• USE — for resources: Utilization (what fraction of capacity is busy: CPU%, disk space), Saturation (how much work is waiting: run-queue length, connection-pool depth, queue depth), Errors (device errors). USE tells you whether a machine is exhausted. This is what you look at when RED says the service is fine but slow.

They pair with Google SRE's four golden signals — latency, traffic, errors, saturation — which are RED plus saturation folded in. Saturation is the interesting signal because it's the cause hiding behind the latency effect: queue depth growing is Lesson 13's Little's law firing, and it precedes the p99 rise by definition. Lesson 21 will lean on USE for the traffic path; Lesson 10's throughput-collapse curve is a saturation story; Lesson 11's "memory slowly growing" is a USE-style signal.

Percentile math: why averages hide the tail, and p99s lie when averaged

Latency is a distribution, not a number. A few very slow requests barely move the mean — 1% of requests at 5s barely lifts an average built from 99% at 50ms — so averages are structurally blind to the tail, and the tail is what users feel and what timeouts are built around (Lesson 7's timeouts are set from tail latency, not the mean). This is why p99/p999 exist: they describe the tail. Three traps that make them lie:

• The aggregation trap: you cannot average percentiles. Instance A's p99 is 50ms, instance B's is 200ms; the fleet p99 is not 125ms — percentiles of a merged distribution must be computed from the merged raw data (or from merged histograms). Averaging per-instance p99s is the single most common way monitoring charts become fiction.
• The mean-vs-tail mismatch: reporting the mean as "the latency" hides the shape. The mean can be fine while 1% of users wait 10× longer — and those are the users whose requests time out and retry (Lesson 7), turning a latency problem into a load problem.
• Tail at scale: for a request that fans out to many services (Lesson 18's scatter-gather, or a page composed of 30 sub-requests), the response time is set by the slowest sub-request, not the average. Dean & Barroso's "The Tail at Scale" shows why: with N components, the probability that some component is slow grows with N, so the tail gets worse the more you parallelize — and the fix is redundancy (send two requests, use the first), not just faster components.

SLOs and error budgets: alarms that don't scream

A service level objective is a quantified promise: "99.9% of requests complete in under 200ms, measured over 30 days." Two numbers it unlocks:

• Error budget = 100% − SLO. At 99.9%, you may fail 0.1% of requests per month — about 43 minutes. The budget is a decision tool: while it lasts, you can ship; when it's gone, fixes get priority. It converts "reliability" from a vibe into a spendable resource.
• Burn rate — how fast you're consuming the budget. Alerting on burn rate (e.g., "burning through a month's budget in 24 hours") catches sustained problems without paging you on every single bad minute. The alternative — alerting on any 5-minute SLO violation — is why dashboards scream at night: single bad minutes are normal variance (Lesson 13's tail), sustained budget burn is a real outage.

Symptom shapes: the diagnosis habit the mission is built on

The mission's first success criterion — reason from the shape of the symptom before guessing — maps directly to what your charts show:

• Sudden spike vs. slow bleed. A sharp step-change in latency or errors at a moment in time means something changed then: a deploy, a traffic shift, a dependency. Correlate with deploy timelines and traffic charts first. A slow, steady degradation over hours means an accumulator: a memory leak, connection-pool leak, thread leak, or queue growth — the Lesson 11/13 shapes. The tell: if a restart clears it, it was an in-memory accumulator, not a load problem. That single test was the baseline assessment's opening question, and it's the shape-vs-root-cause map in one line.
• Correlated vs. isolated. All instances degrade together → a shared dependency: the database, the load balancer, the network, a downstream API. One instance degrades while others are fine → a local resource: CPU, disk, GC, a bad node (Lesson 19's cluster and Lesson 21's traffic path both create this pattern).
• Which RED signal changed. Errors up → look for code changes, bad inputs, or a failing dependency (circuit breaker opening, Lesson 7). Latency up with CPU flat → saturation or network (queueing, Lesson 13; TCP retransmission, Lesson 10). Latency up with CPU up → computation: the query plan changed (Lesson 9), a loop regressed, GC pressure.

None of this replaces investigation — it orders it. The shape tells you which class of root cause to check first, and every class maps to a lesson in this curriculum. That's the entire point of the pass.

Check yourself

Try it for real

Primary sources

The Google SRE book — "Monitoring Distributed Systems", "Service Level Objectives", and "Practical Alerting" — the source of the four golden signals, SLOs, error budgets, and burn rates. Tom Wilkie's "The RED Method" and Brendan Gregg's "The USE Method" define the two monitoring frameworks. Dean & Barroso, "The Tail at Scale", is the reference for why fan-out makes tails worse and what to do about it. Peter Bourgon's "Metrics, Tracing, and Logging" is the shortest good statement of what each pillar is for.

The lens is set. The final lesson of the pass points it at the most common recurring incident in the industry — what happens to a request during a deploy — Lesson 21: the traffic path.