Pairs with Lesson 20: Why the p99 chart lies to you.
| Term | Meaning |
|---|---|
| Metric | Aggregated number over time (rate, latency percentile, queue depth). Cheap, always on, good for alarms and symptom shapes. Tells you that something is wrong. |
| Log | Discrete event with high-cardinality detail. Expensive to search; used after metrics and traces point to a place. Explains why. |
| Trace / span | The causal path of one request across services: spans linked by a trace ID. Sampled. Locates where the time went. |
| RED method | Service-level signals: Rate, Errors, Duration. Answers "is the service doing its job." |
| USE method | Resource-level signals: Utilization, Saturation, Errors. Answers "is the machine exhausted." |
| Four golden signals | Latency, traffic, errors, saturation — Google SRE's list. Saturation (queue depth, run queue) is the cause that precedes the latency effect (Little's law). |
| Percentile / p99 | The value below which 99% of observations fall. Describes the tail, which the mean hides and which timeouts are set from. |
| Aggregation trap | Percentiles don't average. Fleet p99 must come from merged raw data/histograms; averaging per-instance p99s is chart fiction. |
| Tail at scale | Fan-out latency is set by the slowest component; the more parallel components, the likelier one is slow. Fix with redundancy (hedging), not faster components. |
| SLO | Quantified reliability promise ("99.9% of requests < 200ms over 30 days"). |
| Error budget | 100% − SLO. Spendable: ship while it lasts, fix when it's gone. |
| Burn rate | How fast the error budget is consumed. Alert on sustained burn, not on single bad minutes. |
| Symptom shapes | Spike vs. slow bleed; correlated vs. isolated; which RED signal changed. Each shape maps to a root-cause class (see below). |
Alert on a metric (burn rate, not single bad minutes) → pull a representative failing request → read its trace to find the slow hop → open the logs for that span. Metrics point, traces locate, logs explain.
Sudden spike = something changed (deploy, traffic, dependency) — correlate first. Slow bleed over hours = an accumulator (memory/connection/thread leak, queue growth); if a restart clears it, it was in-memory. All instances degraded = shared dependency (DB, LB, network). One instance degraded = local resource (CPU, disk, GC, bad node). Errors up = code change, bad input, failing dependency. Latency up, CPU flat = saturation or network. Latency up, CPU up = computation (query plan, GC, regression).