July 31, 2026
Mean Girls: Dashboard Edition
The mean means nothing: data visualization to debug a latency problem
The dashboard said disaster, but the comments said the real problem was trusting one number
TLDR: A speed tweak looked like a failure if you only watched the average, but a fuller view showed most requests got faster while the slowest ones got much worse. Commenters turned it into a mini-drama about bad dashboards, overconfident averages, and whether the author should have known this lesson already.
A routine speed-up test turned into a full-on stats scandal when one company rolled out a new cache and the main dashboard screamed bad news: average wait time got worse. Cue the imaginary panic, rollback, and blame game. But then the plot twisted — the middle request time actually got way better, while the worst-case slow requests became much uglier. In plain English: most people were faster, some people were way slower, and one lonely average number was out here causing chaos.
That’s where the comment section absolutely lit up. One reader delivered the instant-classic drive-by, “With only having read the headline… Aggregation without distribution always means nothing,” basically admitting they didn’t read the article and still felt spiritually correct. Another crowd favorite was the slightly smug, very internet reaction: why was someone doing performance work without already comparing the full spread of results? Others piled on with nerdy receipts, name-dropping the famous Anscombe Quartet, a classic example showing that identical-looking stats can hide wildly different realities.
And then came the visualization wars. Some backed the article’s hero chart, the cumulative distribution function — basically a graph showing how many requests finish by a given time. Others said, not so fast: the real move is flipping it around and zooming in on the slow tail, because that’s where user pain lives. The mood was equal parts “great lesson”, “how did you not know this already?”, and “welcome to the cursed world where every metric tells a different story.”
Key Points
- •The article describes how production latency dashboards based on aggregate metrics failed to clearly show performance improvements the author saw in benchmarks.
- •A colleague’s use of a cumulative distribution function (CDF) for build-speed analysis prompted the author to explore broader data-visualization methods.
- •The article uses a reproducible synthetic dataset and script to compare how different visualizations represent the same latency data.
- •In the rollout example, mean latency worsens from 112 ms to 122 ms, while median latency improves from 99 ms to 54 ms and tail latencies p95 and p99 worsen sharply.
- •The article concludes that examining distribution shape and CDFs provides more complete insight than relying on a single summary statistic such as the mean.