Trust the Report, Doubt the Data

Here is the rule that matters most when it counts: when the data and the report disagree, trust the report and doubt the data.

The “report” is the ground truth (what actually happened, told by the person who was there): a doctor who says patients kept dropping, a user who says the page was blank, an operator who says orders stalled. The “data” is the dashboard. When the two conflict, the instinct is to defend the dashboard and explain away the report. That instinct is usually backwards. Reality is not wrong. The measurement of it often is.

Third of three. Reading Data is how to read a number correctly, How Data Misleads is the catalogue of ways it goes wrong, and this page is what to do when the number and reality disagree.

Why the data is the usual culprit#

A report of something real is hard to fake: someone experienced it. A metric, by contrast, is a long chain of assumptions: an event fired (or didn’t), was attributed correctly, landed in the right bucket, got filtered the way you think, and measures the thing you believe it measures. Every link can break silently, and the dashboard looks just as confident when it’s wrong.

So when the ground truth says one thing and the number says another, the most likely explanations are, in order:

A missing data point: the metric isn’t capturing the case the report is about (the failures that never logged, the segment excluded by a filter).
Mis-instrumentation: the event, attribution, or bucketing is wrong (see instrument error in How Data Misleads).
A proxy gap: the metric measures something adjacent to what you care about, and they’ve drifted apart.
…and only then, that the report is mistaken.

Defending the dashboard against a credible report is how teams talk themselves out of real problems. Don’t dismiss the ground truth to protect the number. Reconcile them.

Prove better#

Doubting the data is not the end; it’s the start of proving it properly. When a report and a metric conflict, the work is to reconcile them:

Find the missing dimension. Ask what data point would make both true at once: the segment, the time window, the failure mode the dashboard doesn’t split by. Usually the reconciliation is a cut you weren’t making.
Instrument the gap. If the case the report describes isn’t captured, add the event or field so it is. You cannot manage what you never measured; the report just told you where to look.
Triangulate. Bring a second, independent measure. If two unrelated methods agree, trust rises; if they diverge, you’ve found the broken link.
Go and look. Read the logs, watch a session replay, talk to the person who filed the report. The raw trace beats the aggregate when the aggregate is in question.

The output is not “the report was wrong” or “the data was wrong”; it’s a better measurement that both the number and the ground truth now agree with. That measurement is what you carry forward.

Never ship a metric alone#

A single metric, chased alone, always breaks something quietly. Optimise it hard enough and you win the number while losing the goal, which is the Goodhart trap.

The guard is to pair every metric with one that would move if you were gaming it. That pairing is set when the metric is chosen rather than when the report looks wrong, so it lives with goal-setting: see guardrail metrics, which carries the common pairs. The rule to hold here is simply that a metric arriving without its pair is not ready to be trusted, however clean the dashboard looks.

The posture#

The dashboard serves the decision, not the other way round. Numbers are there to sharpen judgement, not replace it. When they sharpen it, lean in. When they contradict a credible account of reality, believe reality, fix the measurement, and come back with data you can trust.