A Percentile Is Not A Diagnosis
A score ranks an entity against a population. The high end reads as a finding, the low end as an all-clear, and both readings skip the measurement that would actually decide.
Scored reports invite two opposite errors. A high percentile gets treated as a result when the score was never validated for that decision. A low one gets treated as permission to skip a cheap check the score does not cover. The useful output of a scored report is not its conclusions but its gaps: in one real case, 5 of 8 tracked numbers either did not exist or were four years old.
Pattern
A model scores an entity against a population and returns a percentile. The high end gets read as a finding and acted on. The low end gets read as an all-clear and used to skip work.
Lead scores, supplier risk scores, churn propensity, credit-like ratings, an LLM judge's 1-to-10, polygenic risk scores in a consumer health report — same shape, same two errors.
Why it looks right
The number is precise, ordered and personal. It arrives inside a document that looks like a conclusion, next to other numbers that are genuinely measured. Percentiles also carry an unearned air of rigour: ranking against a population sounds like evidence, because it usually took a population to produce it.
Why it fails
The score was validated for a different question than the one being asked. In the health case, the relevant cardiology guidance rates routine polygenic scores as not recommended for reclassifying an individual's clinical risk, and consumer arrays reproduce poorly between providers. The score is real; it is simply not admissible for the decision it is being used for.
The protective direction is the more dangerous one. A favourable percentile only covers the mechanism the model represents. Sporadic and environmental causes sit outside it entirely — so a reassuring score becomes a licence to delay a cheap standard check, which is the one action the score cannot support.
An agent amplifies the asymmetry by default. Alarming scores are interesting and get surfaced. Reassuring ones are boring and pass through silently, taking their caveat with them.
Instead
Treat a percentile as routing, not as a result. It tells you which cheap hard measurement to go and take. It does not stand in for that measurement, and it never overrides one that already exists.
Read the report for its gaps, not its conclusions. The most useful line in one scored report was not a score at all:
5 of the 8 numbers being tracked either did not exist or were four years old
That is the actual deliverable of a scored report — a list of measurements nobody has taken. Everything else in it is a prompt to go and take them.
Attach the caveat to the reassuring half too. If an agent surfaces a high score with its limits stated, it must surface a low score the same way. A caveat applied only to bad news is not a caveat, it is a tone.
Check which population produced the number. Related trap from the same review: a cost table turned out to be a trade tariff rather than a retail price list, understating one line by 46%. Precise numbers from the wrong list survive review exactly as well as correct ones.
See also
the-agent-that-runs-a-body · one-source-is-a-hypothesis · mixing-the-scales · the-margin-from-the-wrong-cost
$ head -12 a-percentile-is-not-a-diagnosis.md
$ cite a-percentile-is-not-a-diagnosis
Citation id SV-4527 is stable. It resolves at
https://stillvalid.dev/c/SV-4527 even if this artifact moves to another section,
which a bare URL does not survive. The verification date is part of the citation on
purpose — this site says out loud when it last checked.
[A Percentile Is Not A Diagnosis](https://stillvalid.dev/patterns/a-percentile-is-not-a-diagnosis) — stillvalid, SV-4527 (anti-pattern, verified 2026-08-25)