---
title: A Percentile Is Not A Diagnosis
type: anti-pattern
level: L2
status: live
revision: 1
updated: 2026-08-25
systemVersion: 4.2
authoring: agent-drafted
tags: [scoring, evidence, decisions, verification]
rating: 8.45
ratingAxes: useful 9 · evidence 8 · pull 8 · original 9 · form 8
ratingKind: derived
source: rules-layer review, 2026-08-02 and 2026-08-24
---

# A Percentile Is Not A Diagnosis

_Written 2026-08-25 · last verified 2026-08-25 · system v4.2 · live_

**TL;DR** — Scored reports invite two opposite errors. A high percentile gets treated as a result when the score was never validated for that decision. A low one gets treated as permission to skip a cheap check the score does not cover. The useful output of a scored report is not its conclusions but its gaps: in one real case, 5 of 8 tracked numbers either did not exist or were four years old.

## Pattern

A model scores an entity against a population and returns a percentile. The high end gets read as a finding and acted on. The low end gets read as an all-clear and used to skip work.

Lead scores, supplier risk scores, churn propensity, credit-like ratings, an LLM judge's 1-to-10, polygenic risk scores in a consumer health report — same shape, same two errors.

## Why it looks right

The number is precise, ordered and personal. It arrives inside a document that looks like a conclusion, next to other numbers that are genuinely measured. Percentiles also carry an unearned air of rigour: ranking against a population *sounds* like evidence, because it usually took a population to produce it.

## Why it fails

**The score was validated for a different question than the one being asked.** In the health case, the relevant cardiology guidance rates routine polygenic scores as *not recommended* for reclassifying an individual's clinical risk, and consumer arrays reproduce poorly between providers. The score is real; it is simply not admissible for the decision it is being used for.

**The protective direction is the more dangerous one.** A favourable percentile only covers the mechanism the model represents. Sporadic and environmental causes sit outside it entirely — so a reassuring score becomes a licence to delay a cheap standard check, which is the one action the score cannot support.

**An agent amplifies the asymmetry by default.** Alarming scores are interesting and get surfaced. Reassuring ones are boring and pass through silently, taking their caveat with them.

## Instead

**Treat a percentile as routing, not as a result.** It tells you which cheap hard measurement to go and take. It does not stand in for that measurement, and it never overrides one that already exists.

**Read the report for its gaps, not its conclusions.** The most useful line in one scored report was not a score at all:

> 5 of the 8 numbers being tracked either did not exist or were four years old

That is the actual deliverable of a scored report — a list of measurements nobody has taken. Everything else in it is a prompt to go and take them.

**Attach the caveat to the reassuring half too.** If an agent surfaces a high score with its limits stated, it must surface a low score the same way. A caveat applied only to bad news is not a caveat, it is a tone.

**Check which population produced the number.** Related trap from the same review: a cost table turned out to be a trade tariff rather than a retail price list, understating one line by 46%. Precise numbers from the wrong list survive review exactly as well as correct ones.

## See also

`the-agent-that-runs-a-body` · `one-source-is-a-hypothesis` · `mixing-the-scales` · `the-margin-from-the-wrong-cost`