Five ways a number can mean nothing
How benchmarks lie
Every post here is the same shape: a measurement that looked fine, a system that looked like it was working, and a reason the number could not have told the difference. A failing test announces itself. A meaningless test does not — that asymmetry is why these took weeks to find.
13 posts
Where this stands
- Held to be true
- A benchmark fails silently far more often than it fails loudly, and the failure is almost never in the arithmetic. Across thirteen cases the number was correctly computed every time: what was wrong was the answer key, the population, the headroom, the baseline, or which artifact was measured. The cheapest defence found so far is a second number that must agree.
- What would break it
- A case where the misleading result would have been caught by re-running or by more data. So far every one needed a different measurement, not a bigger one.
- Still open
- Whether these have anything in common besides being hard to see. They are collected as a catalogue because no shorter rule has covered them.
- 1Evals9 min
The baseline that beat my agent was answering the same thing every time
LLM agent evaluation without a null baseline: routing accuracy went from 73% to 90%, and a strategy that ignores the input scored 95% on the same test.
- 2Evals8 min
My agent benchmark was reading the answer off disk
LLM benchmark leakage in an agent eval: the bot produced a 239-line file, the human version on disk was 239 lines, and the diff was empty, filename included.
- 3Evals10 min
Ground truth from git history answers a different question than you asked
Ground truth evaluation from closed tickets gives you a free answer key. It records what someone once changed, not what needs changing now, and those diverge in specific ways.
- 4Evals9 min
Flaky and consistently wrong are different bugs with the same score
LLM evaluation: flaky and consistently wrong both show up as reduced accuracy. One responds to prompt changes and the other does not, and telling them apart takes one extra run.
- 5Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- 6Evals7 min
A tie at four hundred queries was a loss at fifteen hundred
Statistical significance and sample size in search tuning: four hundred queries said one parameter made no difference, and fifteen hundred turned that tie into seven lost answers.
- 7Evals5 min
The span that answered a question it never measured
OpenTelemetry LLM spans on every tool call told me the agent never ran anything in parallel, and I withdrew it. The numbers were not wrong. I had written them myself.
- 8Evals7 min
I fixed the code three times and never checked the symptom
Regression testing that checked the diff, not the output: a flag parsed, documented and silently ignored in JSON. Two rounds of fixes looked correct and changed nothing.
- 9Evals6 min
Three ways I broke one benchmark, and who caught each one
Benchmark methodology: blocked runs said one config was 81ms faster, alternating pairs reversed the sign, and the sort that computed the summary had destroyed the pairing.
- 10Evals6 min
My router scored 100% and a fixed answer scored 40%
Benchmark scoring with expected ⊆ answered: nothing was ever missed, so the score was perfect, and answering three fixed names without reading anything scored 40% on the same rule.
- 11Evals4 min
My benchmark had one slot left, so every change measured zero
Recall at k near the ceiling: the same retrieval change scored nothing on 29 queries and +4 on 90. The first set was at 28 of 29, so no change could have shown more.
- 12Evals5 min
Five ways to measure the wrong thing, and each one was inside the fix for the last
Evaluation methodology failures, five in a row: a timer that decided nothing, an unmeasurable total, a price that was mostly thinking, a per-stage rate, and a feature returning nothing.
- 13Evals5 min
Recall was exactly 1.0000 four times, and that was the measurement telling me
Ground truth leakage: NDCG moved from 0.68 to 0.98 while recall sat at 1.0000 in all four cells, because the answer key had been built by applying rules to the search results.