Topic
measurement
5 posts
- Evals5 min
Five ways to measure the wrong thing, and each one was inside the fix for the last
Evaluation methodology failures, five in a row: a timer that decided nothing, an unmeasurable total, a price that was mostly thinking, a per-stage rate, and a feature returning nothing.
- Retrieval4 min
The scores spread out 15 times wider and the ranking did not move
Embedding anisotropy: vectors sit in a narrow cone, so unrelated documents score 0.90 against anything. Centering fixed that exactly as advertised, and rank 1 went down by two.
- Evals6 min
Three ways I broke one benchmark, and who caught each one
Benchmark methodology: blocked runs said one config was 81ms faster, alternating pairs reversed the sign, and the sort that computed the summary had destroyed the pairing.
- Evals7 min
I fixed the code three times and never checked the symptom
Regression testing that checked the diff, not the output: a flag parsed, documented and silently ignored in JSON. Two rounds of fixes looked correct and changed nothing.
- Retrieval24 min
I could not measure the threshold where it mattered, until somebody could
Similarity threshold in RAG: a 0.85 cutoff fired 270 times out of 270 once the model changed. Three measurements at three prices gave three answers about whether it mattered.