Topic
evals
7 posts
- Retrieval10 min
The index found 39 percent, the agent found 96, and the useful question was neither
Agentic search vs an index on the same 23 pairs: the index found 39 percent and the agent 96, and per repository the trade ranged from five hits for twenty seconds to nothing at all.
- Evals5 min
Recall was exactly 1.0000 four times, and that was the measurement telling me
Ground truth leakage: NDCG moved from 0.68 to 0.98 while recall sat at 1.0000 in all four cells, because the answer key had been built by applying rules to the search results.
- Evals5 min
Five ways to measure the wrong thing, and each one was inside the fix for the last
Evaluation methodology failures, five in a row: a timer that decided nothing, an unmeasurable total, a price that was mostly thinking, a per-stage rate, and a feature returning nothing.
- Evals4 min
My benchmark had one slot left, so every change measured zero
Recall at k near the ceiling: the same retrieval change scored nothing on 29 queries and +4 on 90. The first set was at 28 of 29, so no change could have shown more.
- Evals6 min
My router scored 100% and a fixed answer scored 40%
Benchmark scoring with expected ⊆ answered: nothing was ever missed, so the score was perfect, and answering three fixed names without reading anything scored 40% on the same rule.
- Evals6 min
Three ways I broke one benchmark, and who caught each one
Benchmark methodology: blocked runs said one config was 81ms faster, alternating pairs reversed the sign, and the sort that computed the summary had destroyed the pairing.
- Evals7 min
I fixed the code three times and never checked the symptom
Regression testing that checked the diff, not the output: a flag parsed, documented and silently ignored in JSON. Two rounds of fixes looked correct and changed nothing.