Topic
methodology
7 posts
- Retrieval10 min
The index found 39 percent, the agent found 96, and the useful question was neither
Agentic search vs an index on the same 23 pairs: the index found 39 percent and the agent 96, and per repository the trade ranged from five hits for twenty seconds to nothing at all.
- Evals5 min
Five ways to measure the wrong thing, and each one was inside the fix for the last
Evaluation methodology failures, five in a row: a timer that decided nothing, an unmeasurable total, a price that was mostly thinking, a per-stage rate, and a feature returning nothing.
- Evals4 min
My benchmark had one slot left, so every change measured zero
Recall at k near the ceiling: the same retrieval change scored nothing on 29 queries and +4 on 90. The first set was at 28 of 29, so no change could have shown more.
- Evals6 min
Three ways I broke one benchmark, and who caught each one
Benchmark methodology: blocked runs said one config was 81ms faster, alternating pairs reversed the sign, and the sort that computed the summary had destroyed the pairing.
- Evals7 min
A tie at four hundred queries was a loss at fifteen hundred
Statistical significance and sample size in search tuning: four hundred queries said one parameter made no difference, and fifteen hundred turned that tie into seven lost answers.
- Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- Evals9 min
The baseline that beat my agent was answering the same thing every time
LLM agent evaluation without a null baseline: routing accuracy went from 73% to 90%, and a strategy that ignores the input scored 95% on the same test.