Topic
benchmarks
6 posts
- Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- Retrieval10 min
Repository routing is the retrieval problem nobody benchmarks
Code search across many repositories: every published issue-localisation benchmark assumes you already know which repository to look in. In a multi-repo organisation that is the hard part.
- Evals10 min
Ground truth from git history answers a different question than you asked
Ground truth evaluation from closed tickets gives you a free answer key. It records what someone once changed, not what needs changing now, and those diverge in specific ways.
- Evals8 min
My agent benchmark was reading the answer off disk
LLM benchmark leakage in an agent eval: the bot produced a 239-line file, the human version on disk was 239 lines, and the diff was empty, filename included.
- Inference6 min
I measured Metal wrong three times before it beat the CPU
Apple Silicon GPU benchmarks for embeddings: Metal looked slower than the CPU in every early run, and all three runs were measuring something I had set up badly.
- Evals9 min
The baseline that beat my agent was answering the same thing every time
LLM agent evaluation without a null baseline: routing accuracy went from 73% to 90%, and a strategy that ignores the input scored 95% on the same test.