Measure first. Then decide.
Most of what I believed about my own systems did not survive measurement.
Notes on retrieval, evaluation, and the systems around them — measured before written. Every number here comes from a run against the system that actually serves traffic — not a copy made for the benchmark. Where a result rests on one run, it says so.
- Posts
- 5
- Words
- 11,303
- Sections
- 4
Next: An 80-day agent session and the wall at the end of it · tomorrow
5 in the last 42 days · 31 scheduled
Retrieval
3Code search, embeddings, and ranking.
Inference
1Local inference, Metal/GPU, and batch tuning.
Evals
1Measurement methodology and benchmark design.
Agents
Coding agents and context management.
Recent
- Inference6 min
I measured Metal wrong three times before it beat the CPU
The GPU looked slower than the CPU in every early run. All three of those runs were measuring something I had set up badly.
- Retrieval24 min
I could not measure the threshold where it mattered, until somebody could
A 0.85 cutoff stopped being a test when the model changed, and fired 270 times out of 270. Three measurements at three prices gave three different answers about whether that mattered, and the variable was which one anyone could afford.
- Evals9 min
The baseline that beat my agent was answering the same thing every time
I improved routing accuracy from 73% to 90%. Then I checked what a strategy that ignores the input scores on the same test. It scored 95%.
- Retrieval9 min
Your reranker speaks English. Your tickets do not.
Turning on the default reranker made search worse. The model name had a two-letter suffix I had never looked at.
- Retrieval11 min
Reranking cannot find what retrieval missed
I swapped four rerankers and watched Pass@1 climb while recall sat at exactly 47.6% every single time.