Topic
evaluation
16 posts
- Agents7 min
I told the agent to search its memory. It made 31 tool calls and searched zero times
LLM agent memory search: a 24-run experiment said telling the agent to search was as good as injecting notes. It measured irrelevant notes against nothing, and nobody searched.
- Evals7 min
44 error alerts, 19 automatic fixes, and no gate on how big a fix could be
AI code review on auto-fixed alerts: the alert-to-deploy path had gates for relevance, kind, review, QA and cost, and none for size. What 19 fixes looked like, and why file count lies.
- Evals9 min
Low effort cut agent turns by two thirds. Accuracy, I could not tell
Claude effort level on 40 tickets, three arms: low effort used fewer turns on 21 of 21 paired tickets, and on accuracy the twin arms disagreed with each other as much as with it.
- Retrieval17 min
SQLite FTS5 replaced my vector search, and the engine was the smaller finding
BM25 vs embeddings: a plain SQLite FTS5 table matched or beat my hybrid vector engine on four gold sets, at 28 ms instead of 750. The bigger finding was 170 files nobody indexed.
- Evals7 min
A tie at four hundred queries was a loss at fifteen hundred
Statistical significance and sample size in search tuning: four hundred queries said one parameter made no difference, and fifteen hundred turned that tie into seven lost answers.
- Retrieval10 min
Same bytes, a different folder, a different score
Embedding model evaluation that I made worse by being careful: indexing identical files under a different path returns different vectors.
- Retrieval8 min
The one case it fixed and the ones it broke
A BM25 ranking fix from a correct diagnosis of a real bug: that ticket moved from ninth to first, and eighteen other tickets lost first place.
- Evals5 min
The span that answered a question it never measured
OpenTelemetry LLM spans on every tool call told me the agent never ran anything in parallel, and I withdrew it. The numbers were not wrong. I had written them myself.
- Evals6 min
Eighty-three percent of my zero-result searches had nothing to find
Zero-result searches, decomposed: only a small slice was a search defect. Most of the volume was asking for products that do not exist.
- Retrieval14 min
The biggest repository wins, and it has nothing to do with relevance
Retrieval bias toward big repositories: every repo scored between 0.924 and 0.943 for every query, and the 0.019 that separated them was measuring size.
- Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- Evals9 min
Flaky and consistently wrong are different bugs with the same score
LLM evaluation: flaky and consistently wrong both show up as reduced accuracy. One responds to prompt changes and the other does not, and telling them apart takes one extra run.
- Evals10 min
Ground truth from git history answers a different question than you asked
Ground truth evaluation from closed tickets gives you a free answer key. It records what someone once changed, not what needs changing now, and those diverge in specific ways.
- Evals8 min
My agent benchmark was reading the answer off disk
LLM benchmark leakage in an agent eval: the bot produced a 239-line file, the human version on disk was 239 lines, and the diff was empty, filename included.
- Evals9 min
The baseline that beat my agent was answering the same thing every time
LLM agent evaluation without a null baseline: routing accuracy went from 73% to 90%, and a strategy that ignores the input scored 95% on the same test.
- Retrieval11 min
Reranking cannot find what retrieval missed
RAG reranking cannot recover missed recall: I swapped four rerankers and watched Pass@1 climb while recall sat at exactly 47.6% every time.