All posts
50 published
- Agents7 min
The child process inherited nothing, so the test now walks every spawn site
Child process environment variables: config moved to a table loaded by each entry point's first import, and three scripts the bot spawns were not entry points. A test now walks spawn sites.
- Agents6 min
Four traps when merging parallel agent worktrees into one trunk
Git worktree for parallel agents: sessions edit one repo in separate worktrees and a person merges by cherry-pick. The traps are the tests, the classifier, patch-ids and the leftovers.
- Agents5 min
My LLM tracing store was 138 GB. The traces were 3.2 MB of it
ClickHouse disk usage under self-hosted Langfuse: 117 GB of it profiling its own queries. The fix sat unapplied for 18 days, and the table was too big to drop.
- Agents6 min
A restart sent 30 direct messages in 8 seconds. Then --help sent the daily report three times
Telegram bot duplicate messages, twice in three days: code that reads the live database and talks to a person is only as safe as the rows it finds, and DM is not a fallback.
- Agents6 min
The approval queue said nine fixes were waiting. Eight were already in production
Human in the loop agent approvals: the bot's pending list was its own table while people merged the same branches by hand. The fix was a check at the moment of speaking, not a sync job.
- Agents7 min
I told the agent to search its memory. It made 31 tool calls and searched zero times
LLM agent memory search: a 24-run experiment said telling the agent to search was as good as injecting notes. It measured irrelevant notes against nothing, and nobody searched.
- Evals7 min
44 error alerts, 19 automatic fixes, and no gate on how big a fix could be
AI code review on auto-fixed alerts: the alert-to-deploy path had gates for relevance, kind, review, QA and cost, and none for size. What 19 fixes looked like, and why file count lies.
- Evals9 min
Low effort cut agent turns by two thirds. Accuracy, I could not tell
Claude effort level on 40 tickets, three arms: low effort used fewer turns on 21 of 21 paired tickets, and on accuracy the twin arms disagreed with each other as much as with it.
- Retrieval17 min
SQLite FTS5 replaced my vector search, and the engine was the smaller finding
BM25 vs embeddings: a plain SQLite FTS5 table matched or beat my hybrid vector engine on four gold sets, at 28 ms instead of 750. The bigger finding was 170 files nobody indexed.
- Retrieval10 min
The index found 39 percent, the agent found 96, and the useful question was neither
Agentic search vs an index on the same 23 pairs: the index found 39 percent and the agent 96, and per repository the trade ranged from five hits for twenty seconds to nothing at all.
- Agents4 min
The numbers halved and the fix never ran
Deployment verification the hard way: wall clock down 53 percent, tool calls down 47, and the line the fix exists to produce absent in both runs. The config was a symlink shared by every worktree.
- Retrieval4 min
The error said no such column, and the column was a fragment of a user's query
SQLite FTS5 "no such column" from a pasted HTML tag: parameter binding stopped SQL injection but not the query language underneath, which parsed the bound value a second time.
- Evals5 min
Recall was exactly 1.0000 four times, and that was the measurement telling me
Ground truth leakage: NDCG moved from 0.68 to 0.98 while recall sat at 1.0000 in all four cells, because the answer key had been built by applying rules to the search results.
- Evals5 min
Five ways to measure the wrong thing, and each one was inside the fix for the last
Evaluation methodology failures, five in a row: a timer that decided nothing, an unmeasurable total, a price that was mostly thinking, a per-stage rate, and a feature returning nothing.
- Retrieval4 min
The scores spread out 15 times wider and the ranking did not move
Embedding anisotropy: vectors sit in a narrow cone, so unrelated documents score 0.90 against anything. Centering fixed that exactly as advertised, and rank 1 went down by two.
- Evals4 min
My benchmark had one slot left, so every change measured zero
Recall at k near the ceiling: the same retrieval change scored nothing on 29 queries and +4 on 90. The first set was at 28 of 29, so no change could have shown more.
- Evals6 min
My router scored 100% and a fixed answer scored 40%
Benchmark scoring with expected ⊆ answered: nothing was ever missed, so the score was perfect, and answering three fixed names without reading anything scored 40% on the same rule.
- Evals6 min
Three ways I broke one benchmark, and who caught each one
Benchmark methodology: blocked runs said one config was 81ms faster, alternating pairs reversed the sign, and the sort that computed the summary had destroyed the pairing.
- Evals7 min
A tie at four hundred queries was a loss at fifteen hundred
Statistical significance and sample size in search tuning: four hundred queries said one parameter made no difference, and fifteen hundred turned that tie into seven lost answers.
- Retrieval6 min
Stripping one suffix fixed a quarter of my dead queries
Query normalization for Korean search: a place name plus a station suffix matches nothing. Retrying without the suffix cut zero results from 29.6% to 5.4%.
- Evals7 min
I fixed the code three times and never checked the symptom
Regression testing that checked the diff, not the output: a flag parsed, documented and silently ignored in JSON. Two rounds of fixes looked correct and changed nothing.
- Retrieval10 min
Same bytes, a different folder, a different score
Embedding model evaluation that I made worse by being careful: indexing identical files under a different path returns different vectors.
- Retrieval8 min
The one case it fixed and the ones it broke
A BM25 ranking fix from a correct diagnosis of a real bug: that ticket moved from ninth to first, and eighteen other tickets lost first place.
- Inference7 min
The flag was set and the feature was not there
GPU not being used, silently: the variable that turns on GPU embedding was set, but the binary was built without GPU support, so it ran on CPU. 306 seconds, not 51.
- Agents6 min
The median barely moved and the spread collapsed
Claude Code speed on code review: one line naming the search command. The median barely moved, the worst case fell from 18.0 to 12.2 minutes, and tool calls did not change.
- Evals5 min
The span that answered a question it never measured
OpenTelemetry LLM spans on every tool call told me the agent never ran anything in parallel, and I withdrew it. The numbers were not wrong. I had written them myself.
- Retrieval10 min
A perfect title match had to be popular to count
Hybrid search ranking where retrieval could contribute at most 0.2030 toward a 0.25 cutoff. An exact title match survived only when the product was also popular enough.
- Evals6 min
Eighty-three percent of my zero-result searches had nothing to find
Zero-result searches, decomposed: only a small slice was a search defect. Most of the volume was asking for products that do not exist.
- Retrieval14 min
The biggest repository wins, and it has nothing to do with relevance
Retrieval bias toward big repositories: every repo scored between 0.924 and 0.943 for every query, and the 0.019 that separated them was measuring size.
- Agents9 min
LangGraph yes, LangChain no
LangGraph vs LangChain in an agent pipeline: one earns its place by doing something I would otherwise write badly. The other is a spec I satisfy without importing.
- Agents8 min
Building the guardrails I needed after a session died
Claude Code hooks and a status line for context usage and compaction. Both read the transcript rather than an API field, and that choice is the post.
- Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- Inference10 min
What running embeddings locally actually costs
Local embeddings at scale: 672,000 chunks across 28 repositories, indexed on a laptop. The bill was not money and not really time either.
- Retrieval10 min
Repository routing is the retrieval problem nobody benchmarks
Code search across many repositories: every published issue-localisation benchmark assumes you already know which repository to look in. In a multi-repo organisation that is the hard part.
- Evals9 min
Flaky and consistently wrong are different bugs with the same score
LLM evaluation: flaky and consistently wrong both show up as reduced accuracy. One responds to prompt changes and the other does not, and telling them apart takes one extra run.
- Agents8 min
I instrumented everything except where the decisions happen
LLM agent observability with a connected, green tracing dashboard, and zero traces from the three components that actually make choices.
- Evals10 min
Ground truth from git history answers a different question than you asked
Ground truth evaluation from closed tickets gives you a free answer key. It records what someone once changed, not what needs changing now, and those diverge in specific ways.
- Inference9 min
Bigger batches made my GPU slower
Embedding batch size on a GPU: batch 4 beat batch 256 by 1.9x on the same device with the same weights. A third of the larger batch was computing padding.
- Retrieval11 min
Fusing two rankers made routing worse, and the reason is structural
Reciprocal rank fusion made routing worse: one signal covered every repository, the other 17 of 28, and RRF punished exactly the repositories the second could not see.
- Agents10 min
Context is burned by tool output, not by conversation
Claude Code context usage over a two-month transcript: the largest entries were single command results, around 300,000 tokens each.
- Inference8 min
The GPU was not slow, it was waiting
A tokenizer bottleneck made CoreML indexing three times slower. Utilisation said the accelerator was idle, and the cause was one padding setting.
- Retrieval8 min
Picking an embedding model when the queries and the code are in different languages
Best embedding model for code when queries and code are in different languages: two of five models scored below 4% on Pass@1, where random guessing gets 2%.
- Retrieval13 min
Three percent of my index was eating the top five results
Semantic code search crowded out by docs: 89 documentation files sank one repository, and removing them more than doubled MRR. The same fix on five other repositories moved nothing.
- Evals8 min
My agent benchmark was reading the answer off disk
LLM benchmark leakage in an agent eval: the bot produced a 239-line file, the human version on disk was 239 lines, and the diff was empty, filename included.
- Agents9 min
An 80-day agent session and the wall at the end of it
Claude Code context limit on an 80-day session: fifty compactions, each starting from a full million tokens. Then one morning it stopped answering for nineteen minutes.
- Inference6 min
I measured Metal wrong three times before it beat the CPU
Apple Silicon GPU benchmarks for embeddings: Metal looked slower than the CPU in every early run, and all three runs were measuring something I had set up badly.
- Retrieval24 min
I could not measure the threshold where it mattered, until somebody could
Similarity threshold in RAG: a 0.85 cutoff fired 270 times out of 270 once the model changed. Three measurements at three prices gave three answers about whether it mattered.
- Evals9 min
The baseline that beat my agent was answering the same thing every time
LLM agent evaluation without a null baseline: routing accuracy went from 73% to 90%, and a strategy that ignores the input scored 95% on the same test.
- Retrieval9 min
Your reranker speaks English. Your tickets do not.
A multilingual reranker problem: turning on the default reranker made search worse, and the model name had a two-letter suffix I had never looked at.
- Retrieval11 min
Reranking cannot find what retrieval missed
RAG reranking cannot recover missed recall: I swapped four rerankers and watched Pass@1 climb while recall sat at exactly 47.6% every time.