Topic
retrieval
20 posts
- Agents7 min
I told the agent to search its memory. It made 31 tool calls and searched zero times
LLM agent memory search: a 24-run experiment said telling the agent to search was as good as injecting notes. It measured irrelevant notes against nothing, and nobody searched.
- Retrieval17 min
SQLite FTS5 replaced my vector search, and the engine was the smaller finding
BM25 vs embeddings: a plain SQLite FTS5 table matched or beat my hybrid vector engine on four gold sets, at 28 ms instead of 750. The bigger finding was 170 files nobody indexed.
- Retrieval10 min
The index found 39 percent, the agent found 96, and the useful question was neither
Agentic search vs an index on the same 23 pairs: the index found 39 percent and the agent 96, and per repository the trade ranged from five hits for twenty seconds to nothing at all.
- Retrieval4 min
The error said no such column, and the column was a fragment of a user's query
SQLite FTS5 "no such column" from a pasted HTML tag: parameter binding stopped SQL injection but not the query language underneath, which parsed the bound value a second time.
- Evals5 min
Recall was exactly 1.0000 four times, and that was the measurement telling me
Ground truth leakage: NDCG moved from 0.68 to 0.98 while recall sat at 1.0000 in all four cells, because the answer key had been built by applying rules to the search results.
- Retrieval4 min
The scores spread out 15 times wider and the ranking did not move
Embedding anisotropy: vectors sit in a narrow cone, so unrelated documents score 0.90 against anything. Centering fixed that exactly as advertised, and rank 1 went down by two.
- Retrieval6 min
Stripping one suffix fixed a quarter of my dead queries
Query normalization for Korean search: a place name plus a station suffix matches nothing. Retrying without the suffix cut zero results from 29.6% to 5.4%.
- Retrieval10 min
Same bytes, a different folder, a different score
Embedding model evaluation that I made worse by being careful: indexing identical files under a different path returns different vectors.
- Retrieval8 min
The one case it fixed and the ones it broke
A BM25 ranking fix from a correct diagnosis of a real bug: that ticket moved from ninth to first, and eighteen other tickets lost first place.
- Retrieval10 min
A perfect title match had to be popular to count
Hybrid search ranking where retrieval could contribute at most 0.2030 toward a 0.25 cutoff. An exact title match survived only when the product was also popular enough.
- Evals6 min
Eighty-three percent of my zero-result searches had nothing to find
Zero-result searches, decomposed: only a small slice was a search defect. Most of the volume was asking for products that do not exist.
- Retrieval14 min
The biggest repository wins, and it has nothing to do with relevance
Retrieval bias toward big repositories: every repo scored between 0.924 and 0.943 for every query, and the 0.019 that separated them was measuring size.
- Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- Retrieval10 min
Repository routing is the retrieval problem nobody benchmarks
Code search across many repositories: every published issue-localisation benchmark assumes you already know which repository to look in. In a multi-repo organisation that is the hard part.
- Retrieval11 min
Fusing two rankers made routing worse, and the reason is structural
Reciprocal rank fusion made routing worse: one signal covered every repository, the other 17 of 28, and RRF punished exactly the repositories the second could not see.
- Retrieval8 min
Picking an embedding model when the queries and the code are in different languages
Best embedding model for code when queries and code are in different languages: two of five models scored below 4% on Pass@1, where random guessing gets 2%.
- Retrieval13 min
Three percent of my index was eating the top five results
Semantic code search crowded out by docs: 89 documentation files sank one repository, and removing them more than doubled MRR. The same fix on five other repositories moved nothing.
- Retrieval24 min
I could not measure the threshold where it mattered, until somebody could
Similarity threshold in RAG: a 0.85 cutoff fired 270 times out of 270 once the model changed. Three measurements at three prices gave three answers about whether it mattered.
- Retrieval9 min
Your reranker speaks English. Your tickets do not.
A multilingual reranker problem: turning on the default reranker made search worse, and the model name had a two-letter suffix I had never looked at.
- Retrieval11 min
Reranking cannot find what retrieval missed
RAG reranking cannot recover missed recall: I swapped four rerankers and watched Pass@1 climb while recall sat at exactly 47.6% every time.