The lever nobody reaches for
What you let into the index
Model comparison is the visible knob — there are model cards to read and leaderboards to check, and it feels like progress. The largest measured improvement in this whole project came from deleting 3% of a corpus. These posts are about the decisions made before any model runs.
6 posts
Where this stands
- Held to be true
- An index is a decision about what is searchable, not a copy of a repository, and the decision is usually made by accident. What hurt in every case was text nobody would ever search for. But the size of that harm did not generalise: the same proportion of prose did different jobs in different repositories.
- What would break it
- A repository where prose crowds the candidate slots and removing it moves the hit rate. That was the original claim and it failed to replicate on five.
- Still open
- Whether there is a cheap test that separates prose that covers an answer from prose that merely occupies slots. Right now the only way to tell is to remove it and re-measure.
- 1Retrieval13 min
Three percent of my index was eating the top five results
Semantic code search crowded out by docs: 89 documentation files sank one repository, and removing them more than doubled MRR. The same fix on five other repositories moved nothing.
- 2Retrieval8 min
Picking an embedding model when the queries and the code are in different languages
Best embedding model for code when queries and code are in different languages: two of five models scored below 4% on Pass@1, where random guessing gets 2%.
- 3Retrieval10 min
Repository routing is the retrieval problem nobody benchmarks
Code search across many repositories: every published issue-localisation benchmark assumes you already know which repository to look in. In a multi-repo organisation that is the hard part.
- 4Evals10 min
Measure the index you actually serve
Retrieval evaluation on a benchmark copy drifts from production the moment anyone changes an ignore rule, and nothing tells you it has.
- 5Retrieval4 min
The scores spread out 15 times wider and the ranking did not move
Embedding anisotropy: vectors sit in a narrow cone, so unrelated documents score 0.90 against anything. Centering fixed that exactly as advertised, and rank 1 went down by two.
- 6Retrieval24 min
I could not measure the threshold where it mattered, until somebody could
Similarity threshold in RAG: a 0.85 cutoff fired 270 times out of 270 once the model changed. Three measurements at three prices gave three answers about whether it mattered.