I measured retrieval quality across three repositories that share a codebase lineage, expecting roughly similar numbers. Two came out where I expected. The third was less than half as good.
The bad one was the biggest and most actively developed of the three, which made it the least likely candidate for "the index is broken." I assumed it was harder to search: more files, more surface area, more ways for a query to be ambiguous.
It wasn't harder. It had 89 markdown files in it.
Looking at what actually came back
I took a query that should have been easy (an issue title about making a form field optional) and printed the top five results instead of just scoring them.
Five documentation files. Not one line of code. The same file twice.
The correct answers, a component and a diff utility, were in the index, verified by name. They were just below all of this.
And look at the scores: 0.941 through 0.944. Three thousandths of a point separating first place from fifth. The ranking was not choosing between these; it was returning whatever the noise floor happened to order first.
Why documentation wins
Those files describe screens in prose. Written by the team, in Korean, in the same vocabulary a person uses when they file a ticket.
Which is exactly what the query is.
An issue title says something like "make the request field optional on the product form." A documentation file says "the product form has a request field, which is currently required." Same words, same register, same language. The embedding model does what it was trained to do and puts them close together.
The actual fix lives in a component that mentions none of those words. It has an identifier, a prop name, a conditional. A model matching an issue title against a codebase is doing cross-lingual, cross-register retrieval (natural language on one side, code on the other) and prose in the index short-circuits that. It gives the model something easy to match, and easy matches crowd out the correct ones.
Documentation isn't noise. It's the most fluent competitor in the index, and the metric rewards fluency because that's what similarity measures.
What I chased first
Before I printed a result list I spent a day on two other explanations, and both were reasonable enough that I want to record them.
Corpus size. This repository is roughly four times the others by chunk count, so I assumed a bigger haystack. That is a real effect in general, but it predicts a gradual decline, and what I had was a cliff, less than half the MRR of a sibling built from the same code by the same team.
Chunking. The repository is TypeScript with large component files, so maybe chunks were landing badly and splitting functions in the middle. I looked. Chunk sizes were within a few percent of the sibling repositories, and spot-checking the boundaries showed nothing unusual.
Both of those investigations involved reading configuration and comparing statistics. Neither involved running a query and looking at the answer, which took ninety seconds and produced the entire explanation.
I think that's the actual lesson, more than anything about documentation. I had a system that returns ranked lists and I spent a day not looking at any of them. Metrics are a compression of the output, and I was debugging the compression.
The size of the thing
I expected the docs to be a substantial fraction of the corpus. They weren't:
Three percent of the index was taking the entire first page of results on most queries.
Removing them
One line in an ignore file:
Then a rebuild. This is where I lost twenty minutes, so it deserves a sentence. An incremental reindex does nothing:
Ignoring a file makes it undiscovered, not deleted. The indexer compares what it finds against what it has, and a file it no longer finds is simply absent from the comparison. Nothing tells it to remove the 89 entries already stored. You have to drop the index and rebuild.
| with docs | docs excluded | |
|---|---|---|
| Pass@1 | 9.5% | 26.2% |
| Pass@5 | 19.0% | 45.2% |
| Pass@10 | 35.7% | 47.6% |
| MRR | 0.142 | 0.333 |
Pass@1 nearly tripled. MRR more than doubled. Same model, same queries, same chunking, 89 files fewer. It is also the answer to the question the reranking post left open: recall does not move by reordering candidates, it moves by changing what is in the index.
And the result lands almost exactly where its sibling repositories were all along:
The repository was never worse at being searched. It had a competitor the others didn't.
Simulating before rebuilding
I checked this before doing the rebuild, and I'd recommend the same order. Rather than dropping the index on a hunch, run the queries as normal and filter documentation out of the results afterward:
That predicts what exclusion would give you, in the time it takes to run the queries once. My simulation said 26.2 / 45.2 / 47.6 / 0.333, and the actual rebuild produced 26.2 / 45.2 / 47.6 / 0.333, identical to three decimals.
Cheap, and it means you rebuild once knowing the answer instead of twice hoping.
Where the docs went instead
I didn't delete anything: the files are still in the repository, and people still read them in the editor. They're only excluded from the search index.
That distinction matters if you're weighing this change. Nobody lost documentation. What they lost was documentation appearing when they searched for the code that implements it, which nobody wanted in the first place.
If your team does search for prose, the answer isn't to keep it mixed in. It's to keep two indexes and let the interface decide which to query, because the failure mode here is not that documentation is bad. One ranking has to serve two questions that want opposite things. A search for "how does billing work" wants the prose. A search for "billing amount is wrong on the detail page" wants the component. A single index ranks both the same way and gets one of them wrong.
I had one index doing two jobs and hadn't noticed, because for months the only thing I looked at was a number.
What I check on a new repository now
Print the top five for a few queries, not just the score. A score tells you something is wrong. The result list tells you what. I had been scoring for weeks without once looking at what came back, and everything I needed was in that first printout.
Look for prose in a code index. Anything written in the language your queries are written in (design documents, screen descriptions, ADRs, translated README files) competes with code on the model's terms rather than yours. Ask whether someone searching for a bug would want to land there. Usually the answer is no.
Suspect clustered scores. When the top five are within a few thousandths of each other, ranking has stopped discriminating and you're reading noise. That's a signal in its own right, and it's visible without a gold set.
The uncomfortable general lesson: retrieval quality is set by what you let into the index as much as by the model you pick. I spent days comparing embedding models on this corpus. The largest single improvement I found came from deleting 3% of it.
The same shape, elsewhere
Once I knew what to look for I checked the other repositories in the group, and found the pattern twice more in different clothing.
One backend repository had test fixtures: recorded API responses saved as JSON, the largest a single 182 MB file. That one file produced 106,170 chunks, which was 48% of the repository's entire index. Four such files together were 86%. The repository was not a codebase with some fixtures in it; by index volume it was a pile of recorded HTTP responses with some code attached.
That one doesn't crowd results the way prose does (nobody's issue title resembles a serialised price object), but it wrecks everything else. Indexing that repository took over two and a half hours and did not finish. After excluding those four files it took sixteen minutes, and searches for provider integration code started returning provider integration code.
The other case was subtler: a repository where the framework's build output was not in .gitignore, so minified bundles were being indexed alongside source. Those don't compete for issue-title queries either, but they mean a fifth of your embedding budget goes to machine-generated text nobody will ever search for.
Three repositories, three different kinds of file, one question underneath all of them: is this text something a person would ever search for? Prose about screens, recorded API responses, minified bundles, all real files, all correctly in version control, none of them things anyone types a query hoping to find.
The index is not a copy of the repository. It's a decision about what is searchable, and I had been treating it as a copy.
Then it did not generalise
The paragraph above says "three repositories, one question underneath". That was a claim about generalisation, made from one measured case and two that looked similar, and it has since been tested.
Five other repositories showed the same silhouette: prose is 0.6 to 2.2 percent of their chunks and takes 20 to 58 percent of the six candidate slots. Better still, the three repositories that scored zero hits were exactly the three where prose held twenty percent of the slots or more. The correlation was not subtle.
So a prediction was written down before running anything: hits should go from 9 of 23 to about 12 of 23, and anything outside 10 to 14 means the model is wrong.
Removal happened in two steps so the effect of each was visible.
| step | rule | chunks removed | prose share of candidates | hits |
|---|---|---|---|---|
| before | 17% | 9/23 | ||
| folders | **/docs/** | 1,697 | 5% | 9/23 |
| file type | *.md | 750 | 0% | 9/23 |
Not one cell moved. Not the total, not any repository, not any individual pair. One front end went from half its candidate slots being prose to none of them, and stayed at zero hits out of two.
Twenty-four slots were handed back to code, and none of them contained an answer.
What came into the empty slots
That is a stronger statement than "the total did not change", because the window genuinely changed. Four pairs had candidate lists recorded both before and after, so it is possible to look at what actually arrived.
| pair | prose displaced | what took the slots |
|---|---|---|
| option popup styling | 4 | an order popup, a marketing nav bar, a test file, a terms page |
| room fare deduplication | 1 | a file named for exactly that topic, which was not the file the fix touched |
| payment discount error | 2 | two payment notice components, same screen, different concern |
| keeping an airline filter | 2 | a card-benefit filter hook and a hotel review filter, matching only on the word "filter" |
The replacements were not random. Same repository, usually the same application, sometimes the same screen, in one case matching on a single shared word. One was a test file. One was topically exact and still wrong, because the person who fixed the issue had edited something else.
Removing the prose did not reveal the answer hiding underneath. It let the search return six more plausible wrong things. The answer was never in that window.
Two ways a result fails to hold
It matters which of these happened, and they get conflated constantly.
A reproduction failure is running the same thing again and not getting the same number. That is not what happened here. The original measurement stands: prose was 3 percent of chunks and held all five top slots, scores were bunched between 0.941 and 0.944 with no discrimination left, and removing 89 files moved MRR from 0.142 to 0.333.
A generalisation failure is the same proportion doing a different job somewhere else. Prose held more slots in these five repositories than it did in the original one, and clearing it changed nothing, because in these the prose was not covering an answer. It was occupying a window that had no answer in it either way.
The correlation that made this look promising was real. Repositories with prose-heavy candidate lists really do score zero. The causal direction was the part that was invented, and it survived because it was the direction that suggested a fix.
Two smaller things worth keeping
The precondition was written separately from the result. Before running, one line said: if prose is not below five percent of candidates after reindexing, do not read the hit rate. After the folder rule it was exactly five percent overall, and thirteen and seventeen percent in two individual repositories. Reading the hit rate at that point would have been reading a half-applied change and calling it a result. The second rule went in for that reason and not because anybody expected the number to move.
The cost estimate was wrong by two orders of magnitude. Excluding files sounded expensive because a previous reindex of a similar size had taken two and a half hours. The actual run across six repositories took eleven seconds. The two and a half hours had been spent embedding a 323 MB dump, and an exclusion rule embeds nothing at all. Estimating a job from the largest number previously attached to a similar-sounding job is how a five-second task acquires a two-hour reputation and stops being tried.
Sample, stated plainly: 23 pairs, 6 repositories, hit rate 39 percent with a confidence interval from 22 to 59. A swing of three pairs would be indistinguishable from noise. What is not noise is that every cell is identical. This is not a small effect measured imprecisely. There is no effect.