Someone working on another part of the system sent me a routing failure with a diagnosis attached, which is the most useful kind of bug report and the most dangerous kind of evidence.
The report was specific. A ticket that refers to one of our smaller products by its internal nickname does not reach the repository that serves it. I had a fix for that, or thought I did, and it had already failed a measurement.
The word was in the document and did nothing
Some of our products have internal names that appear nowhere in the code: a nickname everyone in the company uses and no file contains. I had written those down, one line each, and tried putting them into the per-repository descriptions the router ranks against.
That made routing worse. Not on one gold set but on two, built separately for exactly this reason:
So when the report arrived, the first question was whether this ticket was the case that overturned it. It looked ideal: the nickname is in the ticket, the nickname is in my glossary, and the routing is wrong.
I ran that query with the glossary on and off and diffed the output. Character-identical. Same repositories, same order, same scores. The word was in the description document and contributed nothing at all.
That is a fair description of what dense retrieval does. The description is embedded as a sentence, and a ticket about coupon forms and notice copy sits close to every administrative repository in the corpus. One rare noun in the middle of a sentence does not move its average very far. The glossary was not being ignored because it was wrong. It was being averaged away.
Which is exactly what lexical search is for
Rare-term exact matching is the thing BM25 has always been good at and dense retrieval has never been good at. My description index was running dense-only, because there is a fast path that skips the lexical half when the vector results look confident, and I know it was firing here because turning it off changed the answer.
From off the shortlist to the top of it, on the exact case that was reported, with a mechanical story that holds together: the nickname is a rare term, rare terms are what the lexical ranker is for, and it had not been running.
At that point I had a fixed bug, an explanation, and one of my own conclusions to overturn. The glossary was not useless after all. It had just never been reachable. I was ready to write that up.
Then I ran it on everything
Both switches, all four combinations, on the same gold set of 90 tickets:
And on the smaller set of real tickets I keep separately, sampled evenly across product lines rather than by where the answers happen to fall:
The configuration that fixes the reported case is the worst row in the table. Twelve tickets left the top three to bring one in.
I did not ship it.
The order I did things in
The sequence matters more than the result, and I did it in the wrong order.
I had a conclusion supported by two gold sets. A single case arrived that appeared to contradict it. I found a mechanism for the case, confirmed the mechanism moved that case from ninth to first, and started composing the correction, to the earlier write-up and to a data file elsewhere that quotes its numbers, before running the change against anything except the case that suggested it.
The measurement that stopped me took twenty-five minutes, and I nearly skipped it because the case was already explained. That is the part worth naming: the case was not weak evidence. It was strong evidence for a claim I was not making. It showed that a lexical ranker can rescue a rare term that dense retrieval averages away. It said nothing about what that ranker does to the eighty-nine queries where nothing needed rescuing.
A case with a mechanism is the dangerous kind
The report came with a diagnosis, and the diagnosis was right in every part: the nickname does not appear in the code, dense retrieval cannot recover it from one occurrence inside a sentence, this is a vocabulary gap. All true.
A case that arrives with a mechanism feels like a proof. It has a before, an after, and an explanation joining them, and the explanation makes the next step obvious. What it does not have is a denominator. Mine had n = 1 and a good argument, and the argument was strong enough that I was ready to spend an afternoon writing a reversal rather than twenty-five minutes checking it.
I have argued on this site that on small samples you should print per-case ranks next to the aggregate, because an aggregate hides regressions. This is the same mistake from the other end. I had one case in complete detail and no aggregate at all, and one case hides everything an aggregate is made of.
The case was not a stable measurement either
One more thing I only noticed while writing this up. When I first ran the failing query I recorded the correct repository at twelfth. Reproducing it for this post, the same query against the same index put it at ninth, and at ninth again under a second description build. The direction is the same and it is large either way, but the number I had written down was not repeatable.
The reason is mundane: the descriptions had been rebuilt in between, and a description index is a corpus of twenty-eight documents where changing one line changes everyone's neighbours. The "before" in a before-and-after is a measurement too, and mine had drifted while I was busy being convinced by the "after."
Twelve is a net, and fifty-three tickets moved
Having said all that about aggregates, the number I stopped on was itself an aggregate. Top-3 fell by twelve. So I went and got the per-case ranks, which is the thing I keep telling myself to do first.
Twelve is what is left after thirty-four regressions and nineteen improvements cancel. Fifty-three of ninety tickets changed position to produce a net of twelve, and no summary row shows that.
Half the damage is small: sixteen of the thirty-four regressions are a single rank, the kind of movement that means nothing on its own. The rest is not: one ticket went from first to eleventh, and eighteen tickets that had been in first place were not any more.
That last pair is the number I would show someone deciding this. Eighteen lost first place and twelve gained it. If you only read the top-1 row you see a drop of six and might call it noise on ninety samples. It is not noise. It is thirty tickets changing status in opposite directions, and the fact that they nearly cancel says nothing about whether the change was good.
What I still cannot say
I know which tickets moved. I do not know why any individual one moved, and I want to be clear that the paragraphs above stop where the measurement stops.
The lexical ranker participates in the fusion for every query, not only the ones with a rare term, and I did not instrument per-query contributions. So "shared vocabulary pulled in the wrong siblings" is my reading of the regression list, not a measurement of it. It is consistent with what I measured elsewhere, where words shared across repositories are what makes descriptions blur together, but I have not traced a single one of these thirty-four to that cause.
Where it stands
The ticket that started this still routes to the wrong repository. I know roughly what would fix it and I have not built it: the nickname has to be able to matter without every word it shares with five other repositories mattering too. Neither of the switches I have is that. Turning on the lexical half weights the shared words along with the rare one, which is why it wins one case and loses eighteen firsts.
I told the person who reported it that their diagnosis was right, the fix works, and I am not shipping it. That is an unsatisfying answer to give someone who handed me a correct analysis of a real bug, and it is the true one.