I could not measure the threshold where it mattered, until somebody could

A code search tool has a shortcut. If the top five vector results are all close enough, it skips the lexical half of retrieval and returns what it has. The cutoff is a distance of 0.15, which is a cosine similarity of 0.85.

In this embedding space, documents with nothing to do with each other score above 0.90. A hotel scraper against a query about tagging: 0.90. A payments tool: 0.90. The lowest of twenty-nine unrelated documents: 0.9021.

Nothing can fail a 0.85 cutoff in a space where the floor is 0.90. The condition is not a test. It is true.

Counted directly, on ninety queries across three indexes:

Two hundred and seventy for two hundred and seventy. It has never once not fired.

So the lexical half of this hybrid search has never run. Not rarely. The BM25 index is built, the fusion is implemented, the fusion constant is tuned per query type, and control has never reached any of it.

The cutoff was correct when it was written

0.85 was not a guess. It was chosen against a different embedding model, a 768-dimensional one, where similarity scores were spread widely enough that a document above 0.85 really was unusually close. Under that model the condition selected something.

Then the model was replaced with a multilingual one, and its scores sit near 0.93 for everything. Same constant, same comparison, same code. The condition became true.

That is the part worth slowing down for. Nobody edited this line. It appears in no diff, no review, no changelog. A parameter written against one distribution kept its value while the distribution moved out from under it, and the moment it stopped being a test produced no output at all.

A cutoff on absolute similarity is a claim about the model, stored in a file that does not mention the model.

The obvious conclusion, measured

If half of retrieval has been silently disabled, the system has been losing whatever that half contributes. That was the prediction. Restoring it should recover something.

Ninety queries, a frozen copy of the index, all twenty-nine documents scored rather than a shortlist reranked:

Forty-three queries moved. Nineteen improved and twenty-four got worse. Turning the lexical half back on cost three at rank 1 and five at top 5.

On this index, the broken threshold was doing the right thing. Not because it was tuned to, but because a constant has to land somewhere, and on twenty-nine short documents it landed on the better side.

Hold that sentence loosely. It does not survive the end of this post.

With one condition attached, and it matters. The lexical half in that comparison had been repaired the day before. It was defaulting to AND across terms, so a query containing one word absent from the code returned nothing at all, taking the identifiers with it; on one query set that was 0 results out of 30 before the fix and 12 after. A second bug left an operator character unescaped, killing whole queries outright. Both had been invisible because nothing ever reached that path.

So the comparison is against lexical retrieval at its best so far, not against what was actually being suppressed all this time. What the threshold was hiding was worse than what the measurement restored. That cuts against the finding rather than for it, which is why it belongs next to the table rather than in a footnote.

Two questions that look like one

The name of that constant is a confidence check. What it does is disable lexical retrieval unconditionally. Those are different statements, and only the first one is false.

It is easy to collapse them, because a check that never checks anything reads as obviously broken, and obviously broken things get fixed. Fixing this one would have been a quality regression shipped with a commit message about restoring intended behaviour.

The separation to hold on to: is this doing what it claims and is what it does correct are two measurements. The first needs a code read. The second needs a gold set. Only the first one was done here for months, and it is the one that produces confident wrong conclusions.

Why lexical hurt here

Twenty-nine short documents, one paragraph each. There is very little for a term-frequency ranker to work with, and every document shares vocabulary with every other because they all describe software.

Reciprocal rank fusion takes positions, not scores. A document that a weak lexical ranker happens to place third contributes as much as a document the vector ranker placed third with real evidence. When one ranker has almost no signal, fusion does not average away its noise. It promotes it.

That is the mirror image of something I measured earlier on the same pipeline, where fusion failed because one ranker could not see eleven of twenty-eight repositories and RRF structurally punished exactly the ones it was blind to. Here the second ranker sees everything and knows nothing, and fusion overweights it anyway.

Same fusion, opposite failures, and the variable is how much signal the weaker ranker actually carries.

Where the claim stops

The quality comparison above is a description index: twenty-nine short documents, one per repository. The real code indexes are hundreds of thousands of chunks, and lexical matching has far more to grab there. There is no file-level answer key for those, so which setting wins on them is not measurable today. That last clause is the one this post ends up retracting, and it is worth watching how confident it sounds.

What is measurable is how much the setting changes:

The effect grows with the index. On twenty-nine short documents the shortcut changes the top result for 42 of 90 queries. On a real codebase it changes it for 70.

Which points somewhere uncomfortable. The one place the shortcut was measured for quality is the place where it matters least, and it won there for a reason that does not travel: with twenty-nine paragraphs there is almost nothing for a term-frequency ranker to find, so fusion was promoting noise. Give it a real codebase and that argument runs backwards.

So the honest reading is not that the shortcut is good. It is that the shortcut is unverified exactly where it does the most, and it happens to have won the one comparison that was cheap enough to run.

Reading eight of them by eye

No answer key does not mean no information. Eight queries against a real code index, looking at what entered the top five when the shortcut was disabled and what left:

Small, subjective, one repository. What makes it worth reporting is that the wins and the losses have different shapes.

Both wins were rare proper nouns. One query named a partner brand, and lexical retrieval pulled in the component named after that partner while pushing out an unrelated integration screen. Another was a ticket about a scheduling change, using one specific domain word four times; lexical brought three chunks of the dialog component that implements exactly that, where the vector ranker had latched onto cancellation and refund instead.

The loss was the opposite shape. A query about a marketplace listing screen, where lexical promoted generated API schema and end-to-end test specs and demoted the component that actually handles it.

And three of the five draws were the same generated schema file trading places with itself at different offsets. Both settings were wrong, differently.

That is a mechanism, not a preference. Term matching earns its keep on rare tokens that an embedding smooths away, and loses on documents that are large enough to contain everything. Which is a property of the query, not a property of the system.

It is also, read forwards rather than backwards, a prediction. Twenty-nine paragraphs contain almost no rare tokens, because they were written to describe things in general terms. A codebase is made of them: identifiers, partner names, domain words that appear in four files and nowhere else. If this mechanism is right, the small index is the worst possible place to decide whether lexical retrieval is worth having.

The shortcut replaces that judgement with one number, and that number always answers no.

Eight cases read by a person is not a measurement. It is the thing you do when the measurement is unavailable and you would otherwise decide on nothing.

Then somebody built the answer key

I wrote above that the comparison could not be run on a real code index, because no file-level answer key existed for one. That was true in the sense that no such file was sitting there, and false in the sense I meant it.

The engineer who owns the fork built one the same evening. It is the standard construction for retrieval-based bug localisation: take issue tracker titles as queries, take the files changed by the commits carrying that ticket number as the answer, and keep only tickets whose fix touched between one and five files, since a fifty-file commit makes top-five free.

One detail in that construction is the whole difference between a benchmark and a leak. The queries are the tracker titles, written by the people who reported the problem before anyone looked at the code. Commit titles were deliberately not used, because a developer writes those after reading the code and they contain file and function names. Using them would be scoring a system on a question that already contains the answer.

Forty-two tickets survived the filter, with 101 answer files, an average of 2.4 per ticket. On a real code index:

Every column moves the same way, and first place improves by two thirds. Restricting the answer key to source files only, dropping configuration and generated output, gives the same direction again.

So the sentence I was about to publish is wrong. The shortcut is not pinned on the better side. It is pinned on the better side of the index where almost no work happens, and on the wrong side of this one.

Read that last clause carefully. It says this one.

Forty-two tickets, twenty-six of which changed position, seventeen up and nine down. The evidence is five metrics agreeing, not a p-value on any one of them, and it is one repository in one language. The index is also at current HEAD, so fixes for these tickets are already in it; absolute scores are inflated and only the comparison between the two arms means anything.

The second repository

The same construction, on a backend service in a different language. Forty tickets.

It did not replicate. On the second repository more tickets got worse than better, and on the plain-language subset, which carried the entire effect the first time, the change is exactly zero.

Run through a significance test rather than read off the table, the split is sharper than that. On the front end, top 1, MRR and MAP clear p < 0.05; top 3 and top 5 do not. On the backend nothing clears it at all. So "did not replicate" stops being an impression about a table and becomes a result.

The first guess for why was that the queries are Korean and this codebase is not. That guess was measured and is wrong: the backend contains more Korean than the front end does, 9.4% of chunks against 7.2%.

What the numbers actually say:

On more than a third of the backend tickets, the file that fixed the issue never appears among the candidates at all. No ranker can order a document it was not handed. Those tickets are unwinnable before ranking begins, and they are what the comparison was averaging over.

So the honest statement is not that lexical retrieval fails on this repository. It is that on this repository the experiment was mostly measuring recall while reporting on ranking, and there was not enough material for the ranking question to be asked. Whether lexical retrieval would help there is still unmeasured, and this result is not evidence either way.

I have written that sentence before, about rerankers: a reranker cannot find what retrieval never returned. It applies one layer down and I did not notice until the number forced it.

A stronger claim was nearly made here and did not survive its own check. If the answers are missing from the candidates, perhaps they are missing from the index entirely, which would be a much more serious defect. Twelve files came back as absent. Before reporting that, the query used to test it was pointed at files known to be indexed, and it returned zero for those too. The instrument was broken, not the index. With a probe that had been verified against a known answer, all twelve turned out to be present.

That is the same shape as counting a firing rate from a log that had been switched off, in the same investigation, four hours apart. An absence is only evidence when the thing that reports absences has been shown to report presence.

Checking the answer key for leaks

A benchmark assembled this fast deserves suspicion, and it got some. The question raised against it: issue tracker titles are current values, not the text as filed. If anyone renamed a ticket during the investigation, the conclusion flows backwards into the query.

The change history came back clean. No title on the suspect tickets had ever been edited.

The actual problem was worse and invisible to that check. On several tickets the title matched a sentence in the investigation section of the original description word for word, and that same description listed the paths of the files that turned out to be the answer. The leak was not a later edit. The ticket was filed after the investigation, with the conclusion as its title.

So the question became how much of the result rests on those. Splitting the forty-two by whether the query contains code vocabulary at all, meaning camel case, snake case, paths, file extensions, or API verbs:

The four contaminated queries score perfectly under both settings. They contribute exactly zero to the difference between the arms. The entire gain comes from the thirty-eight ordinary ones, where first place goes from five to eleven.

Their perfect scores are themselves the evidence that the leak is real: a query carrying an identifier lands at rank one no matter which retrieval path it takes. Real, and irrelevant to the comparison.

Three sentences, all of which have to be said together: the leak exists, it inflates the absolute numbers, and it does not touch the A/B conclusion.

That is also the first check today that left a result standing. Every other one this week removed something.

Everything above was rebuilt from parts that already existed

Someone asked whether an off-the-shelf tool does this. It does, and it has for years.

Re-running the whole comparison through that library reproduces the hand-built numbers exactly, to every count. Two independent implementations agreeing is the best evidence available that both are right, and it is also the only reason this section is not a retraction.

The uncomfortable line in that table is the second one. A day went into arguing about whether seventeen against nine meant anything, and the answer to that argument is a parameter with a name.

The tempting conclusion is that using the standard tools would have prevented the mistakes, and that is not true. Four measurement errors happened here over two days, and three of them were in the engine, not the evaluation: a comparison that moved two variables at once, a threshold that silently changed which scale the next threshold read, and a counter that read a suppressed log as a missing event. No evaluation library sees any of that. It would have caught one, the habit of speaking with more confidence than the arithmetic supported, and that one was the cheapest to fix and the loudest in the write-up.

There is a version of this post that ends with a lesson about reaching for libraries. That version would be tidier and it would be claiming a fix for problems the library does not touch.

What "cannot be measured" was actually saying

I had written that this could not be measured. It took one evening.

That sentence was not a fact about the world. It was a statement about what I was willing to spend, wearing the grammar of an impossibility. The two are easy to confuse from the inside, because both of them end the conversation in exactly the same place, and only one of them is falsifiable by somebody who is less tired.

The tell is available in advance, and I even wrote it down: I had already noticed the measurement I could afford was the measurement that mattered least. That is the shape of a budget constraint, not a shape reality tends to have on its own. When the cheap comparison and the important comparison come apart that neatly, the thing to interrogate is the price, not the result.

What I did not expect was that spending more would keep changing the answer. Three measurements, at three prices, in two days:

Each of those is a correct measurement of the thing it measured. None of them is the answer to the question I started with, and the third one is the first that says so out loud rather than by being contradicted later.

That is not an argument for measuring less. The first result would still be sitting there as the conclusion. It is an argument against the moment right after a measurement lands, when a number is in hand and the temptation is to stop, because the number is real and stopping feels like rigour rather than the end of a budget.

The commit that diagnosed it and left it alone

None of this was found today. The engineer who owns the fork found the always-firing threshold the day before, traced it to the model swap, and committed a fix. The commit message contains this line:

This is a value you have to revisit when you change the model.

The same commit added an environment variable to override the cutoff, and left the default at 0.15.

That is not a failure to notice. The cause was understood, written down accurately, and the tool for changing it was built in the same change. Then the value stayed where it was, because moving it would have required measuring, and measuring was not that day's work. Adding the switch was what "handled" looked like.

Making something configurable resembles fixing it closely enough to pass for it. The diff grows, the commit message is true, the parameter is now adjustable, and the behaviour in production is identical. A switch is a deferred decision that reads in the history as a decision.

A day later the two positions were measured on the small index, and the untouched default came out ahead. That looked like luck landing well. A day after that the code index was measured and the default lost on every column, so it was not luck landing well, it was luck taking a while to land.

Which is the argument for deciding rather than deferring, and it is not the argument I would have made yesterday. A default reached by not deciding is not wrong because it is wrong. It is wrong because nothing about it responds to evidence, including evidence that arrives later.

The second threshold, and why it is not a wrong number

There is another cutoff downstream. If the top result scores below 0.02, the system marks the answer low-confidence. Counted across the same three indexes:

Zero out of two hundred and seventy, and this is a zero with a denominator: every query produced a score, and the smallest of them is forty-five times the cutoff. That distinction is the whole difference between a measurement and a broken counter, and it is worth spending a column on, because the same investigation had already produced a zero that turned out to mean the logging was off.

The interesting part is why 0.02 is a sensible number that never sees a sensible input. It was written for fusion scores, where a first-place result tops out near 0.05, and against that scale 0.02 means something.

But the shortcut upstream always fires, so the pipeline runs vector-only, and vector-only returns raw cosine rather than a fusion score. The check receives 0.93 where it expected 0.03.

So the two cutoffs are not two instances of the same bug:

The first one is a wrong value. The second one is a correct value reading the wrong unit, and it reads the wrong unit because of a branch in a different file. Nothing in the file containing 0.02 has ever been wrong. What killed it lives somewhere else, connected by one runtime decision, and no amount of reading either file alone would show it.

Which extends the earlier line rather than repeating it. A parameter loses its meaning as an event rather than a state, and the event does not have to happen where the parameter is.

The part that transfers

A threshold expressed in absolute similarity is only a threshold if the model's similarity has an absolute meaning. Compressed embeddings turn cutoffs into constants, and a constant does not announce itself. It produces a well-formed result on every query, forever.

The way to find one is to ask what fraction of the time it fires. Never and always are both answers that mean the parameter is not doing its job, and both produce well-formed output forever. Neither shows up as a failure, because losing its meaning is an event and being broken is a state, and only states have symptoms.

When you find one, the next question is not how to fix it. It is what the system has been doing while the parameter was inert, and whether that was better. Here that came out three different ways in two days, and what separated them was which index anyone could afford to check.

So the question after that one: what would it cost to measure this where it actually runs, and is that number smaller than it feels. Mine felt like a wall and turned out to be an evening, twice, with different answers.

The threshold is still at its original value. Not because leaving it there was shown to be right, which is what I believed for about a day, but because nothing has been shown yet, and a default that survives three inconclusive measurements is in a different position than one that was never questioned. That is a smaller result than I expected to write. It is the one I have.

There is a second question I did not think to ask until the counting was done. A constant that always fires means every downstream component it guards has never run. The fusion here is tuned, has constants chosen per query type, and appears in the architecture diagram. None of that code has executed here. Whatever is wrong with it, nothing in this system would have said so.

The threshold at least has a story: it was right once, and a model swap ended it. The fusion constants did not have one, having arrived from upstream and never run here.

They have one now, and it took three seconds. A standard retrieval evaluation library has a function that sweeps the fusion parameter against a gold set, and pointed at both answer keys it gives:

The value the engine actually uses is 20. On the backend that is the optimum. On the front end it is within 0.002 of the optimum, which is not a difference. The untested constant that has never executed is, as far as two corpora can say, the right one.

I had written this section expecting to leave a loose thread. The thread was three seconds long.