Two ways of finding the files a ticket touches: an embedding index, and an agent that greps around until it decides. Same 23 pairs, same inputs, same scoring.
The index found 9. The agent found 22.
That comparison is nearly useless as stated, and the interesting part is why.
Counting what was found, not what was read
Before any hit rate, one thing had to be checked. The input to both methods is the ticket text as filed, concatenated: title, description and every comment, with an explicit instruction not to rewrite or condense it. If that text names a file path, a system reporting that file has not found anything. It has read.
Raw tickets leak more than summaries would, not less. A summarising step might drop a path as incidental detail. A colleague replying in a comment thread with the file they think is responsible will not.
Twenty-two pairs were checked for this and one of them had the answer sitting in the input. So the agent's number is 21 of 23, or 91 percent, on the conservative reading, and 22 on the generous one.
This costs about ten minutes and it changes what the number means. A search benchmark whose queries contain their own answers measures reading comprehension.
One in twenty-two is a small correction, and the size is not the point. Nobody knew it was one until it was counted. Counting and then reporting one is a different act from not counting and assuming it is rare, even when both end at the same number, because only one of them can be wrong about it.
The trade is per row, not per total
| repository | index | agent | median added time |
|---|---|---|---|
| a backend service | 4/9 | 9/9 | +19.9s |
| a portal front end | 0/5 | 5/5 | +37.0s |
| an admin application | 3/3 | 3/3 | +33.1s |
| a legacy admin | 2/3 | 3/3 | +23.7s |
| a customer front end | 0/2 | 1/2 | +44.0s |
| a partner admin | 0/1 | 1/1 | +39.8s |
| total | 9/23 | 22/23 | 30.2s (6.0 to 47.8) |
One row buys five pairs for twenty seconds. Another buys nothing for thirty-three, because the index already had all three. And the two rows with the largest gains are among the slowest.
An overall figure of "39 versus 96" hides all of that. It suggests a switch, and what is actually here is a decision that should be made per call.
Which raises the question of what to make it on.
Repository is the axis we could measure
Six repositories, between one and nine pairs each. A three-for-three is three pairs. Writing per-repository rules into a configuration from that is the mistake this blog keeps describing, and it would be tempting, because the rows look so different.
More importantly, repository is probably not the thing that varies. What varies is whether the index found anything usable this time, and that is a property of the query and the run, not of the codebase.
So: is there a signal available before calling the agent that says whether the index did well?
The obvious signal does not work
Similarity scores were the first guess, and they are useless here.
| signal | hits, median | misses, median |
|---|---|---|
| top score | 0.048 | 0.053 |
| score spread, first to sixth | 0.011 | 0.005 |
| agreement between two retrievers | 4 | 5 |
Misses score higher than hits on two of the three.
That is worth stating carefully, because it is easy to over-read. The gap on the top score is 0.005, across nine hits and fourteen misses. It is not evidence that the signal runs backwards. It is evidence that there is no signal in that direction either, which is the weaker claim and the one the numbers support.
What it joins is a pattern. I have written here about an embedding space where unrelated documents sit above 0.90 cosine, and about a threshold on absolute similarity that fired on 270 out of 270 queries because the model's scores had moved out from under it. With this, three separate attempts to build a threshold out of absolute similarity have failed, for three different reasons.
The claim those three support together is narrow and useful: at this layer, absolute similarity is not threshold material. Not that it is meaningless, and not that it is inverted. That you cannot cut on it.
The signal that does work is concentration
| signal | hits, median | misses, median |
|---|---|---|
| distinct files behind 40 retrieved chunks | 11 | 31 |
When forty chunks collapse into eleven files, the index is pointing somewhere. When they spread across thirty-one, nothing matched strongly and the ranking is arranging noise.
Similarity cannot say how good a result is. Dispersion can say how much of one place it is looking at, and those are different questions. The second one turns out to be the answerable one.
It is also free. That count is the length of the candidate list before it gets truncated to six, so it already exists at the moment the decision has to be made.
Is concentration just repository size again
If large repositories scatter and small ones concentrate, then this is not a new axis. It is repository with the edges softened, and the whole thing returns to where it started.
Across repositories it mostly does track size, with one repository refusing to:
| indexed files | median depth |
|---|---|
| 406 | 23 |
| 443 | 10 |
| 1,121 | 8 |
| 3,098 | 28 |
| 5,758 | 27.5 |
| 5,802 | 33 |
The smallest codebase in the set scatters about as much as the largest. That is one counterexample and it is not a demonstration, but a perfect correlation would have ended the question and this does not.
The stronger test is within a repository, where size is held constant by construction, and it can only be run where both hits and misses exist. Of six, that is one.
In that one, nine pairs split four hits against five misses. Pairing every hit with every miss and asking which was more concentrated: nineteen of twenty pairs go the right way. A signal with nothing in it lands near ten.
Nine is small enough to enumerate rather than approximate. The depths were 26, 28, 24 and 8 for the hits, and 27, 32, 32, 29 and 29 for the misses. There are 126 ways to choose which four of those nine values are the hits, and exactly two of them reach nineteen or better.
Two things have to stay separate here, and they collapse into each other easily.
The repository was not selected. It was the only one that could be tested, because every other one is all hits or all misses. That means there is no multiple-comparison problem, and equally no independent confirmation. One test was possible and one test was run.
The threshold is still untested. What has a p-value attached is the relationship between concentration and hits. The cut at depth 12 was picked by looking at the outcome and nothing has evaluated it. The relationship is measured. The threshold is still a hypothesis in a table's clothing.
There is also something worth noticing about why this was measurable at all. A paired test that counts successes and failures cannot reach significance below a handful of discordant pairs, no matter what the data does, because there are only so many values the count can take. Ranking a continuous quantity across the same nine observations has 126 arrangements to work with. Same sample, same day, different question, and one of them can produce an answer while the other structurally cannot.
The lesson there is not that small samples are fine. It is that "the sample is too small to tell" and "this metric cannot tell, at any size like this" are different diagnoses, and the second one is fixable by changing what you measure.
The four repositories that cannot be tested are part of the answer. All four are entirely hits (3 of 3) or entirely misses (0 of 5, 0 of 2, 0 of 1). That is exactly why the per-repository table looked so decisive earlier. Repository resembled an axis not because repository is an axis, but because within each one every answer happened to fall on the same side.
One caveat on the depth numbers, and it runs the useful direction. Depth is counted after excluding dependencies, build output and data files. That filter removes more from JavaScript repositories, and removing files lowers depth, which pushes against the pattern actually observed rather than manufacturing it. Not zero, but not flattering either.
What splitting on it would do
| rule | index only | sent to agent | total hits | time |
|---|---|---|---|---|
| index for everything | 9/23 | 9/23 | 0 min | |
| depth ≤ 12 stays | 5/6 | 16/17 | 21/23 | 8.0 min |
| depth ≤ 25 stays | 7/10 | 13/13 | 20/23 | 6.1 min |
| agent for everything | 22/23 | 22/23 | 10.6 min |
Twenty-five percent less time for one lost hit.
That threshold was chosen after looking at the results. There are nine hits in this set, and a split this clean happens by chance often enough at that size that the number is not doing the work people will assume it is doing. Until it is tested on a set nobody has seen, it is a hypothesis wearing a table's clothes.
Where this actually leaves things
The honest summary is longer than a conclusion usually is, and it has to be:
Repository was the axis available, not the axis demonstrated. There is a candidate for a better one, it costs nothing to compute, and on these 23 pairs it separated hits from misses cleanly, 11 against 31. The cut point is a guess made in hindsight and needs a fresh sample.
Concentration is not simply size wearing a different name, since the smallest repository scatters like the largest. Inside a repository the question could be asked exactly once, in the only one holding both hits and misses, and there it came apart almost completely: nineteen pairs of twenty, exact p of 0.016. That is one test, not a settled question, and it says nothing about where the cut should go. What is missing is a second repository with some of each.
So the weakness in the per-repository table does not go away. What changes is that there is now a specific way to make it go away, which is a different position from not knowing what to try next.
And the whole comparison measures retrieval, not outcomes. Whether a better candidate list makes the work finish better was not measured at all, and a confidently wrong list may be worse than none: "start here" pointed at the wrong place costs more than silence.