One failure, read three times
Routing a ticket to a repository
Twenty-eight repositories and a ticket that belongs to one of them. Fusing two rankers lost, then lost again, then won — with nothing about the fusion changed. Read in order, these are the same event from three distances: the mechanism, the verdict I drew from it, and what the verdict was actually a specification for.
5 posts
Where this stands
- Held to be true
- Picking the right repository is a retrieval problem with almost no published benchmarks, and the obvious signals do not work on it. Absolute similarity scores failed three separate times as a basis for any threshold. Size bias is real and survives the usual normalisations.
- What would break it
- A similarity threshold that separates good routing results from bad ones on a held-out set.
- Still open
- Whether concentration replaces similarity as the usable signal. One repository showed clean separation at p 0.016, the cut point was chosen after seeing it, and no second repository has both hits and misses to test on.
- 1Retrieval11 min
Fusing two rankers made routing worse, and the reason is structural
Reciprocal rank fusion made routing worse: one signal covered every repository, the other 17 of 28, and RRF punished exactly the repositories the second could not see.
- 2Retrieval10 min
Repository routing is the retrieval problem nobody benchmarks
Code search across many repositories: every published issue-localisation benchmark assumes you already know which repository to look in. In a multi-repo organisation that is the hard part.
- 3Retrieval14 min
The biggest repository wins, and it has nothing to do with relevance
Retrieval bias toward big repositories: every repo scored between 0.924 and 0.943 for every query, and the 0.019 that separated them was measuring size.
- 4Retrieval8 min
The one case it fixed and the ones it broke
A BM25 ranking fix from a correct diagnosis of a real bug: that ticket moved from ninth to first, and eighteen other tickets lost first place.
- 5Retrieval10 min
The index found 39 percent, the agent found 96, and the useful question was neither
Agentic search vs an index on the same 23 pairs: the index found 39 percent and the agent 96, and per repository the trade ranged from five hits for twenty seconds to nothing at all.