Topic
routing
4 posts
- Evals6 min
My router scored 100% and a fixed answer scored 40%
Benchmark scoring with expected ⊆ answered: nothing was ever missed, so the score was perfect, and answering three fixed names without reading anything scored 40% on the same rule.
- Retrieval8 min
The one case it fixed and the ones it broke
A BM25 ranking fix from a correct diagnosis of a real bug: that ticket moved from ninth to first, and eighteen other tickets lost first place.
- Retrieval14 min
The biggest repository wins, and it has nothing to do with relevance
Retrieval bias toward big repositories: every repo scored between 0.924 and 0.943 for every query, and the 0.019 that separated them was measuring size.
- Retrieval11 min
Fusing two rankers made routing worse, and the reason is structural
Reciprocal rank fusion made routing worse: one signal covered every repository, the other 17 of 28, and RRF punished exactly the repositories the second could not see.