I had two ways of deciding which repository a ticket belongs to. One matched the ticket against natural-language summaries of each screen. The other ran the ticket as a code search across every repository and ranked by score.
Fusing them with reciprocal rank fusion is the standard move, and on paper it should have been free improvement: two independent signals, combine, get the best of both.
It took two of my seven test cases and moved the correct answer from first place to fourteenth, and from sixth to seventeenth.
The measurement
Seven real tickets where I know the answer, because I can see which repository the fix branch was cut in.
| method | rank 1 | top 3 | answer present |
|---|---|---|---|
| summary matching | 2/7 | 4/7 | 5/7 |
| code search | 2/7 | 5/7 | 7/7 |
| RRF fusion | 3/7 | 5/7 | 7/7 |
Fusion looks best on that table. It gets one more first-place hit than either input.
Then the per-case view:
Two cases got dramatically worse, and they're the two where the summary signal had no opinion at all.
Why absence is not neutrality
RRF scores each candidate by summing 1 / (k + rank) across the rankers that returned it. A repository appearing in both lists collects two contributions; one appearing in a single list collects one.
That's the intended behaviour when both rankers could have ranked everything. Absence from one list then means something: that ranker looked and disagreed.
My two rankers do not have that property. The summary matcher works from screen-level descriptions extracted from route definitions, and 11 of my 28 repositories have no screens at all: static asset repositories, a publishing pipeline, backend services, a frontend whose routing does not follow the framework convention the extractor recognises.
For those 11, the summary matcher does not return a low rank. It returns nothing, because it has nothing to return. And RRF cannot tell the difference between "this ranker considered it and placed it last" and "this ranker has never heard of it."
So every repository the summary matcher cannot see is structurally handicapped in the fusion, regardless of what the code search thought. Case 3's correct answer was first by code search and fell to fourteenth after fusion, beaten by repositories that were merely mediocre in both lists rather than excellent in one.
The fusion was punishing exactly the repositories the second signal was supposed to compensate for. I built it to cover a blind spot and it deepened the blind spot instead.
The other thing the code ranker couldn't do
While I'm being honest about the inputs: the code search that saved those two cases has a weakness of its own, and it shows up in the same table.
Its scores across repositories cluster inside a few thousandths:
Five repositories separated by one thousandth of a point. That ordering is not a judgement, it's a coin flip with extra decimal places, the same clustering problem that shows up whenever similarity scores are compared across corpora rather than within one.
Which is why I use the code ranker's order and never its scores. Rank position survives the clustering; the raw number does not separate one repository from the next on a given query.
I want to be careful with that sentence, because I later measured a version of it that is not true. Across many queries the raw numbers do carry something: a repository's average score correlates with how often it is the answer at all (r = 0.448 over 28 repositories). That is a fact about the repository, not evidence about this ticket, and it is exactly the kind of thing that survives averaging and dies at the level of a single query. What I can say from the measurement above is the narrow claim: on one query, a thousandth of a point between two repositories is not a judgement.
It also explains why the code ranker gets "answer present 7/7" but only "rank 1: 2/7". It reliably puts the right repository somewhere in the list, and is close to arbitrary about where in the top five. That is a genuinely useful property (narrowing 28 candidates to 3 is most of the work) but it is not the same as being right, and I try not to describe it as if it were.
The aggregate number hid it
Look at the summary table again: 3/7 first place versus 2/7. By that number, fusion won and I should ship it.
The per-case list is the same data and says the opposite. Two catastrophic regressions, offset by one modest gain, presented as an improvement because the metric counts cases rather than measuring distance.
I'd been running the aggregate for a while before I printed the per-case breakdown, and there was nothing in the aggregate that would ever have told me. A rank-1 counter cannot distinguish "went from 3rd to 4th" from "went from 1st to 14th". Both are just a case that isn't in the numerator.
Since then I print the per-case ranks alongside any aggregate on a small sample. On seven cases it costs seven lines and it is the only way to see a regression that averaging hides.
What I do instead
Not fusion. Order.
Code search decides, because it covers all 28 repositories. Summaries explain, because when they have an opinion it is often a good one and always more readable than a file path.
Two things this gets right that fusion did not.
A repository with no screen index is still a candidate. It just carries a note saying why the second opinion is silent, which is information rather than a penalty.
And when both signals agree, the reader sees it (summaries also rank this 4th) without that agreement being baked into a number that hides which signal produced it.
The tickets that fusion broke are back to first and second place, because nothing gets demoted for being invisible to a ranker that cannot see it.
Sibling repositories are a separate problem
One more limitation worth naming, because it affects all three methods equally and no amount of fusion fixes it.
Some of these repositories are near-duplicates of each other by construction: an application and its predecessor, sharing components with a flag deciding which product is being served. A ticket about one of them is almost always textually indistinguishable from a ticket about the other.
On a gold set of 22 tickets belonging to one such repository, the summary matcher put it first zero times. Not occasionally wrong: never right. Its sibling won every single case. The two share vocabulary, share screens and share most of their code, which is my reading of why rather than something I traced case by case.
That is not a ranking bug. There is no signal in the ticket text that separates them, and a method that claimed otherwise would be overfitting to something incidental. The information needed to break that tie lives outside the ticket (which product the reporter uses, which deployment the bug appeared in) and none of my rankers have it.
I mention it because it changes what "good" looks like for this task. When two candidates are genuinely indistinguishable from the input, the right target is both of them in the top three, not the correct one in first place. Optimising for rank-1 accuracy on a corpus with near-duplicate members is optimising for luck.
When fusion is the right tool
I don't want to leave this as "RRF is bad." It is very good at the thing it was designed for, and that thing has a precondition.
RRF works when your rankers have comparable coverage, meaning every candidate is genuinely rankable by every ranker, so absence from a list is a judgement. Dense retrieval and BM25 over the same document set are the textbook case: both see every document, both produce a full ordering, and disagreement is meaningful. Inside a single repository, my own search uses RRF for exactly that, and it works.
The precondition fails when one ranker has structural blind spots. Then absence is missing data wearing the costume of a low score, and any method that treats rank position as evidence will penalise the wrong things.
Two ways out, if you need to fuse anyway:
Impute a neutral rank. Place unseen candidates at the midpoint of that ranker's list rather than omitting them. Absence stops being a penalty. I didn't do this because the neutral value is arbitrary and I couldn't defend a specific number.
Weight asymmetrically. Let the complete ranker set the order and use the partial one only as a bonus for candidates it ranked highly, never as a penalty for the rest. That's very close to what I ended up with, minus the arithmetic.
What I'd check before fusing anything
Does each ranker see every candidate? Count the candidates each one can return, not the ones it did return on your test queries. Mine was 28 versus 17 and I had never counted.
Print per-case ranks on small samples. Aggregates hide regressions by construction. Seven lines told me what a summary table could not.
Ask what absence means in your data. If a ranker omits something, is that a judgement or an inability? RRF assumes the first. Verify it rather than assuming.
Postscript: the condition held, but not the fix I planned
I came back to this and fused two rankers on the same problem, and this time it won: top-5 went from 86% to 94%, first place from 41% to 51%.
Nothing about RRF changed. What changed is that the second ranker now covers every candidate.
Above I ask myself why I don't just fix the incomplete ranker, and answer that the eleven repositories it cannot see are eleven separate problems: one has no routes, one is static assets, one uses a convention the extractor doesn't parse. That was true and I never did fix them. I stopped trying to extract a second opinion from the repositories and just wrote one instead: a few sentences per repository saying what it is and who uses it, one file each, the same for a repository with two hundred screens as for a repository with none.
That took an afternoon, against eleven parser problems I had been quietly avoiding for weeks. Coverage was the requirement; extraction was only ever one way to get it, and it was the expensive one.
The check I recommend above (count the candidates each ranker can return) is still the right check. I'd add one thing to it now. If the answer is bad, the next question is not "how do I make this ranker see more" but "what else could produce an opinion about everything?" Those have very different costs, and I spent weeks on the first before trying the second.