Repository routing is the retrieval problem nobody benchmarks

The published work on issue localisation is good and I have learned a lot from it. It also answers a question I do not have.

Every benchmark I could find hands the system a repository and asks which file to change. SWE-bench does this. Its derivatives do this. The retrieve-then-rerank literature does this. The repository is a given.

In my organisation the repository is not a given. There are 28 of them, a ticket usually needs one to three, and choosing wrong wastes everything downstream: the agent writes correct code in the wrong place, or never opens the file that mattered.

Nobody publishes numbers on that step.

Why the gap exists

It is not an oversight. It is a consequence of how the benchmarks were built.

They are constructed from public repositories with rich issue histories: one repository, many issues, clean linkage between issue and fix. That is exactly the data you need for file-level localisation, and it is abundant.

Multi-repository ground truth is not abundant. You need an organisation where one ticket routinely spans several repositories, where the linkage between ticket and commits is preserved across all of them, and where that data can be published. Most places that have the first two cannot do the third.

So the literature optimises the part with public data, and the part without public data goes unmeasured. That is not a criticism. It is just worth knowing when you read an Acc@10 of 82% and wonder why your own pipeline feels so much worse.

What the numbers do not transfer to

Here is the concrete mismatch. A file-level benchmark reports something like:

Acc@10 = 82%     the right file is in the top ten, within the right repository

My pipeline's equivalent question is two-stage:

which repository?        28 candidates
which file within it?    hundreds to thousands of candidates

If the first stage is 70% accurate, the second stage's 82% is multiplied by it. And the published number implicitly assumes the first stage is 100%, because the repository was given.

I do not think anyone is claiming otherwise. But the number gets quoted as "issue localisation is basically solved," and it is solved for a setup that skips the step I spend most of my time on.

What industry does instead

The tools that operate at organisation scale do not solve routing either. They avoid it.

Enterprise code search indexes every repository into one searchable corpus and lets a single query span all of them. Repository selection is either not made at all (you search everything) or it is made by the user with an explicit filter, or by pre-defined groupings someone configured by hand.

That is a legitimate design and it removes the problem cleanly. It also does not help an agent, which has to decide where to work before it starts. A human searching everything can glance at the results and pick; an agent that opens a branch in the wrong repository has already wasted the run.

So the two available answers are "give the user everything and let them choose" and "the repository is a given." Neither is what an autonomous pipeline needs.

The two things I tried

Before settling I built two rankers, and the contrast between them is most of what I learned about the problem.

Summaries. Describe every screen in each repository in natural language, then match the ticket against those descriptions rather than against code. The reasoning is sound: a ticket and a screen description are both prose written by people about what a screen does, while code is neither. Matching language to language avoids the cross-domain jump entirely.

It works well where it works. It also cannot see 11 of my 28 repositories, because the summaries are generated from route definitions and those eleven have no routes to extract: static asset repositories, a publishing pipeline, backend services, and one frontend whose routing does not follow the convention the extractor recognises.

Not "ranks them poorly." Cannot see them at all. A repository with no screen index is absent from the ranking rather than at the bottom of it, and that distinction turned out to matter enormously.

Code search. Run the ticket text as a query against each repository's index and rank by the best scores. Covers all 28 by construction, because every repository has code.

Its weakness is the mirror image: the scores are not comparable across repositories. Cross-corpus similarity clusters within a few thousandths (0.943 against 0.942 against 0.942) so the ordering below the top position is close to arbitrary. It reliably puts the right repository in the list and is unreliable about where.

One covers everything and ranks vaguely. The other ranks well and is blind to 40% of the corpus. That is the actual shape of the problem, and it is not the shape any file-level benchmark has.

What I ended up measuring

Seven real tickets, ground truth taken from which repository the fix branch was actually cut in:

Both around 30% at rank one, both much better at top three. And the more useful column is the last one: the correct repository is somewhere in the list 7 times out of 7 with code search.

That reframed what I was building. Getting first place right is nice; getting the correct repository into a shortlist of three is what actually saves the agent from wasted work. The asymmetry is the point: glancing at three candidates is cheap and starting work in the wrong repository is not, though I should say I have never timed either, so treat the size of that gap as a guess and only the direction as something I would defend.

Fusing them made it worse

The obvious move is to combine the two, and reciprocal rank fusion is the standard way. I tried it and it was worse than either input on the cases that mattered.

ticket    summary    code    fused
  3       ABSENT     1st     14th
  6       ABSENT     6th     17th

RRF sums a contribution per ranker that returns the candidate, so a repository present in both lists collects twice. That is correct when both rankers could have ranked everything, since absence then means one of them looked and disagreed.

My summary matcher does not have that property. For those 11 repositories it returns nothing because it has nothing to return, and fusion cannot distinguish "considered and placed last" from "never heard of it." So every repository the summaries cannot see gets structurally penalised, which is precisely the set the code search was there to cover.

The fusion deepened the blind spot it was meant to fill.

What replaced it was an explicit order rather than a blend: code search decides because it covers everything, summaries annotate when they have an opinion, and a repository with no screen index carries a note saying so rather than a penalty for it.

Postscript: the condition was the whole story

Months later I fused two rankers again on this same problem and it won, which means the paragraph above is right about the mechanism and wrong as a verdict on fusion.

The sentence that turned out to matter is this one: that is correct when both rankers could have ranked everything. I wrote it as an aside explaining why my case failed. It was actually the specification for making it work.

What changed is the second ranker, not the fusion. Instead of screen summaries, which exist only where a repository has screens my extractor recognises, every repository got one short description of what it is and who uses it. Written by hand, one per repository, whether or not it has any screens. That index can rank all twenty-eight, so nothing is ever absent for structural reasons, and the double-counting that was penalising the blind spots became a genuine second opinion.

                          top-1   answer in top-3   answer in top-5
code chunks only           41%          73%              86%
descriptions only          41%          72%              90%
both, fused                51%          89%              94%

I want to be careful about what this does and does not overturn. The failure recorded above was real, the numbers were real, and dropping fusion was the right call at that time. With a ranker that cannot see 11 of 28 repositories, no amount of tuning the blend fixes it. What I got wrong was reading a property of my inputs as a property of the method.

The measurement that would have told me sooner is embarrassingly cheap: for each ranker, what fraction of the candidate set can it return at all? Mine was 100% and 61%, and I never wrote it down.

Some of it is unwinnable

The most useful thing I learned is that a chunk of this problem has no solution in the ticket text.

Several of these repositories are near-duplicates by design: a product and the older one it replaced, built from shared components with a flag choosing which of the two is being served. On a gold set of 22 tickets belonging to one of them, the summary matcher put it first zero times. Its sibling won every case.

That is not a ranking bug to fix. A ticket saying "the date is wrong on the reservation detail screen" is textually identical whether it concerns the new product or the old one. The information that distinguishes them (which deployment the reporter was using, which product they have access to) is not in the ticket at all.

A system that appeared to solve this would be exploiting something incidental, and would break the first time the incidental thing changed.

So the honest target is not rank-1 accuracy. It is getting all genuinely plausible candidates into a small shortlist, and being explicit that the tie cannot be broken from the input.

What I would want benchmarked

If someone builds this, three things would be worth having:

Multi-repository ground truth. Tickets that span more than one repository, with all the commits linked. The interesting cases are exactly the ones where the answer is a set rather than a single item.

A null baseline reported alongside. In a corpus where three repositories change together constantly, answering "those three" every time scores remarkably well. Any published routing number without that comparison is uninterpretable. I have measured a system at 79% that was losing to a constant answer scoring 95%.

Present-tense labels. A closed ticket's correct routing answer today is often "nothing, it shipped." Ground truth mined from history says otherwise, and that difference is not a model failure.

None of this needs new science. It needs a dataset that does not currently exist, from an organisation willing to publish its ticket-to-commit linkage across repositories.

What I do in the meantime

Route with code search across all 28 repositories, because it is the only signal that covers all of them. Use the ordering rather than the scores, because cross-repository similarity scores cluster within a few thousandths and are not comparable. Present three candidates rather than one, and annotate each with why it is there.

And measure it locally, since nobody else is going to. Seven tickets is a small gold set, and it is enough to have told me that fusing two rankers made things worse, that summaries cannot see 11 of my repositories, and that the sibling-repository cases are unwinnable.

That is not a benchmark. It is enough to stop me shipping things that do not work.