Your reranker speaks English. Your tickets do not.

I turned on reranking and search got worse. Pass@1 dropped from 26.2% to 23.8%, and the whole run took 63% longer to produce that worse answer.

My first instinct was that I'd wired it up wrong. My second was that reranking just doesn't help on code, which is a comfortable thing to believe because it means you can stop. Both were wrong, and the actual reason was sitting in a variable name I had read past a dozen times.

Checking whether it was real

Before digging I ran the same comparison on a second repository, because one gold set of 42 queries is not much and I wanted to know if I was chasing noise.

Same direction both times, and the second one was worse — five and a half points of Pass@1 gone. So it was real, and it was systematic rather than a quirk of one corpus.

The two letters

The tool I use has a --rerank flag and no way to pick a model, so I went looking for what it loads.

impl NeuralReranker {
    pub fn new() -> Result<Self> {
        Self::with_model(RerankerModel::JINARerankerV1TurboEn)
    }
}

JINARerankerV1TurboEn. The model directory on disk said the same thing: models--jinaai--jina-reranker-v1-turbo-en.

Every query in my gold sets is a Korean issue title. I had been feeding Korean text to a cross-encoder trained on English and then concluding things about reranking in general.

This is embarrassing in a specific way. I hadn't picked that model — it was the library default, and defaults are invisible. I read the flag as "turn on reranking," not as "turn on this particular English-only cross-encoder," and nothing in the output ever said which model was running.

Swapping it

The embedding library underneath already ships two multilingual rerankers, so this turned out to be a one-line change plus a rebuild.

let model = match std::env::var("CODESEARCH_RERANKER").as_deref() {
    Ok("bge-m3")  => RerankerModel::BGERerankerV2M3,
    Ok("jina-v2") => RerankerModel::JINARerankerV2BaseMultiligual,
    _             => RerankerModel::JINARerankerV1TurboEn,
};

Then I ran all three against the same 42 queries.

The language mismatch was worth 4.8 points of Pass@1. Going from the English-only model to bge-reranker-v2-m3 recovered all of it and then some.

But look at the middle column before concluding that reranking won. Pass@5 was 45.2% with no reranker at all — higher than every reranked configuration including the best one. So the honest summary is not "reranking helps once you fix the language." It's:

• with the wrong language, reranking loses on every metric
• with the right language, it wins on Pass@1 and MRR and still loses on Pass@5
• and Pass@10 never moves regardless

That is a mixed result, and I'd rather report it as one. What the language fix bought me was not a working reranker — it was the ability to see what reranking actually does here, instead of measuring a model that couldn't read the input.

The part I don't love

Now the last column. The 2.4 points of Pass@1 that bge-reranker-v2-m3 gained over no reranking cost 15.6× in wall clock — 33 seconds became 514, about twelve seconds per query. That's fine for a nightly job and impossible for anything a person waits on.

I would not take that trade in an interactive search box, and given the Pass@5 result I'm not confident I'd take it in a batch job either. What I would take is knowing the number, instead of assuming reranking is free accuracy the way most write-ups imply. The flag stays off by default in my setup; the change I actually shipped was making sure that if someone does turn it on, they don't get a model that can't read their queries.

There's also a quieter result in that table: Pass@10 was 47.6% in all four runs, unchanged to the decimal — the same 47.6% I could not move with any reranker. Reranking reorders a candidate set and never fetches new candidates, so recall cannot move no matter which model you use. Whatever the language mismatch was costing me, it wasn't costing me recall.

The same mistake, one layer down

Once I'd seen it in the reranker I went back to look at the embedding model, because that's the other place a language assumption can hide. I'd benchmarked several models earlier against 56 queries from the same corpus:

The bottom two are English-only. They are not "somewhat worse" — they are near-random. A retrieval system that answers correctly 3.6% of the time is not doing retrieval.

And a multilingual model is not automatically the answer either. paraphrase-multilingual-MiniLM handles Korean fine but is trained for sentence similarity, which is a symmetric task: two sentences, how alike are they. Search is asymmetric — a short question against a long document — and models trained for it, like the E5 family, behave differently on exactly this. Multilingual gets you into the room. Trained-for-retrieval gets you a result.

Why the papers didn't warn me

While debugging this I read through the recent issue-localization literature. SweRank reports Acc@10 of 82.12% on SWE-Bench-Lite with a retrieve-then-rerank setup — a number so far above mine that I initially assumed I had a broken pipeline rather than a different problem.

I did have a different problem. SWE-Bench issues are written in English against English-commented codebases. Every retrieval benchmark I could find is monolingual English, so the failure mode I hit cannot appear in any of them. The literature isn't wrong; it just answers a question I wasn't asking.

That gap is worth naming, because "reranking improves localization" is now repeated as settled. It's settled in English. If your issues are in Korean, Japanese, German, or Portuguese, and your code is in English identifiers with comments in your own language, you are running a cross-lingual retrieval problem that most published results have never touched.

How this was measured

Three details matter for reading the numbers above, and all three are places where I've seen measurements go quietly wrong.

The queries are real. Each one is the title a person wrote on a tracker ticket when something was broken. The ground truth is the set of files changed by the commit that closed that ticket. I tried building the gold set from commit titles first and abandoned it — most commits in these repositories are merge messages like Merged in feature/x (pull request #214), which contain no description of the problem and no Korean at all. Using them would have measured how well the system matches branch names.

The index is the real one. Not a copy staged for benchmarking. A benchmark copy diverges from production the moment anyone touches the ignore rules or the chunking, and then you are tuning something nobody uses. Running against the served index costs nothing extra and removes an entire category of doubt.

The runs are cold in the ways that matter. The embedding cache is keyed by content hash and does not distinguish backends, so switching models without clearing it silently reuses the previous model's vectors. I lost an afternoon to that once — two "different" models produced byte-identical results and I briefly believed I'd discovered something profound about model equivalence.

If you want to reproduce this shape of measurement on your own corpus, the whole thing is roughly:

# build a gold set from closed tickets: query = ticket title, answer = files it changed
node build-goldset.mjs <repo>
 
# Pass@k and MRR against the index you actually serve
node measure-quality.mjs <repo>
 
# same gold set, reranker on and off, one model per run
CODESEARCH_RERANKER=bge-m3 node measure-rerank.mjs <repo>

Forty-two queries is small. It's enough to catch a 4.8-point effect that reproduces on a second corpus, and not enough to argue about half a point. I've tried to keep the claims here inside what that sample can carry.

What I check now

Before trusting any number from a retrieval pipeline, I go through the models one at a time and ask what language each was trained on:

# every model in the path, not just the one you configured
grep -rn "RerankerModel::\|EmbeddingModel::" src/
ls ~/.codesearch/models/

That second command is the useful one. The directory names carry the truth — jina-reranker-v1-turbo-en says what it is right there in the path, and I'd had it on disk for weeks.

Three things I'd tell myself a month ago:

Defaults are decisions someone else made for a different corpus. A flag named --rerank reads like a capability. It's a model choice, and the choice was made by someone who probably had English data.

Test the swap, not the feature. "Does reranking help" was the wrong question and it gave me a wrong answer twice. "Does this reranker help these queries" is the question that has an answer.

Near-random is a fingerprint. When a component scores 3.6% instead of 25%, that isn't a tuning problem. Something in the stack cannot read your input at all, and no amount of parameter adjustment will change it.

Frequently Asked Questions

Which multilingual reranker should I use

In my measurements bge-reranker-v2-m3 beat jina-reranker-v2-base-multilingual on both Pass@1 and MRR, and it was the only configuration that improved on no reranking at all. But it was also the slowest of the four, so measure on your own queries before committing — the gap between the two multilingual models was smaller than the gap between either of them and the English-only default.

Is a multilingual embedding model enough

Not by itself. The model also has to be trained for retrieval rather than sentence similarity. A symmetric similarity model handles your language fine and still ranks poorly, because matching a short query to a long passage is a different task from comparing two sentences.

How much did the language mismatch actually cost

4.8 points of Pass@1 on 42 queries, and 5.4 points on a second set of 56. Recall did not change at all — reranking cannot affect recall regardless of which model runs.

Should I turn reranking on at all

Only if ranking is your bottleneck rather than recall. Check whether the correct answer is in your candidate set and simply ranked too low. If it isn't in the set, reranking has nothing to work with, and you should be looking at chunking, the embedding model, or what you allow into the index.