I added a reranker because everyone adds a reranker. Then I swapped it three more times, and the number I actually cared about never moved once.
Pass@1 went from 23.8% to 28.6% depending on which model I used. Pass@10 was 47.6%. Every time. All four configurations, identical to the decimal. I checked the script twice because a number that stable usually means you measured the same thing twice by accident.
I hadn't. It's just what reranking does.
What I was measuring
Queries are real issue titles — the kind a person types into a tracker when something is broken. Ground truth is whatever files the closing commit touched. Forty-two of them, run against the index that actually serves search rather than a copy made for the benchmark, because I've been burned by benchmark copies before.
Retrieval is hybrid, dense plus BM25, fused with RRF. The reranker gets the top 50 and reorders them. The candidate set is the same in all four runs — the only thing that changes between rows is the model doing the reordering.
| Reranker | Pass@1 | Pass@5 | Pass@10 | MRR | Wall clock |
|---|---|---|---|---|---|
| none | 26.2% | 45.2% | 47.6% | 0.333 | 33s |
| jina-reranker-v1-turbo-en | 23.8% | 42.9% | 47.6% | 0.326 | 54s |
| jina-reranker-v2-multilingual | 26.2% | 42.9% | 47.6% | 0.330 | 282s |
| bge-reranker-v2-m3 | 28.6% | 42.9% | 47.6% | 0.352 | 514s |
That fourth column is the whole post.
First, these are counts wearing percentage costumes
Before I draw any conclusion from that table I should put the denominator back, because at n=42 a percentage is a rhetorical device more than a measurement.
| Reranker | Pass@1 | Pass@5 | Pass@10 |
|---|---|---|---|
| none | 11/42 | 19/42 | 20/42 |
| jina-reranker-v1-turbo-en | 10/42 | 18/42 | 20/42 |
| jina-reranker-v2-multilingual | 11/42 | 18/42 | 20/42 |
| bge-reranker-v2-m3 | 12/42 | 18/42 | 20/42 |
The entire spread across four models is ten, eleven, eleven, twelve.
"bge-reranker-v2-m3 improves Pass@1 by 2.4 points" means it got one more query right. One. And it gave back one at Pass@5 to buy that. I wrote the percentage sentence first and it read like a finding; the fraction reads like noise, which is nearer the truth. My rule now is that any difference smaller than 1/n gets written as a fraction before it gets written as a claim.
The Pass@10 column is different. Twenty out of forty-two, four times, with four different models. That is not noise. That is a wall.
The reason is boring, which is why it's easy to miss
A reranker takes a list and returns the same list in a different order. It has no way to go back and ask for more. Fifty documents go in, fifty documents come out.
So if the file you needed isn't in those fifty, nothing downstream can save you. Pass@10 asks whether the answer landed somewhere in the top ten, and shuffling fifty items can't move something into the top ten when it was never among the fifty.
Stated that way it sounds obvious. It didn't feel obvious while I was reading posts about which reranker to pick.
What reranking fixes is the case where the answer sits at rank seven and deserves rank one. That's real, and it's what Pass@1 and MRR measure. What it cannot fix is the answer that was never retrieved at all — and that's recall, which stayed frozen at 20/42 no matter what I put on top.
So when a post says reranking "improves retrieval quality," it's worth asking which half of the problem the author had. Mine was the other half.
How to tell which half you have
The diagnostic is one number: how often the answer appears anywhere in the candidate set you hand the reranker. Not in the top ten — anywhere in the fifty. That number is the ceiling on everything that runs after retrieval, and no amount of reordering can raise it.
Here is where I have to separate what I measured from what I inferred.
Measured: Pass@10 was 20/42 in all four runs, to the decimal.
Inferred: the candidate set beyond rank ten contained essentially nothing useful. I did not log recall@50 as its own number, which I now regret, because it is the single most informative number in the whole experiment and it would have cost me one line of script. But four different models — including two multilingual ones that disagree with each other about almost everything else — reordering the same fifty candidates and landing on the identical top-ten hit count is hard to explain if there was something down at rank thirty waiting to be pulled up. At least one of them would have found it.
Taking that at face value, for twenty-two of my forty-two queries the reranker was carefully sorting fifty documents, none of which was the answer. It cannot win those. It never could. Its entire working surface was the other twenty, and on those the answer was already inside the top ten before it ran.
That reframes the 2.4-point swing completely. It isn't a small improvement to my search. It's a small reshuffle inside the minority of queries that were already working.
The practical version: if recall at your candidate size is 90% and Pass@1 is 40%, you have a ranking problem and a reranker is the right tool. If they are both 47.6%, you have a retrieval problem wearing a ranking problem's clothes, and you will spend weeks swapping models that cannot reach it.
The price
The best model here bought me one query and cost 15.6× in wall clock. Thirty-three seconds became five hundred and fourteen.
Per query that's 12.2 seconds. The reason is structural, not an implementation detail I could tune away: a cross-encoder doesn't embed documents, it scores query-document pairs, one forward pass each. Fifty candidates times forty-two queries is 2,100 forward passes through a model that isn't small, and none of them can be cached between queries the way a document embedding can. Halve the candidate set and you halve the cost, but you also lower the ceiling you were trying to raise.
Twelve seconds is fine for something that runs overnight. It is absurd for a search box where somebody is watching a spinner.
There's a second story hiding in that table — the default reranker was English-only and every query I have is Korean, which is why the first swap made things worse before they got better. That one needs its own post.
The experiment I should have run
Not a fifth model. A wider window.
Fifty was a default I inherited and never questioned. The one knob in this whole area that can actually move 20/42 is how many candidates retrieval hands over: at 100, or 200, more of the missing answers come into range, and then the reranker has something to do. It costs linearly in reranking time, which is exactly the cost I was already paying without getting anything for it.
I didn't measure that, and I'd rather say so than imply otherwise. What I can say is that I spent four model swaps on the one knob that provably cannot change the number I cared about, while the knob that can sat untouched at its default the entire time. That is the actual mistake in this post, and it took writing the results down in a table to see it.
Where recall actually comes from
If recall is your problem, the reranker is the wrong place to be standing. Recall is decided earlier: how you chunk, which embedding model you use, how you weight the hybrid, how many candidates you keep, and what you let into the index in the first place.
The biggest win I've had came from that last one, and it was embarrassingly cheap. A code index had prose documentation sitting in it — 3% of the chunks, describing screens in the same language the issue titles were written in. Those docs were eating the top five results on almost every query. Excluding them took one line in an ignore file and moved MRR from 0.142 to 0.333.
That single line was worth more than every reranker in the table combined, and it ran in the same 33 seconds as before.
So: measure recall at your candidate size before you touch anything that runs after retrieval. If the answer isn't in the candidate set, everything downstream is decoration.
Frequently Asked Questions
Can reranking improve recall
No. It reorders what retrieval already handed it. A document outside that candidate set does not exist as far as the reranker is concerned.
Then is reranking useless
Not at all — it fixes ranking, which is a real problem when the right answer is buried at position seven. Just check that ranking is what's actually hurting you. Mine wasn't.
Isn't a 2.4-point gain still a gain
At forty-two queries, 2.4 points is one query. Run the same benchmark on a different forty-two and I'd expect that one to move around. Treat any difference below 1/n as unmeasured rather than small.
Isn't it suspicious that Pass@10 was identical four times
It falls out of the architecture. The reranker receives the top 50 and returns the same 50 reordered, so which documents sit in the top ten can change while "is the answer in the top ten at all" cannot — provided nothing useful was sitting between rank eleven and fifty. That last clause is an inference from the four identical runs, not something I measured directly.
How did you build the gold sets
Each query is an issue title and the answer is the files its closing commit changed. I tried using commit titles as queries first and it went nowhere — most of them are merge messages that say nothing about the actual problem.