A perfect title match had to be popular to count

Some queries in our customer-facing travel product search returned nothing. They were short: a single place name, no other words. When I ran the lexical index by itself, the right product came back at rank one with a comfortable score. The final result was still empty.

I traced one query all the way down. There were four separate defects stacked on it, and each one produced the same empty page.

Three of those were bugs. Layer 3 was arithmetic.

ceiling for a document found only by lexical search
  = the lexical fusion weight itself                    0.35
 
retrieval contribution = retrieval_weight × hybrid
  = 0.58 × 0.35                                         0.2030
 
candidate cutoff (min_candidate_score)                  0.25
 
shortfall the metadata bonuses have to cover            0.0470

A document whose title matched the query exactly, with the lexical component at a full 1.0, cannot reach the cutoff on retrieval alone if it is absent from the vector top 50. Improving retrieval would never have moved it, because the highest score that path can produce is 0.047 short of the number it has to beat.

It is not quite a wall, and the exact shape matters. Score is retrieval_weight × hybrid + metadata bonuses, and those bonuses sum to at most 0.48 across theme, season, companion, region, product type, duration, mobility, and popularity. Any one of several could cover a 0.047 gap.

But these queries are bare place names. No theme, no season, no companion, no duration is recognised in a two-syllable string, so every one of those bonuses is zero. I printed the breakdown for three of the traced documents and only one component was ever non-zero:

query      hybrid   popularity   season   theme   companion   region   type   total bonus
A          0.4500   0.0293       0        0       0           0        0      0.0293
B          0.4500   0.0294       0        0       0           0        0      0.0294
C          0.4500   0.0000       0        0       0           0        0      0.0000

The hybrid column there is from the equal-weights run; bonuses are added after fusion and do not move with the weights, which is why the same breakdown explains the failure under the original 0.55 / 0.35.

So for this class of query the only thing that can bridge 0.047 is the popularity bonus, whose weight is 0.06. A lexical-only exact match clears the cutoff if and only if its normalised popularity is at least 0.047 / 0.06 = 0.783. The three above sat at roughly 0.49, 0.49, and 0. They did not clear.

That is the real shape of the defect: a perfect title match was only admissible if the product also happened to be popular.

Why the vector top 50 is only 0.05 wide

This is layer 2, and it is also the reason layer 3 ever mattered. If lexical-only documents appeared in the vector list, they would carry a vector component and clear the cutoff easily. They don't, and the shape of the vector scores explains why.

Cosine similarity across the top 50 candidates sits in a very narrow band. These are the three zero-result queries I traced:

query   max      min      spread   rank 40 after max-normalisation
A       0.3507   0.2570   0.094    0.745
B       0.4137   0.3541   0.060    0.868
C       0.5392   0.4916   0.048    0.921

The fusion normalises each path by dividing by the observed maximum. With a spread that narrow, rank 40 comes out at 0.745 to 0.921. The top 50 become a near-tie.

So the lexical winner, at 0.35 × 1.0 = 0.350, loses to vector rank 40 at 0.55 × 0.745 = 0.410. In all three queries the correct product landed at exactly rank 51 in the merged list, the position immediately after the 50 vector candidates. The pipeline cuts at 40. It was gone before scoring ever ran.

Attempt one: reciprocal rank fusion

Two paths on incompatible score scales looked like the whole problem, so rank-based fusion looked like the answer. I A/B'd it over the top 1,500 real user queries by volume, which cover 93% of all typed search traffic.

Zero-result rate improved. Rank-1 hits fell from 698 to 672.

I opened the 224 queries whose top result changed: 11 better, 37 worse. The regressions had one shape. The query was the product name itself, and that product lost its own name to a different product in the same city.

Inferred, not measured: under rank fusion, a document whose title matches completely and a document that matches partially become rank 1 and rank 2: 1/61 versus 1/62, a difference of 0.0003. The vector weight covers that gap without effort. What I measured is the 37 regressions and their titles. The mechanism is my reading of them.

This contradicts an earlier post, and both are right

An earlier post on this site pulled RRF out of repository routing and concluded:

Which is why I use the code ranker's order and never its scores. Rank position survives the clustering; the raw number does not separate one repository from the next on a given query.

That is the opposite of what I just wrote. The difference is in the score distributions.

that post's ranker    per-repository scores 0.942 – 0.943    spread 0.001
this system's vector  top-50 spread                          0.048 – 0.094

At a spread of one thousandth, the ordering is a coin flip with decimal places, and rank is the only trustworthy signal. At a spread of fifty to ninety thousandths, the numbers carry real separation, and converting them to ranks discards it.

So the rule is neither "use rank" nor "use scores." It is whether your scores separate. Measure the spread of your candidate band before you pick a fusion method. That measurement takes one query and a sort.

There is one more thing worth saying, because that post has a postscript where fixing coverage made fusion win. Its failure came from asymmetric coverage: one ranker simply could not see 11 of 28 repositories, and RRF cannot distinguish "ranked last" from "never seen." My two paths cover the same corpus. Coverage was already symmetric here, and RRF still lost. Symmetric coverage is necessary for rank fusion to behave, not sufficient.

Attempt two: min-max normalisation

If the vector scores are compressed into a narrow band, spread them out. That was the next idea, and the standard fix. I ran three normalisation schemes against six weight pairs, eighteen combinations, on the top 400 queries.

Min-max did restore the discrimination. It also dropped the mean result count from 6.38 to 4.70. Pushing the bottom of each list to zero sends a large share of candidates under the absolute cutoff, and the result sets go thin. I re-ran all eighteen with the cutoffs lowered to compensate, and divide-by-max still won at every setting.

My reading: the compression is not a bug that survived review. It is the counterpart of having an absolute cutoff at all. Vector similarity lives in a narrow band, so dividing by the maximum is what lifts it into the range the cutoff was calibrated against. I did not confirm this with whoever wrote it.

Attempt three: give both paths the same weight

The compression was never the problem. The problem was that lexical and vector were not on the same scale of the same balance. Vector had 0.55, lexical had 0.35, and the cutoff had been calibrated against a number only the vector path could reach.

0.58 × 0.45 = 0.2610  >  0.25

The ceiling clears the floor.

Same query set, same baseline, same run as the RRF arm above, with this weight change as the only toggle. Every other change in the pipeline was present in both arms:

Four queries improved, one regressed. Weighted by search volume the improvements outweigh the regression by roughly twelve to one. Five zero-result queries recovered, none appeared.

The zero-result column is identical for RRF and for equal weights: same 193 queries, same 4.24%, same five recovered queries. That is not a transcription slip; I checked both lists. The two changes get there by different routes, and I traced one recovered document through both:

                       hybrid   × 0.58    + popularity bonus vs cutoff 0.25
0.55 / 0.35            0.3500   0.2030    0.2323             below
reciprocal rank fusion 0.3889   0.2256    0.2549             above by 0.0049
0.45 / 0.45            0.4500   0.2610    0.2903             above by 0.0403

Equal weights raise the lexical ceiling directly. RRF replaces the score scale, and a lexical rank-1 document lands at (0.35/61) ÷ (0.90/61) = 0.3889 instead of 0.35. Different mechanisms, same quantity moved, so the set that escapes zero results comes out the same and only the order differs.

The middle row is worth staring at. Under RRF that document cleared by 0.0049, and the popularity bonus of 0.0293 is what carried it. Without the bonus, 0.2256 fails. Reconstructed, not measured: the first and third rows are live numbers from the running pipeline; the RRF row I recomputed from the fusion formula after reverting it, so treat 0.2549 as arithmetic rather than an observation.

The number I would look at first if I were reviewing this is not in that table. Only 43 queries changed their top result at all, 1.4% of search volume. RRF had moved 8.1%. A change that fixes a structural defect should be surgical. When a ranking change shuffles eight percent of your traffic, it is doing something other than what you asked.

What this measurement cannot see

The metric is whether every word of the query appears in the title, region, or country of the top result. That is a string test, and it is blind to popularity.

I ran a grid over the popularity bonus weight. Lower was monotonically better, and zero was best: rank-1 hits went from 285 to 288 out of 400 as the bonus went from 0.10 to 0.00. Following that would mean deleting a signal my metric is structurally unable to credit: of two hotels in the same city, putting the one people actually book on top scores zero points here.

So layer 4 is still there. On query C above, the exact-title match finished at 0.2610 with no popularity bonus at all, while an unrelated product in the same region finished at 0.2910, the same 0.45 hybrid plus 0.0300 of popularity. Both numbers are final scores; I printed them side by side. What I do not know is what turning that signal off would cost, and I will not know until I have click data rather than string matching.