slowbench
RetrievalInferenceEvalsAgentsFindingsSeriesBenchmarksArchive
Retrieval4 min777 words

The scores spread out 15 times wider and the ranking did not move

Embedding anisotropy: vectors sit in a narrow cone, so unrelated documents score 0.90 against anything. Centering fixed that exactly as advertised, and rank 1 went down by two.

Contents · 5 sections
  1. 1The diagnosis has a name
  2. 2The prescription, and what it did
  3. 3Why the fix could not have helped
  4. 4What this does and does not license
  5. 5The shape worth keeping

Every document in an index scored between 0.90 and 0.93 against a query, whatever the query was. Top result 0.9277, bottom result 0.9021, twenty-nine documents in a band 0.0256 wide.

That band contains a hotel scraper, a payments admin tool, and a publishing pipeline, ranked against a query about a tagging screen. None of them is the answer. All of them score above 0.90.

The diagnosis has a name

This is anisotropy. Embeddings from many models occupy a narrow cone of the vector space rather than spreading over the sphere, so cosine similarity between any two of them starts high and the useful signal lives in the last few decimal places.

It is measurable directly. Every vector this model produces is unit length, so if they pointed in genuinely different directions their mean would be short. Across 23 separate indexes:

‖mean vector‖   0.926 to 0.945

The mean of a few thousand unit vectors is almost as long as the vectors themselves. They are all pointing the same way.

The means of different indexes agree with each other too. Pairwise cosine between one index's mean and another's: minimum 0.945, median 0.982, maximum 0.999. These indexes hold completely different codebases, and their centres of mass are nearly the same direction. That is a property of the model, not of what was indexed.

The prescription, and what it did

The standard treatment is whitening: subtract the mean, then normalise the covariance. Centering alone is the first half.

It worked. The band went from 0.0256 wide to 0.3976, a factor of 15.5. Scores that had been crowded into two decimal places now use most of the range.

Then, on a 90-query gold set against a frozen copy of the index, scoring all 29 documents rather than reranking a shortlist:

centeringrank 1top 3top 5top 12MRR
off49/9070/9085/9089/900.6949
on47/9071/9085/9089/900.6848

Sixteen queries changed position at all. Seven improved, nine got worse.

The treatment did precisely what the diagnosis said it would. The diagnosis was correct and independently measurable. The spread was real, the fix produced the spread, and the ordering came out the same or slightly worse.

Why the fix could not have helped

The retrieval this feeds ranks by fusion, and reciprocal rank fusion consumes ranks. It never reads a score. Spreading the scores out changes nothing about what the fusion sees, because the fusion was already looking only at the ordering, and centering barely changed the ordering.

The thing anisotropy actually breaks is anything that reads a score as an absolute quantity. A threshold, for instance: this system has a shortcut that skips half of retrieval when the top results are "similar enough," expressed as a cosine cutoff. In a space where unrelated documents score 0.90, a cutoff of 0.85 is not a filter. It is a constant.

So the measurement was pointed at the wrong consequence. Compressed scores were never hurting the ranking. They were hurting the code that treated those scores as meaning something on their own, and no amount of improving rank 1 would have surfaced that.

What this does and does not license

It says centering produces no ranking improvement for this model on this corpus at this index size. That is the whole claim.

It does not say whitening is bad. Centering is one endpoint of a family with a regularisation parameter, and the rest of that family is untested here. It does not say anisotropy does not matter, only that it does not matter to this consumer.

There is published work reporting that standard whitening degrades ranking for contrastively fine-tuned models, which is the family this model belongs to. My result points the same way. I am deliberately not treating that agreement as support, because agreeing with a prediction is the exact condition under which a broken measurement survives review. Earlier in this same investigation a result matched a paper's expectation and was wrong for an unrelated reason, and the match is what stopped anyone looking.

The shape worth keeping

A correct diagnosis, a correct treatment, and no improvement is not a contradiction. It means the thing that was broken and the thing that was being measured were different things.

The useful question after a null result is not whether the fix worked. It is which consumer was supposed to benefit, and whether that consumer ever read the quantity you fixed.