My benchmark had one slot left, so every change measured zero

A retrieval change measured as no effect. I ran it on the standing evaluation set, top-5 hit rate before and after, and the number did not move.

The set was scoring 28 out of 29 before the change. There was one point available.

Whatever the change did, the most it could have shown on that metric was plus one, and plus one on 29 queries is indistinguishable from a query that flipped for its own reasons. The measurement was not saying the change did nothing. It was saying that this instrument had one graduation left on it.

The same change on a 90-query set, where top-5 was 85 out of 90, scored +4.

Headroom is a property of the metric, not the system

Here is the same 90-query set read at three depths.

One system, three numbers, and the amount a change could possibly move them differs by an order of magnitude. Choosing top-5 as the headline is choosing to work with five available points. On the smaller set it was choosing to work with one.

This is not a subtle statistical effect. It is arithmetic that anyone can do before running anything, and I had never done it. The evaluation set was chosen because it was the one with labels, and the metric was chosen because it was the one already in the report.

The obvious fix is wrong

If headroom is what matters, take rank 1. It has 49 slots free, nearly ten times what top-5 offers, and any real improvement has room to appear.

Except nothing downstream consumes rank 1.

The router hands a set of candidate repositories to an agent, which searches all of them. Moving the correct answer from position three to position one changes no behavior anywhere in the pipeline. A change that improves rank 1 and leaves the top-3 membership alone is, from the perspective of the system that exists, no change at all.

So rank 1 has room and measures something nobody uses. Top-5 measures something everyone uses and has no room. Both are bad choices, for opposite reasons, and picking either one on a single criterion produces a report that is wrong in a way the report cannot show.

Two conditions, not one

The metric has to satisfy both:

• it has slots left, so an improvement has somewhere to appear
• the thing it measures is what the consumer actually depends on

For this pipeline that intersection is top-3: fourteen slots free, and the candidate set the agent receives is genuinely bounded near there. It is not the metric with the most headroom and it is not the metric closest to the user's experience. It is the one that is both, adequately.

Writing that down makes an uncomfortable thing explicit. There may be no metric in the intersection. A mature system whose consumer-facing metric is at ceiling has no honest way to show incremental improvement on that metric, and the correct response is to say so rather than to reach for a looser one that still has room. A number that can move is not the same as a number worth moving.

What it changed

The colleague running the routing measurements applied this to their next experiment before I had written any of it down. They counted headroom first, on the set they were about to use, and reported rank 1 and top 3 as the primary numbers with top-5 kept as context rather than as the result.

That is the part I want to record, because it is the rare version. Most measurement lessons get written up and then the next experiment is run the old way, because the lesson lives in a document and the experiment lives in a shell. This one changed the shape of the next measurement in the same week, before the writeup existed.

The cost of not doing it is quiet. A change that works scores zero, gets abandoned, and nothing in the output indicates that the instrument was full. I have that exact case in hand: a retrieval change that read as worthless against a set with one slot left, and as +4 against a set with five.

Count the slots before you run the comparison. It takes one subtraction, and it tells you whether the answer you are about to get is capable of being the answer you want.