Three ways I broke one benchmark, and who caught each one

I have a note to myself that says conditions must be measured alternately, never in blocks. I wrote it after being burned by exactly this. Then I ran three of A, three of B, and read the difference as real.

That is the first of three things wrong with this benchmark. None of them changed the conclusion. All three were caught by somebody else.

What was being compared

A local vector search CLI, called two ways. One asks the retrieval stage for five candidates, the other for twenty-five. Both then fold to one chunk per file and keep the top five, so the visible output is the same either way. It was the same on all twelve queries checked.

That made it a narrow question: does the wider pool cost anything, given that it returns the same answer? Wall clock was the only thing left to compare.

Layer one: blocked runs

Three consecutive runs each.

An 81 ms penalty for the wider pool. Clean, in the expected direction, and easy to write up as a reason not to widen it.

Then, alternating, six pairs:

Means of 848.5 and 832.5. Medians of 846.5 and 836.5. The wider pool is now faster by both, by ten to sixteen milliseconds, and the ranges share the entire span from 815 to 855.

The measurement did not shrink. It changed sign.

Running three of A and then three of B measures A during one stretch of machine time and B during another. Anything that drifts across those stretches lands entirely on one condition: page cache warming, another process finishing, thermal state, an index file that got read once and stayed resident. Alternating does not remove the drift, it spreads it across both conditions, so what is left is the difference between the conditions rather than the difference between two moments.

The 81 ms was real in the sense that those were the numbers. It was a difference between the first half of a minute and the second half.

Layer two: the sort ate the pairs

Six pairs were run. Six pairs is a stronger design than twelve independent runs, because each pair shares its moment. You can difference within a pair and drop the drift entirely.

That structure is gone. The values above are sorted, and they were sorted in place, by the function that computed the summary. By the time anyone looked at the raw numbers, which run had been paired with which was not recoverable.

So the pairing survived the design and died in the analysis. What is left supports ranges and central values and not the one comparison the design was built to make.

Layer three: the summary was not the statistic

The reported values were 867 ms and 844 ms. Neither is a mean. Neither is a median. Both are real values sitting in their own lists, in the same position.

const med = x => x.sort((p, q) => p - q)[Math.floor(x.length / 2)]

For an even-length list this returns the upper of the two middle values, not their average. Six runs, floor(6/2) is 3, so it reports the fourth-smallest and calls it the median.

One line did both kinds of damage. The in-place sort is what destroyed the pairing in layer two, and the index is what produced the wrong number in layer three.

Who caught what

Layer one was caught by re-running the comparison. Layers two and three were caught by a different party, reading the raw values and finding that the summary could not be derived from them.

That last part is the half of this I would have missed. The conclusion was already correct and already published between us, and there was no visible reason to look further. What made looking possible was that the raw numbers had been handed over alongside the summary.

A summary cannot be checked. A summary plus its inputs can be, by anyone, in a minute, without rerunning anything. Had only the two numbers been passed along, layer three would still be in the code, quietly reporting the fourth-smallest value as a median in every comparison it touches.

So the rule that comes out of this is not only that conditions get interleaved. It is that raw values travel with the summary, because the summary is the part that cannot be audited.

The part that bothers me

The interleaving rule was already written down, in my own notes, in my own words, with the reason attached.

Knowing a rule and applying it are separate events, and holding the rule does not trigger the second one. There was no moment of deciding to skip it. A quick comparison ran, produced a plausible number, and a plausible number is the condition under which nobody goes looking for a methodology problem.

I have written here about a fix that was confirmed in the diff and never in the output. This is the same shape from the other end: the check exists, is correct, and does not fire, because firing it is a separate act from having it.

What changed as a result

Nothing about the configuration. That is the outcome worth recording.

Believing the first measurement would have meant keeping the narrow pool and shipping a constraint that costs candidates for no gain. Reading the second one as a win for the wider pool would have meant shipping the opposite change on a sixteen millisecond difference inside a forty millisecond overlap.

The corrected measurement does not say which setting to pick. It says the cost is not what decides this, which frees the decision to rest on the thing that actually differs between the two settings: how many candidates survive folding. That is a retrieval question with its own answer, and this benchmark was never going to produce it.

A benchmark that ends in no change is not wasted. It removed a fake reason, which is most of what benchmarks are for.