Measure the index you actually serve

The benchmark harness I inherited made a copy of the repository, indexed the copy, and measured against that. Sensible-looking: benchmarks should not touch production, and a copy is reproducible.

I switched to measuring the index that actually serves search, and it removed a class of question I had been quietly carrying.

What a copy stops representing

The copy was made once. Everything after that diverges:

Ignore rules. I excluded 89 documentation files from the production index and MRR more than doubled. The benchmark copy still had them. Any measurement taken there was of a system nobody was using.

Chunk boundaries. Chunking changes with the indexer version. Two indexes built by different builds of the same tool split files differently, and the numbers are not comparable across that boundary.

Content. The copy was a snapshot; the repository kept moving. After a few weeks the benchmark was scoring a system's ability to find files that had been renamed.

Each of those is individually manageable if you remember to re-sync. The problem is that nothing tells you when they have drifted. The harness runs, produces numbers, and the numbers look exactly the same whether the copy is current or four months stale.

The version that reads the live index

Measuring the served index turns out to be simpler than maintaining a copy:

const root = path.join(os.homedir(), "repos", repo);
 
for (const g of gold) {
  const r = spawnSync("codesearch",
    ["search", g.query, "--path", root, "-m", "10", "--json", "-q"],
    { encoding: "utf-8" });
  const hits = JSON.parse(r.stdout).results.map((x) => x.path.replace(root + "/", ""));
  // score against g.gold
}

No copy, no staging, no sync step. It runs the same query path a user runs, against the same index a user searches.

Reading an index while it serves is safe here because the storage layer supports concurrent readers, which is the common case for embedded vector stores. Worth confirming for yours, but it is usually true, and a read-only benchmark is exactly the workload those systems are designed for.

What it caught

Three things I would not have seen otherwise.

An incremental reindex that did nothing. I added an ignore rule and rebuilt. The tool reported success. The scores did not move at all, because excluded files are undiscovered, not deleted, and the 89 stale entries were still in the index. Against a fresh copy this would have looked like the change worked.

A silent model mismatch. One repository was still on the old embedding model after a migration I thought had finished. Searching it returned nothing useful, and the only reason I noticed was that its numbers were bizarre while its siblings were normal. A benchmark copy would have been built fresh with the current model and scored fine.

The docs problem itself. The 89-file exclusion was found by printing the top five results from a live query. Metrics said one repository was worse than its siblings; the actual result list said why. That investigation only makes sense against the index people search.

The simulation that predicted itself

One thing measuring live made possible: testing a change before making it.

I suspected documentation files were crowding out code, but confirming it meant dropping an index and rebuilding: twenty minutes, and a rollback if I was wrong. Instead I ran the gold set normally and filtered documentation out of the results before scoring:

let hits = JSON.parse(out).results.map((x) => x.path);
if (dropDocs) hits = hits.filter((h) => !h.includes("docs-content"));
hits = hits.slice(0, 10);

The simulation predicted 26.2 / 45.2 / 47.6 / 0.333. The actual rebuild afterwards produced 26.2 / 45.2 / 47.6 / 0.333, identical to three decimals.

This only works against the live index. Filtering the results of a stale copy predicts what would happen to the stale copy, which is a question nobody asked. And it turns an expensive irreversible experiment into a cheap reversible one: you rebuild once, knowing the answer, rather than twice, hoping.

I use the same trick now for anything expressible as "what if these files were not in the index." Not everything is (a chunking change is not simulable this way) but exclusions are, and exclusions turned out to be where the largest wins were.

What I record with every run

The metadata line matters more than it sounds, so here is the whole thing:

repo         admin-app
model        multilingual-e5-small (384 dims)
chunks       25,079
files        1,123
indexed      2026-09-04 07:41
queries      42 (gold set v2)

Three of the mistakes in this project would have been caught by one of those fields.

The model line would have caught the repository stuck on an old embedding model. The chunk count would have caught the incremental reindex that changed nothing: same count before and after a change that should have removed a thousand chunks. The index timestamp would have caught a run scored against an index built before the change I was trying to measure.

None of that is analysis. It is bookkeeping, and it is the difference between "the number moved" and "the number moved and here is what else was different."

The reproducibility objection

The obvious counter: a live index changes, so results are not reproducible.

Two responses.

A stale copy is not reproducible either. It is just wrong consistently. Reproducing a number that describes a system nobody runs is not a property worth protecting.

Pin what you can name. Record the model, the chunk count, and the index build time alongside the score:

repo         admin-app
model        multilingual-e5-small (384d)
chunks       25,079
indexed      2026-09-04 07:41
Pass@1 26.2%  Pass@5 45.2%  Pass@10 47.6%  MRR 0.333

That is enough to know whether two runs are comparable, and if they are not, the metadata tells you why. A chunk count that moved by 1,030 between runs is a much more useful signal than a score that silently drifted.

Where a copy is still right

I am not arguing against isolated environments generally, and there is one case where a copy is mandatory.

When the system under test can read the answer. For agent benchmarks, the agent has a file-reading tool and the finished work is often merged into the checkout. There you need an isolated clone at an older commit, or you are measuring the agent's ability to find its own answer key. That is a different problem from retrieval benchmarking and it has the opposite conclusion.

When you are comparing tool versions. Two builds that chunk differently need the same input to be comparable, which means a fixed corpus that neither of them can change.

The distinction is what varies. For "is this index good," measure the index. For "is this model better than that model," fix everything except the model, which usually means a copy.

What this does not fix

Measuring the live index removes doubt about which system you measured. It says nothing about whether the questions are any good, and I want to be clear that the second problem is larger.

My gold set is 120 queries across three repositories, mined from closed tickets. Measuring it against the live index tells me the score describes the thing people search. It does not tell me that 120 tickets represent the queries people actually type, that the labels are current, or that the sample is balanced across the kinds of question the system will face.

All three of those turned out to be wrong at various points. The labels were past tense while the question was present tense. One sample was drawn in a way that collapsed nineteen of twenty cases onto the same answer. Neither problem is visible from the index side at all.

So the honest ordering is: the questions matter more than the corpus, and the corpus is easier to fix. Measuring live is a cheap win that removes one source of error. It is not the expensive work, and doing it first can create a false sense that the measurement is now trustworthy.

I did exactly that for a while, pleased with a clean harness pointed at the real index, producing numbers from a sample that could not distinguish a working system from a constant.

The cheap habit that came out of it

Alongside every score, the harness now prints what it measured:

repo · model · dimensions · chunk count · index timestamp

Five fields, one line. It is the thing that turns "the number went down" into "the number went down and also the chunk count dropped by a thousand, which is the ignore rule I added this morning."

Most of the mistakes in this project were not wrong analysis. They were correct analysis of something other than what I thought I was looking at, and a line of metadata is the cheapest defence against that I have found.

Serving an index is not the same as maintaining one

Measuring the live index catches the difference between a copy and the thing. It does not catch the thing quietly going stale, and I found out how far that can go.

One of my indexes had last been built three weeks earlier. Fifty-four of the eighty-one files were in it; the twenty-seven added since were not. Nothing had errored. Search worked, returned plausible results, and answered every query from a three-week-old snapshot of the corpus.

Worse, it was still on the previous embedding model. Everything else had been migrated months ago and this one had been missed, so it was answering in a vector space nothing else shared. A separate index in the same setup had the same problem and I had spent weeks reading its behaviour as this collection just ranks badly. It was returning scores from a different scale entirely, and I had been comparing them to the others as though they were the same number.

Neither of those is a benchmarking mistake. They are maintenance mistakes that a benchmark reads as quality.

The fix that stuck was making freshness impossible to skip rather than scheduled: the search command reindexes before it searches. Incremental reindex costs 0.03 seconds when nothing changed, which is small enough that there is no argument for deferring it, and it removes the failure mode entirely: you cannot query a stale index because there is no code path that leaves it stale.

The general form, which I did not have when I wrote the rest of this post: measure the index you serve, and make sure the index you serve is the corpus you have. Those are two different checks, and the second one has no gold set to tell you it failed.