Same bytes, a different folder, a different score

I had a comparison to redo and I redid it carefully: one variable at a time, four configurations, the same gold set, everything rebuilt fresh. The table that came out was wrong, and it was wrong in a way I had introduced by being careful.

The tell was small. One arm of the comparison would not sit still.

same configuration, measured twice, 90 queries
 
baseline          46 / 80 / 85        46 / 80 / 85     identical, three times
the other arm     44 / 76 / 85        42 / 78 / 86

My first reading was run-to-run noise. Someone reviewing it pointed out the problem with that: if the pipeline were noisy, both arms would move. One arm moving means the noise is inside that arm, not in the measurement.

That is the sentence that made the rest of the day useful, and I would not have got there on my own. I had already decided it was noise and moved on to writing the number down with an error bar, which is what you do when you want to stop thinking about something.

Splitting it in two

Two runs, fifteen queries each, comparing the full ranking string rather than the summary counts. Summary counts can agree while the ordering underneath them differs, and I wanted to see the ordering.

A   same directory, queried twice              0 of 15 differ
B   identical bytes, indexed in another folder  8 of 15 differ

The description files in the two folders are byte-identical. I built them twice from the same inputs and compared with cmp, all twenty-eight of them. The only difference between A and B is which directory the index was built in.

More than half the queries came back in a different order.

It is upstream of the ranking

The scores are exposed per result, so I looked rather than guessed.

same query, same 28 documents, two indexes
 
                   index A      index B
document 1        0.920149     0.920149      identical
document 2        0.917787     0.917780
document 3        0.917176     0.917200
document 4        0.915900     0.915720
document 5        0.915857     0.916047      ← 4 and 5 swap

These are cosine similarities from the vector side, not fused scores, so this is not an artefact of rank fusion. Something upstream of the ranking differs, by around one part in ten thousand.

One part in ten thousand is nothing in most contexts. It is not nothing here. Across all twenty-eight documents the scores for that query span 0.023, but a ranking is decided at the top and the top is compressed: the five above sit inside 0.0043 of each other. That is the same compression I have written about on the code index, where every repository scored inside 0.019 for any given query. A displacement of 0.0002 is a twentieth of the band these five occupy, and two documents a thousandth apart change places.

The hypotheses I killed

In order, cheapest first.

re-index the same folder (delete the database first)   identical to 7 decimal places
GPU backend, two fresh folders                          differs
CPU backend, two fresh folders                          differs
thread counts pinned to 1                               still differs
raw directory read order of the two folders             identical

The first line is the interesting one. Deleting the index and rebuilding it in place reproduces every score exactly, so the pipeline is not simply non-deterministic. Something about being in a different place changes the answer.

Not the backend: both the GPU path and the CPU path show it. Not threaded floating-point reduction, which was my favourite theory for about ten minutes. Pinning the thread counts to one changed nothing. Not the order files are visited: I compared the raw directory listings of both folders and they enumerate identically.

Then I read my own code, which is where it got worse rather than better.

The text handed to the model is assembled from the chunk's heading context, its signature, its docstring, and its content. The file path appears in exactly one place: it selects a label, and for the model I use that label is a constant. The chunk's hash, used for deduplication and caching, is computed from content alone. And the embedding cache is global, living in one directory keyed by content hash, which means identical content anywhere on disk should return the same cached vector.

So I checked whether the cache is actually used. Indexing a fresh copy of the same twenty-eight files logs cache hits and zero computed embeddings for every batch it reports. If the vectors are being served from the cache rather than recomputed, they are the same vectors, which removes the explanation I had been carrying since the first table, and leaves the difference somewhere between the stored vector and the number the search prints.

Every mechanism I can name says the path cannot matter. The measurement says it does. I do not have the reconciliation, and the leading candidate changed while I was writing this paragraph, which is a reasonable summary of the whole investigation.

What this did to the thing I was measuring

Here is the part I would rather not write.

I had an earlier version of this comparison that I considered sloppy: all four configurations built into the same working directory, one after another, overwriting each other. Redoing it "properly" meant giving each configuration its own folder, so that nothing could contaminate anything else.

That decision is what broke it. Each configuration got its own path, so each one carried its own path displacement, and the differences I was reading as treatment effects had a component of "this is a different folder" mixed in. The effect I was chasing was two to four points. The folder displacement is worth about two.

The sloppy version was correct. Building every variant in the same directory meant the only thing that changed between them was the text, which is exactly the condition a controlled comparison needs. I had thrown away the control in the name of hygiene.

So the harness now uses one directory and swaps the contents between runs. It looks worse and it is right.

What I have not done

I have not fixed the tool. The tool is mine, a fork I maintain, and I could go change something in the indexing path this afternoon.

I do not know what to change. Every mechanism I can name says the path is not an input, and I have not found the one that says otherwise. A fix written now would be a guess dressed as a repair, and if it happened to make two folders agree I would have no way to tell whether I had removed the cause or hidden it. That is the same mistake as the one this post is about, one level up.

What I have done is make the measurement immune to it, which is a smaller claim than fixing it and is the one I can support.

The general version

If you are comparing configurations by rebuilding an index for each one, the rebuild is part of your treatment. Anything that varies between rebuilds, and you may not know what does, is inside the difference you are attributing to the change you made.

The check is cheap. Build the same configuration twice, in the two places your comparison will actually put them, and diff the rankings rather than the summary statistics. If the rankings differ, your comparison cannot see effects smaller than that difference, and you should find out how big it is before you trust a table.

Mine was two points on a ninety-item set, and I had been reporting differences of two points.

Postscript: it was the breadcrumb

I found it the next morning, and the section above is wrong about one thing. I said I did not know what to change. I do now.

The text handed to the model is assembled from the chunk's heading context, its signature, its docstring, and its content. I checked the path and found it only selects a label. What I did not do is ask where the context comes from.

chunker: let file_context = format!("File: {}", normalize_path(path));
embed:   parts.push(format!("Context: {}", context.join(" > ")));

The first breadcrumb is the file path, and normalize_path only fixes separators. It does not make the path relative. So every chunk was embedded with a prefix naming the machine and the checkout:

Context: File: /Users/…/repos/some-repo/src/pay/refund.ts > …
Code:
<the actual content>

One file, byte-identical, in directories differing by a single character, cache disabled:

before   0.89013606   0.89162624
after    0.9052133    0.9052133

The reason I could not find it by reading is worth naming. I had checked every mechanism that touches path and concluded none of them could matter. The path was not reaching the model as a path. It was reaching it as prose, inside a field I had already classified as "context" and stopped looking at. That is the same failure as a trace field that was computed rather than recorded: in both cases I read the code, understood it correctly, and still missed the thing, because the miss was in how I had filed the field, not in how I read it. Classification is where looking stops.

There was a second bug underneath. The embedding cache keys on a hash of the chunk's content, while the text actually embedded includes the context. Two files with the same body in different directories share a key and can be served each other's vector. That one had been there as long as the first and would have survived fixing it.

The fix keeps the directory structure (src/pay/refund.ts is signal a query can ask for) and drops the part that says whose laptop this is.

It cost a full reindex of every index, because the input to the model changed and mixing the two spaces silently is worse than the wait. And the migration had a side effect sharper than the bug: my search wrapper kills a subprocess after sixty seconds, the re-index runs inside that subprocess when a rebuild is triggered by a search, and twelve repositories were left with half-written indexes before I noticed. They came back from a backup I had taken an hour earlier for exactly no articulated reason beyond habit.

The habit was the useful part of that day.