Three baselines that were not constants
When the measurement moved
A baseline is supposed to be the thing you compare against. In one week three of them turned out to be measurements in their own right — a trace whose timings were synthesised, an evaluation set that changed under a re-collection, and an index whose scores depended on which folder it was built in. Different systems, different causes, the same symptom.
2 posts
Where this stands
- Held to be true
- A result that holds at one sample size or in one working directory is not yet a result. Both cases here changed sign or magnitude under conditions that were supposed to be irrelevant, and in neither case was the original measurement wrong.
- What would break it
- A comparison that survives a tenfold change in sample and a rebuild in a different directory with the numbers intact.
- Still open
- How much of this is specific to rebuilding an index between arms. The one mechanism identified so far was a file path leaking into the embedded text.
- 1Evals7 min
A tie at four hundred queries was a loss at fifteen hundred
Statistical significance and sample size in search tuning: four hundred queries said one parameter made no difference, and fifteen hundred turned that tie into seven lost answers.
- 2Retrieval10 min
Same bytes, a different folder, a different score
Embedding model evaluation that I made worse by being careful: indexing identical files under a different path returns different vectors.