The flag was set and the feature was not there

I spent a day last month getting GPU embedding working, measured a solid speedup, wrote it up, and moved on. Weeks later I found the binary in my PATH had been compiled without GPU support the entire time.

Everything in between had been running on the CPU. Nothing errored. The environment variable that turns GPU on was set, the runs completed, the numbers looked plausible if you did not have anything to compare them to.

What the three paths actually cost

Same corpus, 1,840 chunks, same model, same machine:

GPU requested, binary built without it     306s
default path (ONNX runtime)                130s
GPU requested, binary built with it         51s

The middle row is the one that stings. The default path, the one you get by doing nothing, was two and a half times faster than the path I had deliberately configured for speed. My careful optimisation was slower than not optimising, and had been for weeks.

How it fails

The library exposes a device constructor. Asking for the GPU when the crate was compiled without GPU support returns an error, and the code did this with it:

let device = Device::new_metal(0).unwrap_or(Device::Cpu);

That is a reasonable-looking line. The comment above it, which I wrote, said acceleration should not be a hard requirement: if the device is missing, fall back and keep working rather than refusing to start.

The reasoning is fine and the outcome is not, because the fallback is silent. There is no log line, no warning, no field in the output that says which device ran. The only observable difference is that everything takes six times longer, and "slower than expected" is not a signal anyone acts on when they have no baseline for what expected means.

I had also never checked the second half of the setup. The scheduled job that rebuilds indexes overnight does not set the GPU variable at all, so it had always been on the default path. Two separate gaps, one of which I created and one of which I never looked for.

The version I ship now

let device = match std::env::var("CODESEARCH_CANDLE_DEVICE").as_deref() {
    Ok("cpu") => Device::Cpu,
    _ => Device::new_metal(0).map_err(|e| anyhow!(
        "GPU embedding was requested but the Metal device could not be opened: {e}\n\
         The usual cause is a binary built without `--features metal`.\n\
         To run on CPU, say so explicitly: CODESEARCH_CANDLE_DEVICE=cpu"
    ))?,
};

Asking for the GPU and getting the CPU is now an error. If you want the CPU you say so, and saying nothing at all still gets you the default path. The failure mode I removed is specifically asking for one thing and silently receiving another.

It has one hole I know about. The check runs when the embedding backend is constructed, which only happens if there is something to embed. A run where nothing changed still exits cleanly on a binary that cannot do what it claims. I have not fixed that, and I am mentioning it because a guard with a known gap is worth less than the version of it in your head.

The same shape, worse

The GPU flag was the second time that week. The first cost me an hour and a half of rebuilding.

I ran an indexing command by hand, the plain form with no options. It rebuilt four repositories with the default embedding model rather than the one they had been built with. Both models produce 384-dimensional vectors, so nothing downstream had any way to notice: the shapes matched, the writes succeeded, the searches returned results.

                      before   after
MRR, repository A      0.333    0.040
MRR, repository B      0.342    0.061
MRR, repository C      0.284    0.077

The tool did print a warning. It went by in a wall of progress output, and the operation it announced was not "I am about to discard and rebuild four indexes with different weights" but a single line about the model changing.

The fix there is the same as the GPU one, one layer up. If a stored index records the model that built it and you ask for something else without naming it, the command now stops and tells you which model to pass. It costs a second the first time you hit it and it makes an entire class of quiet corruption impossible.

The one that had been broken longest

While chasing those two I found a third, older, and the reason I mention it is that its symptom was the opposite of alarming.

One repository had been sitting at the bottom of every ranking I produced, and I had read it for weeks as thin content. It had been indexed with a different embedding model than the rest (the same migration that missed one entry) so its scores were on a different scale from everything I was comparing them to. I wrote that up elsewhere as a measurement mistake, which it also was. Here is the number that made it obvious:

same query, same command
  a repository on the current model    0.9293
  the repository on the old model      0.0476

Two orders of magnitude apart, and I had been reading the small number as "this repository is not relevant" rather than "this number is not comparable." Rebuilding it moved it from 0.0476 to 0.9293 and it stopped being last.

What connects them

None of these is a bug in the sense of something crashing or returning a wrong answer to a well-formed question. Every command succeeded. Every output was well-formed. The system was, at each point, doing exactly what it had been told by some combination of a flag, a default, and a build option, just not what I believed it had been told.

What they have in common is that the wrong behaviour and the right behaviour produce the same shape of output. A slow run looks like a fast run with a bigger number. An index built with the wrong model looks like an index. A repository answering in a different vector space looks like a repository with poor content.

Timing did not catch the GPU one because I had no baseline. Quality metrics did not catch the model one until I ran them for an unrelated reason. Neither would have caught the third, because a low score is what a bad result is supposed to look like.

I had met this shape before, which is the least comfortable part of the post. Months ago the lexical half of my retrieval turned out not to have run for weeks, same silence and same well-formed output, and I wrote it up and moved on. What I did not do was turn that finding into something I checked anywhere else. Then I set a GPU variable on a binary that could not use it and did not look.

So the rule below is not a discovery. It is the thing I should have generalised the first time: for each component I believe is running, find a number that proves it ran. Not that the pipeline succeeded, because pipelines succeed. Not that the dashboard is green. A number that is different when the component runs and when it does not, printed where someone will see it.

For the GPU that is the device name in the startup line. For the model it is the model name in the index metadata, checked against what was requested. For the third one it would have been the simplest check of the three and I did not have it: do all of my indexes report the same model? One line, and it would have saved several weeks of misreading one repository as bad.