The end of a working day, in order: fix a feature, commit it, move the bot's branch onto that commit, restart the bot, and import the merged module directly to confirm it produces what it should. It did.
Then run one real task.
| before | after | |
|---|---|---|
| wall clock | 2,350s | 1,094s |
| tool calls | 146 | 77 |
| candidate file lines in the prompt | 0 | 0 |
Time halved. Calls halved. And the third row is the one the fix exists to create, absent in both runs.
The fix did not run. The improvement is one sample against one sample, and the difference between them is that this run happened not to hit a retry.
Where it was running
The bot's process was started in a worktree, on the right branch, and its working directory was correct. That was checked, and the check passed.
Its configuration launched a subprocess like this:
An absolute path into the main checkout. And that configuration file was a symlink, the same file in every worktree.
So the bot's body ran on the new branch and everything the bot spawned ran on the old one. Moving the branch could never have reached it. The two halves of the same program were on different commits, and nothing anywhere reported a conflict, because from each half's point of view it was doing exactly what it was told.
"Where is the bot running" turned out not to have one answer. The process directory was verified and was right. It was half the question.
Two claims that get merged
Everything up to the last step was real work correctly done. The module was imported and produced the expected output, which proves the code is correct. What it does not prove is that the running system reaches that code.
Those are separate claims and the language does not keep them apart. "I fixed it" and "the numbers improved" sit next to each other in a sentence and imply a third thing that neither states: that the improvement came from the fix.
The only evidence that would have connected them was in the prompt itself, one line long, and it says zero in both runs.
Without that check the day ends with a 53 percent speedup attributed to a change that never executed. Not a wrong number, since 1,094 seconds is what the run took. A correct number with the cause attached backwards, which is worse, because it survives review. Everyone agrees the number is real, and it is.
What the numbers actually support
The halving is not evidence against the fix either. The fix was absent from both runs, so neither run says anything about it.
What is left is narrow: with zero candidates supplied, this task costs somewhere between 77 and 146 tool calls. That is a factor of two, from two observations of the same task. It is a range, not a measurement, and the honest use of it is as a reason not to read any single run of this task as a result.
The 53 percent has no owner. Something made this run cheaper and it was not the change under test.
The check that would have caught it
Verifying the artifact and verifying the deployment are different acts, and only one of them was performed.
Importing the module tests the code. Reading the prompt tests the system. The distance between them, here, was a symlinked config with an absolute path in it, which is exactly the kind of thing no code-level test can see, because the code is fine.
The concrete version, for a system that assembles inputs: look at the input the running system actually produced. Not the function that builds it, not a test that calls that function, but the artifact that went out. A line that should be there and is not is the cheapest possible signal, and it costs one glance.
I have written before about a feature that returned empty for eighty consecutive runs while every log stayed silent. This is the same failure from the deployment side. In that one the code was broken and nothing said so. In this one the code was fixed and nothing said the fix was not in the path.
Both are cases where the system's output was well formed and the thing you cared about was not in it.