I had three consecutive runs where the agent passed QA on the first attempt, which had never happened before. I was fairly pleased about it.
Then I diffed one of its files against what a person had shipped for the same ticket weeks earlier.
Not similar. Identical. It had also created a helper component with the same filename as the human's, which is not the kind of thing that happens by convergent evolution.
The agent wasn't solving tickets. It was finding the finished work on disk and copying it.
How it could see the answer
The setup was: give the agent a closed ticket, let it work in a scratch worktree, compare its output to what was actually shipped. Clean idea. The problem is where that worktree lived.
The worktrees were created from the main checkout, the same clone I use for daily work, sitting at the latest commit. Every ticket in my evaluation set had been merged into that checkout months ago. The answer to every question was one directory traversal away.
I had thought about this, sort of. I'd blocked the obvious route:
That did nothing, because git was never the interesting path. The agent has a file-reading tool with no directory restriction, and my own analysis prompt hands it the repository path in the first paragraph so it knows where to look. It didn't need git history. It needed ls.
Blocking commands while leaving the filesystem open is theatre. The data was reachable by ordinary means and the agent used ordinary means.
What that invalidated
This is the part worth being precise about, because the temptation is to throw out everything and start over.
Invalid: every quality signal.
- "Passes QA on the first attempt, three runs in a row". Of course it did, it was submitting known-good code.
- "Injecting API documentation stopped it from inventing endpoints". It wasn't using the documentation. It was copying the deployed implementation, which naturally has correct endpoints.
That second one stings more than the first. I had built a feature, measured an improvement, and written down a causal story about why the feature worked. All three steps felt like engineering. None of them were.
Still valid: timing. Runs took as long as they took. Copying a file is not free. The agent still read, planned, and wrote. Latency measurements survived.
So the damage was bounded but it was exactly the part I cared about. I could tell you how long the agent took and nothing about whether it was any good.
Why I didn't catch it sooner
Three consecutive first-attempt passes should have been suspicious on its own. It wasn't, and I've thought about why.
Partly it was that the improvement arrived alongside a change I'd made. I had just started injecting API documentation into the prompt, so there was a ready explanation sitting right there. A result that confirms the thing you just built does not feel like it needs auditing.
Partly it was direction. Bad results get investigated, in my experience, because they are a problem to solve. Good results get written down. I had a note explaining why documentation injection worked, drafted the same afternoon the numbers came in, and never once opened the output files to see what the agent had actually produced.
And partly the numbers looked plausible. Three for three is good but not absurd, since an agent that solves easy tickets on the first try is a thing that could exist. If it had passed forty in a row I'd have gone looking on day one. Contamination that produces a believable number is much harder to notice than contamination that produces a perfect one.
Reachability is the thing to enumerate
What I had was a denylist of commands, which felt rigorous because it was written down and specific. What I needed was a list of paths the agent could read, which I had never written at all.
The exercise takes ten minutes and is uncomfortable in a useful way. For each thing on the list, ask whether the answer to any evaluation question is inside it:
Two of those five surprised me. Sibling repositories mattered because tickets in this system routinely touch more than one repository: of the 117 tickets in the answer key, 45 name one repository, 53 name two, and 19 name three. Isolating one and leaving the others at the latest commit leaks the shape of the solution even if the specific file is gone.
And the last line is the one I'd never have found by thinking about tooling. My own prompt told the agent where to look. No amount of command blocking survives that.
Isolating it
The fix is a repository the agent can reach that does not contain the future. Roughly thirty minutes of work.
The one thing not to do is rewrite history in the original repository. The agent's worktrees are attached to that .git, so surgery there breaks the thing you're trying to measure. Clone instead:
--local makes it cheap, since it hardlinks the object store instead of copying, so a large repository clones in seconds and costs almost no disk.
The three lines after the checkout are the ones I'd have skipped if I hadn't been burned already. Rewinding the working tree isn't enough: origin still points at the real repository, and anything that fetches gets the answer back. Repointing the remote at the isolated clone itself makes that impossible rather than merely unlikely.
Then point the runner's config at the isolated path, delete any cached worktrees, and run. To go back, restore one config line.
Choosing the old commit: the parent of the merge that closed the ticket. Not "a few weeks earlier": repositories in a group get their changes at different times, and one being ahead is enough to leak the shape of the solution.
The option that looks equivalent and isn't
The runner had a baseRefs parameter that pins a run to a specific commit. That looks like the same thing and it is not.
baseRefs rewinds the working tree. The main checkout is still on disk, still at the latest commit, still readable. An agent with a file-reading tool walks over and reads it.
Rule of thumb I now use: if the correct answer exists anywhere on the filesystem the agent can reach, assume it will be found. Not because the agent is devious, but because reading nearby files is exactly what you want it to do the rest of the time. You cannot ask for resourcefulness and then be surprised by it.
Pin the commit if you're measuring latency. Clone if you're measuring quality.
What the real numbers looked like
Same ticket, isolated:
One repository failed and needed a person. That's a worse number and a real one.
The other passed, and the way it passed is what I'd been trying to measure all along: same files touched, same structure, independently written body. Convergent solutions rather than a copy. That is a signal about capability. The 239-line diff-zero result never was.
There's something almost funny about the failure reason. The agent tried to hardcode a spec that the backend supplies. That's a genuine, diagnosable, fixable mistake. Under contamination I'd never have seen it, because on that ticket the human had already gotten it right and the agent had copied the human.
Two things I'd check on any agent benchmark
Diff the output against the reference before believing a pass. Not a similarity score, an actual diff. Zero is not an impressive result, it's a smoke alarm. This takes one command and I hadn't been doing it.
Enumerate what the agent can reach, not what you told it to use. I had a list of blocked commands, which felt like a security posture. What mattered was a list of readable paths, and I'd never written one. The question is not "did I tell it not to look" but "is the answer physically present."
The general version: a benchmark is only measuring capability if solving the task is genuinely easier than finding the answer. When it isn't, you get a good number and no information, which is worse than a bad number, because a bad number gets investigated. That is the same failure as a benchmark where doing nothing scored 95%, arriving from the opposite direction.