An agent gets 79% on a task. That number is compatible with two situations that have nothing in common.
In the first, it answers correctly most of the time and occasionally wanders. In the second, there is a class of input it gets wrong identically, every single time, with complete consistency.
The metric cannot tell you which. And the fixes point in opposite directions, so if you guess wrong you spend a week on a technique that cannot work.
Running everything twice
The change that separated them was cheap: run the evaluation set twice per configuration and record whether each case gave the same answer both times.
That third column is the whole method. It costs one extra run and it splits the failures into two piles that want different treatment.
I had been treating all nineteen as one bucket labelled "accuracy problem."
Why one run per configuration is so tempting
Worth saying plainly, because the fix is obvious in hindsight and I still didn't do it for months.
Agent evaluations are slow. Each case is a full model loop: reading files, running commands, thinking. My twenty-case set took long enough that running it twice felt like a real cost, and I was iterating on prompts several times a day.
So the arithmetic in my head was: two runs means half as many experiments. And more experiments felt like more progress to me, because each one produces a number and the numbers move.
They were moving because of variance. Half my experiments were measuring nothing, and the way to find that out was to spend the runs I was trying to save. Halving the number of experiments while making each one mean something is not a trade. It's a strict improvement, and it took me embarrassingly long to see it that way.
There's a version of this that generalises past agents: if your measurement has variance you have not characterised, extra experiments do not accumulate into knowledge. They accumulate into a plausible-looking changelog.
The stable failures were rule problems
The eight stable ones turned out to be interesting in a way I did not expect: several were not failures at all.
One case answered "no repositories need changing" on every run, against an answer key that said three. The agent was right. The work had shipped months earlier and there was nothing left to do. The answer key recorded what someone changed at the time, not what needs changing now.
The rest were genuine, and they had a shape. Every one involved the agent applying a rule it had been given, correctly, to a case where the rule did not apply. One kept assigning work to a backend repository on tickets that mentioned an API, when the change was entirely in how the frontend rendered a response it already received. The ticket wording is the obvious suspect and I never confirmed it.
That is a specification problem, and a stranger one than I first wrote down. The agent behaves as though it holds a rule like "if the ticket concerns API behaviour, include the backend," and it applies that rule faithfully. When I went looking for the rule, it was not in the prompt. What is in the prompt is close to the opposite: backend work is listed as a person's, not the agent's. So the rule is somewhere I did not put it, and wherever it lives, nothing about resampling reaches it and nothing about temperature reaches it. I checked the current prompt, not the one in place when I recorded these runs.
Which means the fix is to read the instruction, find the sentence, and make it more precise. Four out of four runs producing the same wrong answer is not noise. It is the system telling you exactly what rule it applied.
The varying failures were mostly ambiguity
The eleven that varied split further once I read the traces.
Some were the answer-key problem again: one run says "already deployed, nothing to do" and another says "three repositories," and both are defensible readings of a ticket that has been closed for a year.
Others were genuinely ambiguous tickets. "The date is wrong on the reservation detail screen" can be a display bug or a data bug, and those live in different repositories. Two runs, two readings, both reasonable. A person would ask a follow-up question.
Almost none of it was the thing I had assumed, a model wobbling at random. When I looked at what varied, it varied between defensible interpretations, not between an answer and a non-answer.
That changes the fix. You do not sharpen a prompt to remove ambiguity that is in the input; you either accept it and return multiple candidates, or you add the missing information from somewhere else.
Why the distinction is worth the extra run
Here is what I had been doing to all nineteen failures: editing the prompt and re-measuring.
For the eight stable ones that was right, and it worked once I could see which they were.
For the eleven varying ones it was worse than useless, because prompt edits move variance around without reducing it. I would change a sentence, the aggregate would move by two points, and I would record that as an improvement. Run it again and the two points went somewhere else. I was reading noise as signal for as long as I had one run per configuration.
The stability column also gives you a sanity check that costs nothing. If a configuration is stable on 90% of cases and you make a change that drops stability to 60%, you have made the system less predictable even if accuracy went up, and that is usually a bad trade in a system where someone has to trust the output.
What two runs cannot tell you
Two runs distinguish "always the same" from "not always the same" and nothing finer. That's enough for the split I needed, and it is worth being clear about what it isn't.
It cannot tell you the shape of the variation. A case that alternates between two defensible answers and a case that produces something different every time both read as "varies," and those are not the same situation. Three runs starts to separate them; I have not needed to.
It cannot tell you whether stability is earned. A case can be stable because the answer is obvious, or stable because the system is ignoring the input in the same way every time. Only reading the answer distinguishes those, and the second one is exactly what the null baseline does.
And it interacts badly with a small evaluation set. On twenty cases, the difference between 11 varying and 9 varying is one or two cases changing behaviour, which is not a signal about anything. I use the stability column to sort failures into piles, not to compare configurations against each other.
The honest summary: two runs is a triage tool, cheap enough to run always, and it earns its cost by telling you which of two completely different debugging paths to take. It is not a statistical treatment of variance and I do not use it as one.
What I record now
For every evaluation run:
Recall matters most in my case, because a missed repository means the agent never opens the file that mattered and everything downstream is wasted. Precision failures are cheaper: an extra repository costs a wasted branch and a review comment.
Stability is the one I would have skipped if I hadn't been forced into it. It is the only column that distinguishes a system that is right for reasons from one that is right by luck, and it's the only one that tells you which kind of failure you're looking at before you start fixing.
The uncomfortable corollary
A perfectly stable system can be perfectly wrong, and it will look calmer than a system that is mostly right.
The null baseline I used elsewhere in this project, answering the same three repositories regardless of input, scores 100% stability by construction. It never varies because it never reads anything. If you optimise for stability alone you converge on exactly that.
So stability is a diagnostic column, not a target. It tells you what kind of wrong you have. Accuracy tells you how much. Neither is meaningful without the other, and I had been running with only the second one for months.
It happened again while I was writing this
I changed some comments in a Rust project, renaming a few example identifiers and nothing executable, and ran the test suite before committing. One test had been failing before my change, a known environment thing. After my change, two were failing.
My first thought was the obvious one: I broke something. The second failure was in a completely unrelated module, which should have been a clue, but a comment-only diff that turns one failure into two is exactly the kind of thing where you assume you've missed something subtle.
I ran it twice more. Both times: one failure, the same known one. The second failure never came back.
So it was flaky, and my change had nothing to do with it. Had I stopped at the first run I would have reverted a correct commit and gone looking for a bug that did not exist, in a diff that changed no code at all.
What made the difference was three seconds of repetition. Not a better test, not more logging, not reading the diff again more carefully. Just running the same thing twice and noticing that the number moved.
The version of this that scales is the same discipline the rest of this post argues for. A count of failures is not a fact until you have seen it twice. One run tells you a number. Two runs tell you whether the number means anything, and the second run is almost always cheaper than the investigation you would otherwise start.