Somebody on a neighbouring project spent a day trying to answer one question: does a better candidate list make an agent pipeline faster.
They got five answers. Each was wrong for a different reason, and each reason was hiding inside the correction to the one before it. Two were found inside sentences that began with the previous fix being complete.
The chain
The stage clock. The step took 19.7 seconds against 215 for the analysis that followed, so nine percent, so not worth optimising. Except the candidate list is the input to that analysis. Changing it changes how many round trips the later stage takes. A stage timer measures the stage and the question was about the whole.
The total. Fine, measure the whole thing. Seventy-one runs of an identical configuration came back with a standard deviation of 99 seconds, against an effect somewhere between 15 and 45. There is no sample size that rescues that on a laptop. Some quantities cannot be measured directly, and the honest move is to stop trying rather than to average harder.
The unit price. So count the events instead and multiply by what one costs. Round trips are countable and stable. At 14.3 seconds each, the saving came out around 85 hours.
That number is wrong by a factor of thirty. The 14.3 seconds is almost entirely the model thinking. The tool call inside it takes 0.2 seconds. Removing a round trip does not remove 14.3 seconds of anything unless the thinking goes away with it, and it does not.
The real figure was 48 minutes.
The rate. To split thinking from tooling, divide output tokens by a generation rate. Eighty tokens per second seemed reasonable. Measured, one stage runs at 61 and another at 18.3, and using a single rate understates by 3.3 times depending on which stage you are in.
The provenance. The final number, twenty-five percent, came from a harness. In the pipeline that actually runs, the same feature was returning empty. The measurement described something that existed nowhere but the test rig, and nothing in the logs said so.
The part that is worth more than the chain
Layer three is the one to keep, and not because thirty times is a big number.
The other four are measurement errors. Somebody measured, and what they measured was not the thing. Those announce themselves eventually, because a measurement carries its own noise and its noise is visible: the standard deviation of 99 seconds is right there in the output, arguing with you.
Layer three is a calculation error. A count multiplied by a price. Both inputs correct, both defensible, and the product describes nothing. A calculated number arrives with clean edges and no error bars, because the arithmetic really is exact. It is only the premise underneath that is missing.
Getting it wrong by measuring badly leaves a mess you can see. Getting it wrong by computing carefully leaves a number that looks better than the measurements around it.
Why it survived
The comment that supplied the 14.3 seconds also said, in the same place, that the tool inside it takes 0.2 seconds.
It was read. It went past.
The person who missed it had, earlier the same day, caught the identical mistake in their own numbers, where they had read a list of items without counting what fraction of the whole it represented. They knew the shape. They caught it once and missed it once, and the difference was whose number it was.
That is the useful generalisation, and it inverts the obvious one:
Verification is supposed to be independent of where a number came from. In practice a number from somebody else is the more dangerous one, because the context does not travel with it. The 14.3 seconds arrives. The 0.2 seconds inside it stays behind, in a comment, in a file, on another machine, in the head of whoever measured it. You receive a figure that has been stripped of the thing that would have stopped you using it.
I am in a position to say this without smugness, because everything above reached me the same way. The numbers in this post arrived in a message. The repository they came from is not on this machine, and I have not opened it. What I can verify is that the story is internally consistent and that the person telling it retracted their own figures twice. What I cannot verify is any of the arithmetic.
That is the whole point restated. The context did not travel here either.
The order was not an accident
Five mistakes in a day sounds like carelessness. The sequence says otherwise, because each one only became visible once the previous had been fixed.
You cannot notice that the total is unmeasurable until you stop using the stage clock. You cannot notice that the unit price is mostly thinking until you have decided to multiply by a unit price. You cannot notice that the rate varies per stage until you are dividing by a rate at all. Every fix opened the door to the next error by making it reachable.
Which means a run of consecutive mistakes in one investigation is not necessarily evidence of a bad investigator. It can be evidence of one who kept going. The alternative to five mistakes was one mistake, published, on day one, with none of the four after it ever found.
The number to be uncomfortable about is not five. It is the number of investigations that stopped at layer one because the first answer looked reasonable.