Five ways a number can mean nothing

How benchmarks lie

Every post here is the same shape: a measurement that looked fine, a system that looked like it was working, and a reason the number could not have told the difference. A failing test announces itself. A meaningless test does not — that asymmetry is why these took weeks to find.

1 posts

1
EvalsSep 6, 20269 min
The baseline that beat my agent was answering the same thing every time
I improved routing accuracy from 73% to 90%. Then I checked what a strategy that ignores the input scores on the same test. It scored 95%.

The baseline that beat my agent was answering the same thing every time