I reported 100% on a routing benchmark. The number was correctly computed under the rule I had written, and the rule was wrong in a way that pays for guessing.
The task: given a ticket, name the repository or repositories where the fix belongs. The scoring rule: a ticket counts as correct when the expected set is contained in the answered set.
The ten cases
| ticket | expected | answered |
|---|---|---|
| 1 | A, B | A, B |
| 2 | A | A, B |
| 3 | A | A, B |
| 4 | C | C |
| 5 | D | D |
| 6 | E | E |
| 7 | F | F, G, H |
| 8 | I | I, E, J |
| 9 | K | K |
| 10 | L | F, L |
Read down the two columns and the perfect score explains itself. The expected value is present in every single row. Nothing was missed, so there are no false negatives, and a containment rule with no false negatives returns ten out of ten.
Now read the widths. Row 7 answered three repositories to cover one. Row 8 did the same. Row 10 named a repository that had nothing to do with it and got credit because it also named the right one.
| count | |
|---|---|
| correct repositories named | 11 |
| wrong repositories named | 7 |
| repositories missed | 0 |
Recall is 11 out of 11, or 100%. Precision is 11 out of 18, or 61%. Exact match, where the answered set equals the expected set, is 5 out of 10, or 50%.
Same data, same run, three numbers: 100%, 61%, 50%.
What the metric was actually rewarding
Under containment, adding a repository to an answer can never lower the score. It can only help, since a superset of a passing answer still passes. There is no term anywhere in the rule that costs anything for being wrong.
The limit case is answering every repository on every ticket. Recall stays at 100%, and precision falls to 11 out of 270, which is 4.1%. That is an absurd strategy, which is why it is worth writing down: the metric cannot tell it apart from a good one.
The 270 is worth a pause, because I got it wrong first. Twelve distinct repositories appear across these ten tickets, and twelve times ten gives 9.2%, which is not the number the scoring code reports. The scorer answers every repository that appears as an expected answer anywhere in the full ticket set, and that is 27. Thirty are registered in configuration and 29 of those are active, so there are three plausible denominators before you even start, and they differ by more than a factor of two. Which population a baseline is drawn from is part of the baseline.
The non-absurd version is what makes this a real problem. Take the three most common repositories, answer those three names on every ticket, and never read a ticket at all:
| fixed answer | score under containment |
|---|---|
| 1 repository | 2/10 |
| 2 repositories | 3/10 |
| 3 repositories | 4/10 |
| 4 repositories | 5/10 |
Forty percent for answering three constant names. Fifty for four. No model, no retrieval, no reading. That number is the floor my 100% should have been standing on, and I never computed it.
The same trap through a different door
A colleague building a separate router hit this from the opposite direction. Their gold set was assembled from real history, and in that history 19 of the top 20 tickets belonged to a single combination of repositories. Answering that one combination every time, without reading anything, scores 95% on their set.
Their scoring rule was fine. The distribution of their evaluation set was doing the work.
Two different failures with one symptom: a high number that a constant answer could have produced. In one case the rule allowed free credit, in the other the sample handed it over. Neither is visible from the score alone, and both are visible the moment you compute what a fixed answer gets.
What a false positive costs
Precision language undersells this. Seven wrong repositories is not seven units of abstract error.
This router feeds an agent that clones a repository, indexes it, and searches it. Each wrong name is a full pass over a codebase that could not contain the answer, and then a person reading the output of that pass and discarding it. The rule scored those seven at zero cost. They are the most expensive thing the system produced that day.
That is the general shape of a scoring bug worth caring about. Not that the number is too high, but that the number is silent about the part of the output that costs the most.
What replaced it
Three numbers instead of one, always together: recall, precision, exact match. Any single one of them can be gamed by a strategy the other two catch. Answering everything wins recall and destroys precision. Answering nothing does the reverse. Exact match is the only one that requires being right about the boundaries, and it is the one that fell furthest, from 100 to 50.
Alongside them, one more line in every report: what the best input-free strategy scores on this same set. For these ten cases that line reads 40%, and it means a result of 45% is noise wearing a percentage sign.
Computing it is cheap. It took a few lines over the raw answers, and I only had those because the raw answers were handed over with the summary rather than in place of it. That has now cost me twice in one week. The first time, a summary statistic turned out not to be the statistic it claimed, and the only reason anyone could tell was that the inputs came with it.
The score was never wrong. It was answering a question I had not meant to ask: did you miss anything. It said no, correctly, for ten tickets, while seven repositories that were never going to contain the answer went out the door with a clean bill.