A search evaluation reported recall@5 = 1.0000. That got read as a result. It was the harness describing its own construction.
The four cells
Two things varied. The judging rule, which decides whether a returned document counts as relevant, was rewritten between versions. And the scoring scope was narrowed from all five returned slots to only the product slots, typically three.
| rule | scope | queries | recall@5 | NDCG@5 | precision@5 |
|---|---|---|---|---|---|
| v1 | all slots | 39 | 1.0000 | 0.6792 | 0.5026 |
| v1 | products only | 39 | 1.0000 | 0.9827 | 0.5026 |
| v2 | all slots | 67 | 1.0000 | 0.6849 | 0.5642 |
| v2 | products only | 67 | 1.0000 | 0.9823 | 0.5642 |
NDCG travels from 0.68 to 0.98. Precision moves. The query count changes from 39 to 67 because the rule change alters which queries have any relevant document at all. Every quantity in the table responds to something.
Recall is 1.0000 four times, to four decimal places.
The tell is not the value
A single perfect score is ordinary. Small sets produce them, easy tasks produce them, and a metric can be at ceiling for boring reasons.
What is not ordinary is one column refusing to move while every other column does. Two different rules and two different document sets, and the number that is supposed to describe how much was found does not react to either. A quantity that ignores its own inputs is not measuring them.
That is the general form, and it is cheaper to check than it sounds: change something the metric should care about, and confirm it cares. Here two such changes were already sitting in the table, and nobody read down the column.
Why it could not have been anything else
The answer key was built by taking the search results and applying a rule to them. Whatever the search returned got judged; whatever it did not return was never presented to the rule.
So a document the search missed cannot be marked relevant, because it was never a candidate for marking. There is no relevant document outside the top five, by construction. Recall is the fraction of relevant documents that were retrieved, and the set of relevant documents was defined as a subset of the retrieved ones.
The numerator and the denominator are the same set. The metric was computing one divided by itself and reporting it to four decimal places.
What survives and what does not
Precision and NDCG remain usable here. They ask about the documents that came back, ordered how, and the answer key is competent to judge that.
Recall is unrecoverable from this construction, not badly estimated. A key built retrospectively from system output can never show a miss, because the misses were never in the room. To measure recall you need relevance judgements made against a document set that does not depend on the ranking under test, which usually means a pooled key across several systems, or human labelling over a sample of the corpus.
The distinction worth carrying: some metrics are wrong because the data is thin, and more data helps. This one is wrong because of where the data came from, and no amount of it helps.
The neighbouring failure
I wrote recently about a routing benchmark scoring 100 percent under a containment rule, where the rule gave free credit for guessing wide and a fixed three-name answer scored 40 percent without reading anything.
These are the same disease at two different layers, and they are worth holding side by side. In that case the scoring rule was too generous about what counted as correct. In this one the answer key was assembled from the thing being scored. One is a lenient judge. The other is a ruler cut to fit the object it measures.
Both produce a high number, both are correctly computed, and neither is discoverable by staring at the number.
What I can and cannot vouch for
These figures reached me in a message. The script that produces them is committed and the numbers are reproducible from it; the run file it reads is under a gitignored directory and is not in the repository, and I have not opened either, because the machine holding them is not this one.
One number from the same investigation is deliberately absent here. An earlier console run reported a recall figure well below 1.0, and re-scoring the saved data with the currently committed code does not reproduce it. The cause was not chased down. It has been left out on the grounds that a figure which does not reproduce is not evidence, whichever direction it happens to point.