Building a labelled dataset by hand is slow, so I did what most people do: mined it out of git. Find commits whose message references a ticket ID, collect the files they touched, call that the answer.
A hundred and seventeen labelled examples for an afternoon of scripting. It is genuinely a good technique and I would use it again.
It also encodes assumptions that took me weeks to notice, and every one of them made my numbers wrong in a direction I could not see.
What the label actually records
The label is not "which files need to change to fix this issue." It's "which files one particular person changed while closing this ticket, in one particular week, under whatever constraints applied at the time."
Most of the time those coincide. When they don't, they diverge in ways that are systematic rather than random, which is worse, because random noise averages out and systematic error does not.
Already-done work scores as failure
The one that cost me most.
My agent decides which repositories a ticket needs. For a ticket that was fully deployed months ago, the correct present-tense answer is often none. There is nothing left to change, and an agent that says so is right.
The git-derived answer key says three repositories, because three repositories were changed back when the work was live. So the agent answers correctly and scores zero.
I spent a while chasing what I labelled an instability problem: the same ticket getting three repositories on one run and none on another, apparently at random. The runs were not random. Some of them had correctly noticed the work was finished. My scoring counted every one of those as a failure and I read the pattern as flakiness.
That is the shape of the whole problem: the answer key is in the past tense and the question is in the present tense, and nothing about the numbers tells you which examples have drifted.
The commit message is not a description
My first version used commit titles as the query, which seemed natural: the commit says what was done, the files say where.
Most of them look like this:
There is no problem statement in that. Half the corpus was merge messages, and using them as queries measured how well the system matches branch names to file paths.
Ticket titles are much better, and better for a reason beyond containing words:
The ticket title is what the system will receive in production. It is written by a reporter who does not know the codebase, in their own language, describing a symptom. The commit title is written by the person who already solved it, in the vocabulary of the solution.
Evaluate on the second and you are measuring performance on inputs you will never see.
Refactors poison the file list
A ticket whose fix happened to land in the same commit as a rename, a lint sweep, or a dependency bump gets all of those files in its answer key.
I found examples where the label listed thirty files and two of them were the fix. Any system that correctly identifies those two scores 2/30 on precision, so precision as a metric was reporting something closer to "how well does this predict unrelated cleanup."
I did not fix this properly. What I do now is check the label size distribution and treat anything over five files as suspect. Usually it turns out to be a squashed merge or a formatting pass, and it gets dropped by hand. Ten minutes for the whole set, and it removed a systematic drag on one metric.
The repository moved on
Two subtler versions of the same drift.
Files that no longer exist. A ticket from eight months ago points at paths that have since been renamed or deleted. The agent finds the current file, which is correct, and scores zero because the label names the old path. This shows up as a slow decline in scores as your evaluation set ages, and it is not a capability change.
The architecture changed. A ticket fixed in a component that has since been extracted into a shared package now has a correct answer of "the shared package," while the label says "the old component." The system is right about today's codebase and wrong about the label.
Both make old tickets score worse than new ones for reasons that have nothing to do with difficulty. If you sample your evaluation set by date without knowing this, you can produce a beautiful trend line showing improvement, driven entirely by having sampled more recent tickets.
How much of my set was affected
I went back and counted, because "this happens sometimes" is not useful for deciding whether to trust a number. Then, writing this up, I counted again, and the second count is the one I can show you, because the first one I cannot reproduce.
Here is the second, with the criteria spelled out so you can disagree with them:
Twenty-seven per cent. When I first counted I wrote down a number a little over half, and reproducing it for this post I could not get near it. The artefact I counted from is gone, and the answer key I still have stores file counts rather than paths, so the check I described is not one it can answer. The number I would defend now is the one above, produced by two lines of git per ticket, and it is worse than the number I had been carrying around.
There is a third group I originally counted and no longer do: tickets whose work has already shipped. Nearly all of them have. The answer key is built from commit history, so by construction the fix is already in the branch an evaluation starts from, 112 of the 117. That invalidates one question and not another. If you are asking "would the system make this change today," almost the whole set is unusable and you are left with a handful. If you are asking "which repositories does this ticket belong to," it does not matter whether the work shipped, and the label is fine. I had been mixing those two questions and counting a defect that only exists in one of them.
The ratio is the thing I would have wanted to know on day one. Not because 32 is too few, but because I had been treating 117 as the size of my evidence and reporting percentages computed against it. A metric divided by the wrong denominator is wrong by a constant factor, quietly, in every report.
The first count took twenty minutes, two weeks after I started using the set. The second took ten and produced a different answer, which is its own small lesson about numbers you count once.
The version that would have caught it
None of the four failure modes need judgement to detect. They are all mechanical checks against the repository, and they would fit in the script that builds the set:
Three commands, and each one tells you which examples to distrust rather than throwing them away. I would run all three at build time now and store the flags alongside the labels, so that a low score on a flagged example reads as "check this" rather than "the system failed."
The broader habit: a dataset builder should emit its own caveats. Mine emitted a clean JSON file with no indication that a third of it had known problems, and a clean file invites you to believe it.
What I'd keep and what I'd add
I am not arguing against mining git. The alternative is hand-labelling, and a hundred hand-labelled examples is a week you will not spend, which means you evaluate on nothing, or worse, on a set that cannot tell systems apart.
What I would add, in the order I wish I had:
Print the label size distribution first. One line. Anything over five files is usually a squashed merge rather than a hard ticket, and you want to know that before you interpret precision.
Use the ticket title as the query, never the commit title. The ticket is the input your system will actually receive.
Mark tickets that were fully deployed before the evaluation date. For those, "no change needed" may be the correct answer, and scoring them against a past-tense label produces phantom instability.
Check whether the labelled paths still exist. A file that has been renamed makes an example unanswerable. git log --follow finds the current path, or you drop the example.
Prefer recent tickets, and know why. Not because they're better labelled, but because less has moved underneath them. If you must use old ones, expect a downward bias that is not about capability.
The general shape
Automatically derived ground truth is a proxy, and proxies are fine as long as you know which direction they lean.
Mine leaned in four directions at once: past tense instead of present, solution vocabulary instead of problem vocabulary, whole commits instead of relevant changes, and old paths instead of current ones. All four made my system look worse than it was, which is the lucky direction. I was not tempted to ship anything on the strength of them.
The unlucky direction exists too. A proxy that flatters the system produces confidence rather than confusion, and confidence does not prompt anyone to go back and check the labels.
Either way the question to ask about an answer key is not "is it correct." It is "what question does this actually answer, and is it the one I am asking."