The classification step of my ticket bot reads one ticket and answers a single question: which of the 28 repositories does this touch? It runs on a coding-agent CLI in non-interactive mode, and the CLI has a reasoning-effort setting with four levels. I had never touched it. The question was whether turning it down would make the step cheaper without making it wrong, and the honest answer has two halves that do not sound alike.
Turns went down on every single ticket. Accuracy did not move by any amount this experiment could see, and the second half of that sentence is doing more work than it looks.
The design
40 tickets with a known answer, three arms, 120 calls, run in parallel four at a time on 2026-09-09.
The second arm is the part I would tell anyone to copy. Two runs of the same configuration give you the noise floor for free. Without it, a difference between base and effort-low is a number; with it, the difference is a number you can compare to how much two identical runs disagree.
Every call records the ticket, the arm, the answer (a set of repository names), the number of tool-use turns the CLI reported, and wall-clock seconds. The gold answer is a set too, so the strict score is set equality, which I will call exact.
Turns: 21 of 21, one direction
| arm | turns, median | turns, mean | exact (n=40) |
|---|---|---|---|
| base | 18 | 19.1 | 37.5% |
| base2 (twin) | 17 | 19.4 | 32.5% |
| effort-low | 6 | 7.0 | 37.5% |
(turn statistics over the 21 tickets where all three arms returned a turn count; exact over all 40)
The paired view is the one that settles it. For each of those 21 tickets, subtract the base turns from the low-effort turns:
The effect is four times the noise and every pair points the same way. The twin arm is what makes the table believable: two runs of the same setting land at 18 and 17, and changing the setting moves it to 6. That is what a real effect looks like next to its own noise floor.
Two tickets are not in that 21 because the base arm ran out its clock at 600 seconds without reporting a turn count. The low-effort arm finished the same two tickets in 132 and 138 seconds, in three and four turns.
Accuracy: not the same, but not measurably different
Exact match is 37.5% in base and 37.5% in effort-low, and the twin came in at 32.5%. The temptation is to write "accuracy unchanged" and move on. Here is what the per-ticket disagreements actually look like:
Two arms running the identical setting disagree on 4 of 40 tickets. The real comparison, base against low, disagrees on 8 tickets and splits them 4 to 4. Eight discordant pairs cannot reach p < 0.05 under any test you care to name; that is arithmetic, not a lack of diligence.
So the claim I can make is narrower than "same accuracy": low effort did not reduce accuracy by any amount this design can detect, and this design can detect roughly a 10-point difference and nothing smaller. If the true cost is 5 points, I would not know. That sentence belongs in the write-up, and it is the sentence people leave out.
Recall is unusable here, and it is worth saying why
Label-level recall differed between the two twin arms by 11 points. Not between base and low; between two runs of the same configuration. 18 of the 40 tickets have exactly one gold repository, and on those tickets recall per ticket is either 0 or 1, so a handful of flips swings the aggregate. Any metric whose noise floor exceeds the effect you are looking for is not a metric for that experiment, and you only find out by running the twin.
Wall clock is unusable for a different reason
The runs were parallel, four at a time, and in that window the same prompt took anywhere from 42 seconds to 7 minutes. The seconds column (medians of 230, 242 and 94) is a hint, not evidence. Turns do not care about neighbours; that is why they are the headline number.
What the experiment corrected about my own beliefs
I sealed four predictions before running it, in a file, with the date. Two were wrong.
P4 being wrong is the valuable one. I expected this experiment to end in "cannot tell" and wrote that down, so when it did not, I could count it rather than quietly revise what I had expected all along.
The mechanism model was wrong too. I had pictured effort as "how deeply the model checks before answering", a per-turn depth. The turn counts say otherwise: in this system, thinking and deciding-to-stop are not separate knobs. Lower effort produced shorter loops, not shallower turns.
The twist: base was not the default
After adoption I found that the base arm had not been running at the CLI's default effort at all. The non-interactive CLI reads the same user settings file as my interactive sessions, and that file sets effort to the highest level. Every measurement my bot had ever produced was at maximum effort, and this table is really "xhigh versus low", not "default versus low".
That changes the reading in one way: the 63% turn reduction is the distance from the top of the range to the bottom, not from the middle. It does not change the accuracy conclusion, which was "cannot see a difference" either way. It did change the fix. Automation now launches the CLI with --setting-sources project,local so a person's preferences do not ride along, and effort is set explicitly per stage in the bot's own config.
The bug that would have passed a bad design
The ticket regex was [A-Z]+-\d+. It matched ISO-8601, SHA-256 and UTF-8. 44 of these had entered the gold set as tickets, and the 40-ticket sample drew 9 of them. A non-ticket returns an empty answer in about 3 seconds, so before I filtered them the average cost per call looked 12 times lower than it was (32 seconds against a real 380). On that basis a 25-hour experiment would have been budgeted as 2 hours and approved.
They account for 9 of the 13 tickets that came back blank in every arm, which is most of the reason the turn table has n=21 rather than n=40. The same regex lives in the bot itself; that is a separate fix.
What I adopted
Low effort for the classification step only, on 2026-09-14. The other stages (writing code, diagnosing, QA) keep the value they were already running at, now written down explicitly, because I have no measurement for them and "it was cheaper here" is not evidence there.
Two single-call spot checks after the change, same ticket, same answer both times:
Frequently Asked Questions
Why not just report accuracy unchanged? The numbers are literally equal.
Because two runs of the same configuration produced 37.5% and 32.5%. Equal point estimates from one run each tell you the difference is inside the noise, not that it is zero. Reporting the detectable-effect size (about 10 points here) is what lets a reader decide whether that is good enough for their case.
Is the twin arm worth a third of the budget?
For anything that produces a headline number, yes. It cost 40 calls and it is the only reason the turn result is a result and the recall result is correctly discarded. Without it, both would have been reported, and one of them would have been wrong.
Would a larger sample fix the accuracy question?
Roughly 8 discordant pairs per 40 tickets means you would need several hundred tickets to resolve a 5-point difference, at about six minutes per call at high effort. That is a real experiment, not a bigger version of this one, and it is only worth running if 5 points matter for the downstream step.
Does lower effort just mean the model gives up sooner on hard tickets?
The two tickets where the high-effort arm timed out at 600 seconds and the low-effort arm finished in about 135 seconds say the opposite on those cases. But the design cannot separate "stopped earlier and was still right" from "would have been wrong either way", because on those two tickets the high-effort arm never answered at all.