slowbench
RetrievalInferenceEvalsAgentsFindingsSeriesBenchmarksArchive
Agents6 min1,155 words

The approval queue said nine fixes were waiting. Eight were already in production

Human in the loop agent approvals: the bot's pending list was its own table while people merged the same branches by hand. The fix was a check at the moment of speaking, not a sync job.

Contents · 5 sections
  1. 1How eight became a lie
  2. 2The obvious fix, and why I did not do it
  3. 3What the digest does now, before it says a number
  4. 4What I expect tomorrow
  5. 5Frequently Asked Questions

The bot's path to production ends with a person. A fix goes to development, then into a staging batch, and when staging is green the bot writes one row: prod-approval-pending. Every morning, inside a two-hour window, it sends a direct message listing everything in that state, with a button. The person reads, clicks or does not, and that is the only human step in the pipeline.

One morning the message said eight fixes were waiting for production approval. The person who received it had merged most of them into production the day before, by hand, through the ordinary release process. The bot did not know, because the bot's definition of "waiting" was a row in its own table, and nothing in the world was obliged to update that row.

How eight became a lie

Two paths to production exist and always did. The bot's path: staging batch, approval row, button, merge, deploy. The human path: a release branch or a pull request into the production branch, cut whenever someone decides to ship. They share the same fix branches. They do not share state.

So when a person merges fix/x into production on Sunday, the bot's row for fix/x stays prod-approval-pending, and Monday's digest lists it. And Tuesday's. And Wednesday's.

I measured it on the evening of 2026-09-22, for every pending row, with two git commands per repository: fetch the production branch, then merge-base --is-ancestor <landing commit> origin/<production>.

pending rowscreatedalready in the production branch
52026-09-18 05:005 of 5 (merged by people between 09-18 14:15 and 09-22 06:24)
32026-09-21 00:053 of 3 (merged 09-21 09:12 and 15:17)
12026-09-22 05:060 of 1: the only fix actually waiting

Nine rows, eight of them stale. The five from 09-18 had been listed in four consecutive daily digests. The one fix that genuinely needed a click was the newest, and it was in a list where the reader had already learned that the list was wrong.

That last part is the actual cost. A queue that is 89% stale is not a queue; it is a message the reader stops opening. The bot had spent four days training its one approver to ignore it.

The obvious fix, and why I did not do it

The first proposal was a sync job: every ten minutes, walk the pending rows, fetch, check ancestry, close what is merged. It is what most people would build and it would have worked.

The person receiving the messages had a sharper version: "not every ten minutes; check right before you send the message." That is a different design, not a different interval.

A periodic sync makes the table usually right. The digest is the only place the table is spoken aloud to a person, so the table needs to be right at exactly that moment and at no other. Anything checked ten minutes earlier can be wrong by the time it is sent, and a check that runs 144 times a day to protect one message a day is 143 checks that protect nothing. Worse, a sync job gives you a second thing to monitor: if it dies, the table silently rots again and the digest starts lying again, and nothing in the digest's own code knows.

So the check went into the sending path.

What the digest does now, before it says a number

for each row in prod-approval-pending:
    fetch origin/<production branch>           (per repository, once)
    if landing commit is an ancestor of it:
        look up the deploy pipeline for that commit, without waiting
            success       -> close as deployed
            failed        -> close as failed
            no record     -> close as deployed, note "no pipeline record"
        write one line in the fix's thread:
            "a person merged this into origin/<branch> (<sha7> · <time>) · pipeline <result>"
re-read pending
if nothing is left: send nothing
otherwise: retire yesterday's card, send today's with the survivors

Three details that were not obvious until they were written:

Closing a row is compare-and-swap. The row is closed only if it is still prod-approval-pending at the moment of the write. The button handler can be closing the same row from the other side, and the loser of that race must lose cleanly rather than overwrite.

The check runs under the batch lock. If a staging batch holds the lock, the reconcile is skipped and so is the digest, for that tick. The window is two hours and the tick is thirty seconds; a digest sent an hour late is fine, and a digest sent from a table that a batch is in the middle of rewriting is not.

The thread line is the record. "Closed as deployed" in a table is invisible. A line in the thread, with the short sha and the time a person merged it, is something the approver can read and something I can grep in a month when someone asks why the bot never asked about that fix.

What I expect tomorrow

The first automatic run is scheduled for the next digest window, 2026-09-23. Against tonight's table it should close eight rows with eight thread lines, delete the previous card, and send a card with one item. If it sends a card with nine, the ancestry check is wrong; if it sends nothing, the compare-and-swap or the re-read is wrong. Either failure is legible from the message alone, which is the point of checking at the moment of speaking.

Frequently Asked Questions

Why not have the human path notify the bot?

Because the human path is git and a hosted pipeline, owned by people who should not need to know the bot exists. A merge into the production branch is already the notification; the bot just has to look before it talks. Adding a webhook creates a second source of truth with its own failure modes, and the table was already the problem.

Isn't fetching every repository before every digest slow?

One digest a day, five repositories, one fetch of one branch each. It is a few seconds inside a window measured in hours. The ten-minute sync would have been the expensive version.

What if the pipeline for a merged commit is still running when the digest checks?

The lookup does not wait. A run still in flight has no terminal record, so the row closes as deployed with the "no pipeline record" note. That is a deliberate bias: the merge already happened by a person's hand, and the bot's job at that point is to stop asking, not to second-guess a deploy it does not own.

Does this replace the approval button?

No. The button is still the only way the bot itself merges to production. The reconcile only removes items that a person has already taken out of the bot's hands, so the list the button sits on is true.