Both incidents happened to a bot whose tests were green. Both were caused by code that behaved exactly as written. Both ended in a person's direct-message inbox, which is the one place a bot cannot take anything back.
Incident one: the first tick after a restart
On 2026-09-19 at 18:24 I merged a worker's branch and restarted the bot. One of the new features was a sweep that notices runs which have stopped making progress and says so in the run's card thread.
The first tick of the sweep read the live run table and found 29 rows without a card. Sixteen were pickup tests from 2026-09-07. Four were leftovers from a test fixture. Eight were rows from the alert-driven lane, which does not create cards. One was something else. None of them was a run a person had asked for.
The sweep had a fallback: if there is no card thread to speak in, send to report.to, which is the owner's direct messages. So it did. Thirty top-level DMs in eight seconds.
The expiry sweep, merged in the same branch, had the same fallback and 22 more rows queued. It was due to fire twelve minutes later. I got to it first, and then spent a few minutes calling chat.delete twenty-nine times.
The owner's response was one sentence: only check tickets the bot itself made. Which is a product rule, and it is also the whole bug. "The bot itself made it" has a concrete meaning in the data: a card exists. Rows without a card are not the bot's business to talk about, and the fallback existed precisely to talk about them anyway.
What changed
The last line is the rule, and it is the one I would not have written without the incident. Tests run against a test database. The worker that wrote the sweep had never seen the live table and could not have; the merge gate was green because the gate cannot know what is in production. The only thing that knows is a query against production, run before the switch is flipped, with the number written down.
Afterwards I purged 2,163 test-era rows from the live database (1,008 from evaluations, 40 demo, 483 test, 632 older than the cut-off), with a backup taken first. What remained was real runs only, which is the state the sweep had assumed all along.
Incident two: --help is not a no-op
The bot has about 140 scripts. A checker called smoke.mjs launches each of them with --help to catch import-time breakage: a missing module, a syntax error, a top-level throw. It has been useful. It assumes --help does nothing.
On 2026-09-22 between 06:56 and 06:59 I ran it three times. It produced five real side effects.
daily-report.mjs did not handle --help. It generated and sent the real daily report to the owner's DM. Three runs, three reports.
start-ticket.mjs treated --help as the ticket key argument and sent POST /runs/start {"ticketKey": "--help"} to the running bot, which created two runs in the live database and two DM cards. The same thing had happened the previous day, three times, and nobody had noticed because the runs were no-ops.
The checker had passed the parent's .env and DATABASE_URL to every child. It trusted the convention. Fifteen scripts, when I looked, opened the database or read a token before they got around to checking for --help.
What changed
Running the checker after the fence went up: 20 scripts broken, of which 14 had been silently touching the database or a token on --help and were now caught. After the help-first change: 5 broken, all from before and none able to reach the database or Slack, and 134 fine.
The shape both incidents share
The fallback in the first incident deserves its own sentence. It was written as the safe choice: if there is nowhere to say this, at least tell the owner. That reasoning is how most fallbacks get written, and it is exactly backwards for anything that speaks. A message with no proper home should not be delivered to a person's inbox; it should be dropped and counted. Silence is recoverable. Thirty DMs are not.
Frequently Asked Questions
Why not just rate-limit the DM channel?
A rate limit would have turned 30 messages in 8 seconds into 30 messages over 10 minutes. The messages were wrong, not merely fast. The fix is that the code has no path to that channel, verified by a test that reads the source, plus the per-tick cap as a second wall for the case where the path is legitimate.
Isn't counting live rows before every merge expensive?
It is one query per feature that speaks, run once, by hand, before the restart. The point is not automation; the point is that a person sees the number before the code does.
Why did the smoke checker have secrets at all?
Because it was started from a shell that had them, and inheriting the environment is the default. Nothing in the checker needed a token. The general rule I took from it: a tool whose job is to run other programs "harmlessly" must be the one that removes the means of harm, because it cannot audit every program it runs.
Did the ticket-key validation break anything real?
No. Every key the bot has ever started matches the configured pattern, and the same function already decided what counts as a key in the Slack ticket field. The API door was the only entry that had not been asking.