I told the agent to search its memory. It made 31 tool calls and searched zero times

The bot that fixes tickets was supposed to stop repeating its mistakes. The mechanism was memory: about 170 markdown notes written by human sessions, plus whatever the bot itself had learned, made available to each run. The question was how to make it available. Inline the note bodies into the prompt, or just tell the agent that a search tool exists and to use it?

I ran an experiment, picked "just tell it", shipped that, and two weeks later measured what the agent actually did with the instruction. Zero. This post is about the experiment that pointed the wrong way, why it did, and what replaced it.

The experiment that picked search mode

One ticket (an empty-state message in the admin app), 24 runs, three memory conditions crossed with two other factors, on 2026-09-05.

Read at face value: memory saves about 20% of tool calls, and telling the agent to search is as good as inlining bodies while keeping the prompt small. That is what I read, and the config became memory.mode = search, includeBodies = false.

Three things wrong with that reading

The five inlined notes were all irrelevant. They were selected by rank from a search over the ticket text, and the top five were about a scroll-to-top button, an external marketplace payload, content grouping, internationalisation, and a product feed. Nothing about empty states, nothing about that screen. So the "bodies" arm was really "10,000 characters of unrelated text" against "nothing", and the experiment never measured the thing I cared about, which is what a relevant note does to a run.

The pass/fail judgement had no gate. At the time, whether a run "passed" was the run's own claim, unverified. So the accuracy column I would have used to break ties was not a measurement. Turn and time counts were real; the outcome column was not.

An earlier result was contaminated. On 2026-08-19 I had seen "with memory injected, the bot stopped hand-writing an API spec it should have fetched", and counted it as evidence for memory. That run could read the main checkout, where a person had already shipped the fix; the bot's file matched it at 239 lines, diff zero, down to the file name. It was evidence of a benchmark that could see the answer, not of memory. (That post is about the leak itself.)

None of these were visible in the table. All three were visible in the artifacts behind the table: the five note titles, the judgement code, the checkout state.

What search mode did in production

On 2026-09-21 I pulled the trace of one real run under search mode. The development worker made 31 tool calls. The memory search tool was called 0 times. The diagnosis stage made 16 tool calls; memory search, 0.

The instruction was in the prompt. The tool was available. The model, given a ticket and a codebase, did what models do with an optional tool and a concrete task: it went straight at the task.

Two more things came out of that trace. The search index was seven days old, because search mode had no reason to rebuild it, and a search the agent never ran would have returned stale results if it had. And the notes it could have searched were the human sessions' notes (178 shared, 8 per repository); the bot's own lessons, the reasons runs had been held in QA, dropped, re-fixed or verified, lived only in database tables and never reached the next run on a different ticket.

So the memory design had three failures stacked: the retrieval was optional and did not happen, the index was stale, and the most relevant knowledge was not in the index at all.

What replaced it: code chooses, the model reads

The redesign moved every decision about relevance out of the model and into code.

1  lesson table        at each run ending, code writes one row:
                       hold · pass-after-hold · drop · refix · verify · not-ours
                       same repository + same error signature  ->  always injected
2  memory notes        FTS query built from the alert text or ticket title,
                       per-word bm25 threshold 1.8, at most 2 notes
3  one context section prepended to every model prompt, 8 entry points,
                       first line fixed, body untouched (a test asserts line positions)
4  no context, no call the agent wrapper throws if a stage calls the model without one

The threshold needed its own measurement, because raw bm25 grows with the number of query words and no single cut-off separates relevant from irrelevant. A nine-word gold question's best hit scored 39.7; a fifteen-word irrelevant ticket's best hit scored 22.9. Dividing the best chunk score by the number of distinct query words gave a scale where a line could be drawn:

(index of 169 notes, 2026-09-21; the 8 irrelevant queries are the 2026-09-05 experiment's ticket and its neighbours; irrelevant maximum 1.53, gold answers 1.06 to 6.15)

1.8 is the default. It loses five gold answers to admit nothing irrelevant, and that trade is deliberate: a relevant note the agent does not get costs one run some time; an irrelevant note it does get costs every run that reads it.

Backfilling the lesson table from the old run endings produced 16 rows on the first dry run: 3 drops, 1 staging hold, 12 not-ours verdicts. Those are the bot's own experience, and before this they were rows nobody read.

What is still unmeasured

Whether relevant injection changes outcomes. The 09-05 experiment cannot answer that (wrong notes), and the redesign has been running for a day. The measurement that would answer it is an A/B on an isolated clone, using real held runs, with first-attempt pass rate and tool-call count as the metrics. Until that runs, the honest status is: the retrieval now happens and is relevant by construction, and its value is a hypothesis.

Frequently Asked Questions

Why not make the search tool mandatory in the prompt?

Because "you must search" produces a search whose query the model chooses, and the 09-05 ranking already showed that ranking by ticket text pulls unrelated notes to the top. The fix is not to force the model to retrieve; it is to retrieve in code, with a threshold you have measured, and hand the model the result.

Doesn't the 20% tool-call reduction in the first table mean something?

It means unrelated text in the prompt changed the model's behaviour by about 20%, in the direction of fewer calls, on one ticket. Whether that is good depends on whether the calls it skipped were needed, and with no outcome gate at the time I cannot say. It is not evidence about memory.

How do you know the context section is actually in the prompt now?

A test runs each of the 8 entry points against a fake model binary that records the prompt it received, and asserts the line number of each marker: the tuning block, the lesson block, the memory block, and the first line of the original body. If a refactor moves the section or drops an entry point, that test fails before the bot does.

What counts as a lesson?

Only what code can write at a run's ending without a model: the QA hold reason, the drop reason, the re-fix note, the verification result, the not-ours verdict. A person's requeue note is not a lesson; it is read from its own table. Old transcripts are never injected, because they contain secrets and personal data the bot has no business re-reading.