slowbench
SearchAbout
RetrievalInferenceEvalsAgentsFindingsSeriesBenchmarksArchive

Reading

Series

The posts here are not independent. Several are two halves of one investigation, and a few only make sense read in order. These are the threads.

Three runs, one boundary

What reranking can and cannot do

A reranker reorders a list someone else built. That single fact decides which problems it can touch and which it cannot, and it is the difference between a week well spent and a week spent swapping models that could never have helped. These three posts find the boundary, then find out why the reranker inside it was failing anyway.

  1. 1Reranking cannot find what retrieval missed
  2. 2Your reranker speaks English. Your tickets do not.
  3. 3Fusing two rankers made routing worse, and the reason is structural

The lever nobody reaches for

What you let into the index

Model comparison is the visible knob — there are model cards to read and leaderboards to check, and it feels like progress. The largest measured improvement in this whole project came from deleting 3% of a corpus. These posts are about the decisions made before any model runs.

  1. 1Three percent of my index was eating the top five results
  2. 2Picking an embedding model when the queries and the code are in different languages
  3. 3Repository routing is the retrieval problem nobody benchmarks
  4. 4Measure the index you actually serve
  5. 5The scores spread out 15 times wider and the ranking did not move
  6. 6I could not measure the threshold where it mattered, until somebody could

Four wrong measurements, in order

The GPU investigation

An afternoon's work that took a week, because the first three measurements were of things I had set up badly. Read in order: the mistakes are more transferable than the result, and each one looked reasonable while I was making it.

  1. 1I measured Metal wrong three times before it beat the CPU
  2. 2The GPU was not slow, it was waiting
  3. 3Bigger batches made my GPU slower
  4. 4What running embeddings locally actually costs
  5. 5The flag was set and the feature was not there

Five ways a number can mean nothing

How benchmarks lie

Every post here is the same shape: a measurement that looked fine, a system that looked like it was working, and a reason the number could not have told the difference. A failing test announces itself. A meaningless test does not — that asymmetry is why these took weeks to find.

  1. 1The baseline that beat my agent was answering the same thing every time
  2. 2My agent benchmark was reading the answer off disk
  3. 3Ground truth from git history answers a different question than you asked
  4. 4Flaky and consistently wrong are different bugs with the same score
  5. 5Measure the index you actually serve
  6. 6A tie at four hundred queries was a loss at fifteen hundred
  7. 7The span that answered a question it never measured
  8. 8I fixed the code three times and never checked the symptom
  9. 9Three ways I broke one benchmark, and who caught each one
  10. 10My router scored 100% and a fixed answer scored 40%
  11. 11My benchmark had one slot left, so every change measured zero
  12. 12Five ways to measure the wrong thing, and each one was inside the fix for the last
  13. 13Recall was exactly 1.0000 four times, and that was the measurement telling me

One session, eighty days

Where agent context goes

A coding agent stopped answering for nineteen minutes. The investigation that followed found that context is not spent gradually on conversation — it goes in jumps, and the jumps come from a handful of commands nobody remembers running.

  1. 1An 80-day agent session and the wall at the end of it
  2. 2Context is burned by tool output, not by conversation
  3. 3Building the guardrails I needed after a session died
  4. 4I instrumented everything except where the decisions happen
  5. 5LangGraph yes, LangChain no
  6. 6The numbers halved and the fix never ran

One failure, read three times

Routing a ticket to a repository

Twenty-eight repositories and a ticket that belongs to one of them. Fusing two rankers lost, then lost again, then won — with nothing about the fusion changed. Read in order, these are the same event from three distances: the mechanism, the verdict I drew from it, and what the verdict was actually a specification for.

  1. 1Fusing two rankers made routing worse, and the reason is structural
  2. 2Repository routing is the retrieval problem nobody benchmarks
  3. 3The biggest repository wins, and it has nothing to do with relevance
  4. 4The one case it fixed and the ones it broke
  5. 5The index found 39 percent, the agent found 96, and the useful question was neither

Two days on a customer-facing search

What a zero-result rate is counting

A metric that looks like a defect count, decomposed. Most of the volume turned out to be demand for products that were never in the catalogue; the slice that really was a search defect came down to one multiplication that could not reach its own cutoff, and then to one suffix that made a place name match nothing at all.

  1. 1Eighty-three percent of my zero-result searches had nothing to find
  2. 2A perfect title match had to be popular to count
  3. 3Stripping one suffix fixed a quarter of my dead queries
  4. 4The error said no such column, and the column was a fragment of a user's query

Three baselines that were not constants

When the measurement moved

A baseline is supposed to be the thing you compare against. In one week three of them turned out to be measurements in their own right — a trace whose timings were synthesised, an evaluation set that changed under a re-collection, and an index whose scores depended on which folder it was built in. Different systems, different causes, the same symptom.

  1. 1A tie at four hundred queries was a loss at fifteen hundred
  2. 2Same bytes, a different folder, a different score

Two instruments, one of them lying

Where the agent's time goes

An agent that takes twelve minutes is not twelve minutes of work. Most of it is the model deciding what to do next, and the trace that is supposed to show you this can be computing the numbers rather than recording them. These posts open the instrument first, then use what survives to find out where the time actually sits — and why the intervention that helped did not make anything faster.

  1. 1The span that answered a question it never measured
  2. 2The median barely moved and the spread collapsed