Measure first. Then decide.
Most of what I believed about my own systems did not survive measurement.
Notes on retrieval, evaluation, and the systems around them — measured before written. Every number here comes from a run against the system that actually serves traffic — not a copy made for the benchmark. Where a result rests on one run, it says so.
- Posts
- 50
- Words
- 77,902
- Sections
- 4
Where things stand
All 9 →- How benchmarks lie13 posts
A benchmark fails silently far more often than it fails loudly, and the failure is almost never in the arithmetic. Across thirteen cases the number was correctly computed every time: what was wrong was the answer key, the population, the headroom, the baseline, or which artifact was measured. The cheapest defence found so far is a second number that must agree.
- What you let into the index6 posts
An index is a decision about what is searchable, not a copy of a repository, and the decision is usually made by accident. What hurt in every case was text nobody would ever search for. But the size of that harm did not generalise: the same proportion of prose did different jobs in different repositories.
- Where agent context goes6 posts
Context fills from tool output, not from conversation, and the cost is paid on every subsequent turn rather than once. Continuing a session feels free at each individual moment, which is why the ceiling arrives without warning.
- The GPU investigation5 posts
Every apparent GPU problem here was a plumbing problem, not a capacity one, and the two want opposite responses. Twice the accelerator was idle while something upstream fed it badly, and once it was not running at all despite being configured.
- Routing a ticket to a repository5 posts
Picking the right repository is a retrieval problem with almost no published benchmarks, and the obvious signals do not work on it. Absolute similarity scores failed three separate times as a basis for any threshold. Size bias is real and survives the usual normalisations.
Retrieval
16Code search, embeddings, and ranking.
Inference
5Local inference, Metal/GPU, and batch tuning.
Evals
16Measurement methodology and benchmark design.
Agents
13Coding agents and context management.
Nothing scheduled. The next measurement decides.
42 in the last 42 days · 0 scheduled
Recent
All 50 →- Agents7 min
The child process inherited nothing, so the test now walks every spawn site
Child process environment variables: config moved to a table loaded by each entry point's first import, and three scripts the bot spawns were not entry points. A test now walks spawn sites.
- Agents6 min
Four traps when merging parallel agent worktrees into one trunk
Git worktree for parallel agents: sessions edit one repo in separate worktrees and a person merges by cherry-pick. The traps are the tests, the classifier, patch-ids and the leftovers.
- Agents5 min
My LLM tracing store was 138 GB. The traces were 3.2 MB of it
ClickHouse disk usage under self-hosted Langfuse: 117 GB of it profiling its own queries. The fix sat unapplied for 18 days, and the table was too big to drop.
- Agents6 min
A restart sent 30 direct messages in 8 seconds. Then --help sent the daily report three times
Telegram bot duplicate messages, twice in three days: code that reads the live database and talks to a person is only as safe as the rows it finds, and DM is not a fallback.
- Agents6 min
The approval queue said nine fixes were waiting. Eight were already in production
Human in the loop agent approvals: the bot's pending list was its own table while people merged the same branches by hand. The fix was a check at the moment of speaking, not a sync job.