Topic
llm
5 posts
- Evals9 min
Low effort cut agent turns by two thirds. Accuracy, I could not tell
Claude effort level on 40 tickets, three arms: low effort used fewer turns on 21 of 21 paired tickets, and on accuracy the twin arms disagreed with each other as much as with it.
- Evals9 min
Flaky and consistently wrong are different bugs with the same score
LLM evaluation: flaky and consistently wrong both show up as reduced accuracy. One responds to prompt changes and the other does not, and telling them apart takes one extra run.
- Agents8 min
I instrumented everything except where the decisions happen
LLM agent observability with a connected, green tracing dashboard, and zero traces from the three components that actually make choices.
- Agents10 min
Context is burned by tool output, not by conversation
Claude Code context usage over a two-month transcript: the largest entries were single command results, around 300,000 tokens each.
- Agents9 min
An 80-day agent session and the wall at the end of it
Claude Code context limit on an 80-day session: fifty compactions, each starting from a full million tokens. Then one morning it stopped answering for nineteen minutes.