I set up tracing on my agent pipeline because a routing decision was coming out differently on repeated runs and I wanted to see why. Self-hosted collector, SDK wired in, dashboard up, spans arriving.
Then I went to look at a specific decision and there was nothing there. Not a gap: zero traces from the part I had built the whole thing to observe.
Where the spans were
The pipeline has a shape that turns out to be common:
I had integrated the tracing SDK with the graph runtime, because that is where the framework's callback interface lives and the documentation shows you how. It worked exactly as advertised, tracing every node transition, every retry, every state update.
But the graph runtime does not make decisions. It moves work between things that do. The three components that produce judgements (the analysis step, the QA step, and the agent's own loop) all sit below the layer I had instrumented, and each one shells out to a separate process that reported to nobody.
My dashboard was a detailed record of the plumbing wrapped around a black box.
The instrumentation followed the SDK, not the questions
I want to name why this happened, because it wasn't laziness. It was following the documentation.
Framework SDKs integrate where the framework has hooks. That's the natural seam, it's what the getting-started guide covers, and it produces a dashboard full of spans within an hour. Everything about the experience says you are done.
What nobody asks at that point is: does the layer with the hooks contain the behaviour I care about?
For orchestration questions (is anything stuck, are retries firing, how long does a stage take) the framework layer is exactly right. For judgement questions, it is one or two levels too high. And the pipeline I built had every judgement in a subprocess, which is the single most common place for a trace to stop.
What I actually needed
A routing decision that alternates between three repositories and none is not diagnosable from timing. What I needed for each decision:
None of that was in the dashboard. All of it was in the subprocess.
Instrumenting the subprocess
The fix was to trace where the work happens rather than where the framework can see it. In practice, that meant every place a decision-making process starts writing its own span:
Two things made this less painful than it sounds.
The tracing protocol is not the SDK. My collector accepts OTLP over HTTP, which means anything that can make an HTTP request can emit a span. The subprocess does not need the framework, or the same language, or a shared library. Just the endpoint and a trace ID.
The trace ID is the whole integration. Pass the parent trace ID into the subprocess as an environment variable, have it emit spans against that ID, and the collector reassembles the tree. That's the entire mechanism, and it's what turns four disconnected process traces into one story.
The version of this I'd recommend to anyone starting: get one span out of your innermost component before you integrate anything at the top. If the inner span cannot reach the collector, no amount of framework integration will fix it, and you'll have built a dashboard of the wrong thing first.
The upgrade that hid it for a while
A detail that cost me an evening and is worth passing on.
The collector had moved to a major version that accepts only the wire protocol, and the older ingestion API it used to support was gone. My SDK integration had been written against the old one.
The failure mode is the part that matters: nothing errored. The SDK sent, the endpoint returned a response, no exception surfaced anywhere. Spans simply did not appear.
So for a stretch I was debugging "why are there no traces from the analysis step" while there were also no traces from anywhere, for a completely unrelated reason. Two independent problems presenting as one symptom, and I spent the evening on the wrong one because the coverage gap was the explanation I already had in mind.
What would have caught it in a minute:
Zero from that means the transport is broken. A number means the transport works and your problem is coverage. I now run it before investigating any tracing question, because the two causes are indistinguishable from inside the dashboard and the fix for one does nothing for the other.
What it showed immediately
The alternating decision had been my working example of instability, and I had a story about it: model nondeterminism, temperature, sampling. The traces said otherwise.
The runs that answered "none" had opened the repository, looked at the relevant area, and found the change already present. They were reporting that there was nothing to do, correctly.
The runs that answered "three repositories" were the ones matching my answer key, which was built from git history and therefore recorded what someone changed months ago, when the work was live.
So the "instability" was two different correct answers to two different questions, and my scoring counted one of them as a failure. I had been tuning prompts against that.
One trace, one afternoon, and the entire premise of a week's work turned out to be wrong. That is the argument for instrumenting judgement rather than orchestration, more than any principle about observability.
Two failure modes that look identical without traces
The other thing that fell out of this: I had been treating all inconsistency as one problem, and it is two.
Varies between runs. Same input, different answers. That's sampling, or genuinely ambiguous input, and prompt changes will move it around without fixing it. Some of my cases here were not even errors. They were the correct answer disagreeing with a stale label.
Wrong the same way every time. Same input, same wrong answer, four runs out of four. That is not probabilistic at all. It's a rule the system has learned or been given, and it is fixable by changing the instruction rather than by resampling.
Before traces, both appeared in my metrics as a lower accuracy number, and I attacked both with prompt tweaks. Only one of them responds to that. Telling them apart requires seeing the reasoning, which requires the traces to reach the component that reasons.
Cost, since it comes up
Tracing an agent means storing prompts and responses, which are much larger than typical spans. A single analysis step in my pipeline carries a prompt of a few thousand tokens and a response of a few hundred, and a full run touches that several times.
Two things keep it manageable.
Store the text, not the embeddings or the file contents. The prompt as sent is the thing you need to reconstruct a decision. The 400 KB of file content the agent read to build that prompt is reachable from the repository at the same commit, and storing it in the trace turns a useful record into an archive nobody opens.
Sample orchestration, keep judgement. Retry loops and queue transitions are high-volume and low-information after the first few examples. Decisions are the opposite: low volume, and you want every one. Sampling uniformly across a pipeline throws away the expensive-to-reproduce spans to save room for the cheap ones.
I self-host the collector, so the cost is disk rather than a per-span bill. If you're paying per span, the second point matters more: the default of "trace everything at the top" is both the least useful and the highest volume part of an agent pipeline.
The check that would have caught it
A dashboard that is up and green tells you spans are arriving from somewhere. It does not tell you they are arriving from the component you care about.
Query for a trace by that component's name and confirm the result is not empty. Mine was green for weeks while the part I had built the tracing for emitted nothing at all.