Agent observability: why request logs aren't enough

Most teams instrument an agent the way they instrument a REST endpoint: log the request, log the response, log the latency. That’s enough to know the model answered. It tells you almost nothing about what the agent did — which tools it called, why it called them in that order, where it re-read the same context three times, or which step in a five-hop chain actually produced the wrong answer.

Agents fail in the middle, not at the edges. Debugging them from request/response logs is like debugging a distributed system from the load balancer’s access log.

What a request log misses

A single agent turn can involve a planning step, several tool calls, a retrieval pass, and a final synthesis — each with its own token cost, latency, and failure surface. Flatten that into one log line and you lose:

• Which tool call caused the retry. “The agent took 8 seconds and cost $0.11” doesn’t tell you it called the search tool three times because the first two queries returned empty results.
• Where tokens actually went. Input tokens dominated by a bloated system prompt look identical, in a request log, to input tokens dominated by a runaway conversation history.
• Causality between steps. If step 4 hallucinated a field, was it because step 2’s retrieval missed a document, or because step 3’s tool output was malformed and the model guessed?

You need spans, not lines.

The shape of a useful trace

Treat each agent run as a trace with nested spans: one span for the overall run, one per model invocation, one per tool call, one per retrieval. This isn’t a novel idea — it’s how distributed tracing has worked for web services for a decade — but agent-specific tooling has been catching up. OpenTelemetry’s GenAI semantic conventions (now past v1.41) formalize this: agent, workflow, tool, and model spans, each carrying token counts and latency as first-class attributes, with LangChain, CrewAI, and similar frameworks emitting OTel-compliant spans natively or via instrumentation packages (OpenTelemetry, Greptime).

run: "answer support ticket #4821"
├─ span: model.plan          (1.2s, 340 tokens)
├─ span: tool.search_kb      (0.4s, query="refund policy")
├─ span: tool.search_kb      (0.4s, query="refund policy eu")   ← retry, empty first result
├─ span: model.synthesize    (2.1s, 890 tokens, cache_hit=0.6)
└─ span: tool.send_reply     (0.2s)

Once you have that shape, the questions you actually care about become queries instead of guesswork: which tool has the highest retry rate, which step contributes the most tokens on the median run, which trace shapes correlate with user-flagged bad answers.

What to capture on every span

• Inputs and outputs, truncated but not omitted — the first debugging step is almost always “what did the model actually see.”
• Token counts split by cache status. A cache-miss-heavy trace and a cache-hit-heavy trace can have identical latency and wildly different cost; conflating them hides your biggest lever for cost control.
• Tool name, arguments, and result status — not just “tool call succeeded,” but whether it returned zero results, an error, or a truncated payload.
• A stable run ID that ties every span back to the originating user request, so a support engineer can go from “user complained” to “here’s the trace” without grep.

What this buys you that logs don’t

Trace data lets you answer questions before a user files a ticket: which tool is silently degrading, which prompt version regressed retry rate, whether a new model version changed the average number of tool calls per run. None of that is visible in a log line that says POST /agent/run 200 4.3s.

Where to start

You don’t need a full observability platform on day one. Start by giving every tool call and model invocation a span with a shared trace ID, emit token counts per span instead of per run, and store enough of the input/output to reconstruct what happened without re-running the agent. Adopting OTel’s GenAI conventions early — even partially — means you’re not inventing your own schema that you’ll have to migrate off later, and it makes vendor dashboards (Datadog, Honeycomb, New Relic all already parse these attributes) usable out of the box instead of requiring custom parsers.

The teams that ship reliable agents aren’t the ones with the best prompts. They’re the ones who can look at a bad run and know, in under a minute, exactly which span went wrong.