Agent Observability: The 4 Signals Your Stack Must Emit

Agent observability makes an agentic system legible after the fact. State, decisions, tool calls — captured, replayable, auditable.

The vocabulary is borrowed from distributed systems: traces, spans, W3C Trace Context. The bending happens when you apply it to language model calls, because those are non-deterministic.

That single property breaks most of what traditional observability assumes.

In a conventional distributed system, the same input produces the same path. You debug by finding the divergence from expected behavior. With an agent, two runs of the same request may legitimately take different routes — different tools, different order, different number of steps.

So the question shifts. Not “did this behave as specified” but “what did it actually do, and can I reconstruct why.”

That reconstruction requirement is why this article exists. It is the shared dependency underneath every AI control worth having, and it is the one most commonly assumed rather than built.

Key Takeaways

• Agent observability is the unstated dependency underneath every other AI control. Detection, attribution, incident response, compliance evidence and cost attribution all fail without it. The OpenTelemetry GenAI semantic conventions are widely described as standardized. They are not. As of 21 August 2026 the dedicated conventions repository marks them Development with no official release. Two of the most visible LLM observability brands were acquired within a single quarter. Instrument to the convention, not to a vendor. A tool that logs prompt and response pairs is shipping log search. Real agent observability emits spans with tool calls, correlation IDs and full reasoning chains. In the disclosed lab containment failures of mid-2026, two of three affected organisations had not detected the activity at all.

---

Quick Navigation

---

Why Agent Observability Breaks Old Assumptions

Four specific assumptions fail when you move from services to agents.

Fixed call graphs. Traditional tracing assumes a service topology you can draw. An agent decides its own path at runtime, so the trace shape is an output rather than a design artefact.

Errors as the signal. In conventional systems, failures throw exceptions. An agent can complete successfully while doing entirely the wrong thing. A 200 response tells you nothing about whether the action was correct or authorized.

Latency as the metric. Response time matters, but token consumption, tool-call count and reasoning depth matter more for both cost and correctness.

Sampling by volume. Standard practice samples a percentage of traffic to control cost. For agents, the interesting traces are the rare ones — the long chains, the unusual tool sequences, the sessions where something went sideways. Uniform sampling systematically discards them.

The practical consequence: an observability stack that works well for your microservices will produce confident-looking dashboards about your agents while missing the failure modes that actually matter. This is the distinction between systems that generate output and systems that take actions.

---

The Four Signals Agent Observability Must Emit

The OpenTelemetry GenAI conventions define a minimum span shape. Grouped by what they let you answer, four signals matter.

Signal 1 — Model calls. Which model, which provider, which operation. The canonical attributes are gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model and gen_ai.response.model. Requested and served model can differ under routing, and that difference is worth capturing.

Signal 2 — Token usage. gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, per call. This is the foundation of cost attribution, and per-call granularity is what allows cost per completed task rather than cost per month — the distinction that makes inference cost per token actionable.

Signal 3 — Tool calls. gen_ai.tool.name, plus arguments and results. Each tool invocation becomes a child span. This is the security-relevant signal: it is where an agent’s intentions become actions against real systems.

Signal 4 — Agent and correlation context. gen_ai.agent.name, gen_ai.agent.description, and W3C Trace Context correlation IDs binding the chain together across services and across agents.

The fourth signal is the one most implementations skip, and it is the one that makes the other three useful. Without correlation context you have a pile of individually well-formed spans and no way to reconstruct the sequence.

Agent-specific conventions covering tasks, actions, memory and agent-to-agent communication were drafted in 2025 and moved into experimental status through 2026. Framework-specific conventions for CrewAI, AutoGen, LangGraph and Semantic Kernel remain in active development.

---

The Agent Observability Standard Is Not Finished

This correction matters if you are planning around the standard, because a great deal of published material overstates its maturity.

The GenAI Special Interest Group has developed these conventions since April 2024. The semantic-conventions repository cut v1.40.0 in February 2026. In June 2026 the project moved GenAI, provider-specific and MCP conventions into a dedicated repository so they could version independently.

That repository marks the GenAI conventions as Development, and as of 21 August 2026 it has no official release. The gen_ai.* namespace remains experimental.

Two things follow, and they point in opposite directions.

Adopt anyway. OpenTelemetry itself graduated within CNCF in May 2026, which removes the project-maturity objection even if this particular namespace is unstable. Major vendors have already implemented — Datadog added native support in v1.37, Grafana collects LLM traces in Loki. The conventions are the closest thing to a neutral vocabulary that exists.

Pin your versions. Treat the conventions as a versioned contract rather than a stable API. Attribute names in an experimental namespace can change, and silent data breakage — where your dashboards keep rendering while the underlying field stops populating — is the failure mode to guard against.

The churn is concentrated at the edges: multimodal content, agent graphs, and MCP. Core model-call and token attributes are comparatively settled.

---

The Sorting Test for Agent Observability Tools

The term is widely misused, and there is a short test that sorts the market.

A tool that logs prompt and response pairs is shipping log search. That is a legitimate product and it is not agent observability. It cannot show you what tools were called, in what order, on whose authority, or where a chain went wrong.

Real agent observability emits OpenTelemetry-compatible spans with the GenAI conventions applied, supports multi-step trace reconstruction, and correlates across services.

Three questions to ask a vendor:

Do you emit OTel-compatible spans, or import them only? Import-only means you are locked in at the layer where portability matters.

Can I reconstruct a full agent chain, including sub-agent delegation? Multi-agent systems break most trace models, and this is where the gap shows.

What happens to my data if I switch backends? If the answer involves re-instrumenting your application, you instrumented to a vendor rather than a standard.

The platform landscape as of 2026 includes LangSmith from the LangChain team, Arize AI with its open-source Phoenix project, evaluation-first Braintrust, and AWS Bedrock AgentCore Observability. Datadog pulled LLM workloads into its existing APM footprint.

---

Why Every Control Assumes Agent Observability

Here is the argument that motivated this article. Take any AI control you have read about this year and follow it to its dependency.

Prompt injection defence assumes you can see what entered the model’s context and which tool call followed. Without per-call traces, a successful injection is indistinguishable from normal operation.

Containment. In the disclosed lab incidents of mid-2026, two of three affected organisations had not detected the activity at all. The evidence that reconstructed those events came from the labs’ own evaluation logs, not the victims’ monitoring — a pattern set out in how five labs lost containment.

Identity and attribution. Distinct agent identity only produces value if actions are logged against it. A perfect identity architecture with no trace layer answers “who could have done this” and never “who did,” as covered in why shared credentials are the real exposure.

Least privilege. Scoping permissions requires knowing which permissions are actually exercised. Teams without tool-call telemetry over-grant because they cannot see what would break.

MCP security. OWASP includes insufficient logging in its Top 10 precisely because most clients and servers log almost nothing by default.

Red teaming. Measuring attack success rates requires observing outcomes across many attempts. Without traces, you are counting your own attempts rather than measuring the system.

Compliance evidence. Contemporaneous, tamper-evident logs are the strongest available evidence tier. Policies and documentation rank below them.

Cost control. Cost per completed task requires per-trace token attribution. A monthly bill tells you what you spent, not whether it was productive.

The pattern is consistent. Research indicates roughly 47% of deployed agents are actively monitored, which is the constraint underneath the AI agent security gap. Every control in the list above is being recommended into environments where roughly half the agents emit nothing.

---

Agent Observability Costs You Have to Plan For

Three costs surprise teams, and each has a standard mitigation.

Storage volume. A single agent task can generate dozens of spans with full prompt and completion payloads. Traces are large compared to conventional application logs, and volume scales with reasoning depth rather than request count.

The mitigation is tiered retention: keep full payloads briefly, keep span metadata and structure far longer. Structure without payload still reconstructs the chain.

Evaluation cost. Running LLM-as-judge scoring over production traffic means paying for inference twice. Common practice samples 10–20% of traffic for evaluation, which balances quality coverage against spend.

Privacy exposure. Prompts and completions routinely contain personal data. Sending them to a third-party observability backend creates a data-protection question your instrumentation decision has already answered by default.

The mitigation is to sanitize at source — automated scrubbing inside the instrumentation wrapper, before the span leaves your process. Scrubbing at the backend means the data already crossed the boundary.

One design note worth stating plainly: sample for evaluation, not for observability. Evaluate a subset; trace everything. Dropping traces to save money removes exactly the rare sessions that justify having the system.

---

The Vendor Consolidation Problem

In January 2026 ClickHouse acquired Langfuse. In April, Cisco announced its intent to acquire Galileo. Two of the most visible LLM observability brands changed hands in a single quarter.

Neither acquisition is inherently bad for users. Both illustrate a structural risk.

Observability instrumentation is expensive to change. It touches every code path that calls a model or a tool. If your spans are emitted in a vendor-proprietary shape, switching backends means re-instrumenting the application — which in practice means you do not switch, and your negotiating position erodes accordingly.

The defence is straightforward and worth stating as a rule: instrument to the standard, treat the vendor as a detail.

Emit OpenTelemetry spans with GenAI conventions applied. Export via OTLP. Choose a backend that ingests that natively rather than one that requires a proprietary SDK. Then a backend change is a configuration change.

This argument gets stronger, not weaker, from the conventions being experimental. An unstable open standard you can version-pin is a better foundation than a stable proprietary schema you cannot leave.

---

What Agent Observability Cannot Do

A control worth having is worth stating the limits of, and three of these matter for planning.

Observability is detection, not prevention. A trace records that an agent deleted the records. It does not stop the deletion. Teams that instrument thoroughly and then treat the dashboard as a safeguard have bought visibility into harm rather than protection from it. Enforcement belongs at the gateway, in permission scoping, and in approval gates.

A complete trace does not mean a correct one. The hardest agent failures produce clean telemetry. Every span succeeds, every tool call returns 200, and the aggregate outcome is wrong — a specification-gaming failure where the agent pursued its objective through a route nobody intended. Nothing in the trace is flagged because nothing failed. Recognising this requires evaluating outcomes, not inspecting spans.

Volume defeats human review. An enterprise processing millions of agent actions cannot manually inspect traces. Without automated evaluation, anomaly detection and shape-based alerting layered on top, comprehensive tracing produces an archive nobody reads. The archive is still valuable after an incident; it does very little before one.

There is also a measurement subtlety worth naming. Instrumentation changes what it measures — capturing full prompt and completion payloads on every span adds latency and cost to the request path, and aggressive instrumentation of a latency-sensitive agent can degrade the experience you were trying to protect.

The workable position is layered. Trace comprehensively for reconstruction and evidence. Evaluate a sample for quality. Alert on behavioural shape rather than error status. And keep enforcement in a separate layer that does not depend on anyone reading a dashboard in time.

---

Building Agent Observability That Lasts

Six steps, ordered by dependency.

• Instrument every layer, not just the model call. Spans for LLM invocations, retrieval steps and tool calls. Attributing latency or cost to a step requires a span for that step. Auto-instrumentation packages exist for OpenAI, Anthropic, LangChain and LlamaIndex.
• Propagate correlation IDs across agent boundaries. W3C Trace Context is the mechanism. Without it, a multi-agent system produces disconnected traces and delegation chains cannot be reconstructed.
• Attribute every span to a distinct identity. Not a shared service account. This is what converts a trace into evidence.
• Define a single schema source of truth. One module holding every span name, attribute and metric, with no raw telemetry literals elsewhere in the codebase. A drift-detection test then catches divergence automatically.
• Sanitise before you store. Scrubbing belongs in the instrumentation wrapper, not the backend.
• Alert on shape, not just failure. An agent that suddenly takes twelve tool calls where it usually takes three has not errored. It has changed behavior, and behavioural drift is the signal that matters when success responses can accompany wrong actions.

A useful readiness check: pick a task your agent completed last week and reconstruct it end to end — which model, which tools in what order, how many tokens, who authorized it, what it touched. If you cannot, every control built on top of that agent is running on an assumption.

---

Primary sources

• OpenTelemetry semantic conventions repository
• OpenTelemetry GenAI documentation
• W3C Trace Context
• CNCF — OpenTelemetry graduation

Convention status reflects the dedicated GenAI conventions repository as of August 2026 and is changing actively. Verify current release status before relying on attribute stability.

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.