Datadog LLM Observability

Datadog LLM Observability extends the established Datadog monitoring platform to cover AI agents and LLM applications. It provides end-to-end tracing across multi-agent workflows, token-level cost tracking, built-in quality and security evaluations, and cross-correlation with traditional infrastructure metrics — all within the same Datadog dashboard teams already use for APM and infrastructure monitoring.

The core capability is LLM span tracing. Every LLM call in your application generates a span that captures the prompt, completion, token counts, latency, model parameters, and estimated cost. These spans integrate with Datadog's existing APM traces, so you can see exactly how an LLM call fits into a broader request flow — from the user's HTTP request through your application logic, into the LLM call, and back. For multi-agent systems, this means full visibility into how requests flow through different agents, which agent made which LLM calls, and where bottlenecks occur.

Built-in evaluations run automatically on LLM spans to detect quality and security issues. These include prompt injection detection, toxic content identification, off-topic completion flagging, and custom evaluation rules you define for domain-specific quality metrics. The evaluations run server-side within Datadog, so there's no additional latency in your application.

Cost tracking calculates estimated costs per span using providers' published pricing models and the token counts from each call. You can break down spending by model, agent, team, or any custom tag, and set alerts when costs exceed thresholds. This is particularly valuable for multi-agent systems where costs can be difficult to attribute.

The platform supports all major LLM providers including OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and Google Vertex AI. Integration uses the Datadog tracing SDK or OpenTelemetry with GenAI Semantic Conventions. Auto-instrumentation can detect and trace LLM calls without manual code changes in many frameworks.

Pricing is span-based — you pay per LLM span ingested, on top of your existing Datadog infrastructure costs. This can escalate quickly for high-volume AI applications. Some users report costs around $120/day when LLM observability auto-activates on busy applications. The auto-activation behavior (LLM observability turns on automatically when LLM spans are detected) has caught some teams off guard with unexpected bills.