Observability for AI Agents, Building Dashboards with Prometheus and Grafana

Agents fail while looking healthy. Here are the four metric layers, the PromQL, and the dashboards that surface it.

There is a specific failure that makes agent monitoring different from every dashboard you have built before. An agent can return HTTP 200, respond in under a second, throw zero exceptions, and be completely wrong, and your existing dashboards will show that incident as a healthy green wall. It might have looped through the same tool nine times, selected the wrong tool entirely, or produced a well formed answer built on nothing. Researchers describe this bluntly: agent failures often wear the appearance of success.

That gap has a name worth keeping. Structural telemetry answers whether the system is running, which is what request rate, latency, and error rate have always told you. Semantic telemetry answers whether the system is making sound decisions, and it requires metrics that describe reasoning: how many steps a task took, which tools were chosen, how many tokens were spent reaching an outcome, and whether the outcome was actually achieved. This tutorial builds both layers with OpenTelemetry, Prometheus, and Grafana, using the standard vocabulary rather than a proprietary one.

What observability for AI agents means

Observability for AI agents is the practice of instrumenting agent systems so you can answer both whether they are running and whether they are reasoning well, by collecting structural signals such as latency and error rate alongside semantic signals such as token consumption, tool selection, steps per task, and outcome success, then correlating them through traces and aggregating them into metrics and dashboards.

The word correlating matters. Metrics tell you that steps per task jumped this afternoon. Traces tell you which reasoning path caused it. You need both, and they have different jobs, which turns out to be the key to avoiding the cardinality mistake covered later.

Why standard conventions are worth the discipline

OpenTelemetry graduated from the CNCF on 21 May 2026, and in the preceding year its JavaScript API package passed 1.36 billion downloads while Python passed 1.3 billion. Alongside that, the GenAI special interest group has been standardizing gen_ai.* attributes covering model calls, agent invocations, workflows, and tool calls, plus required latency and token usage metrics.

Instrumenting against those conventions rather than a vendor SDK buys you one specific thing: if you change backends later, you re point an exporter instead of re instrumenting every agent. That is worth real discipline now.

One honest caveat before you build on it. Most gen_ai.* attributes still carry Development stability badges, so an attribute you depend on can be renamed with no major version bump signalling the change. The mitigation is the OTEL_SEMCONV_STABILITY_OPT_IN environment variable, which emits both legacy and current attribute names during a transition so dashboards do not break the day you upgrade an instrumentation library.

The four metric layers

Most teams instrument layers one and two, then wonder why their dashboards never predicted an incident. The value is concentrated in three and four.

Step 1: Instrument with OpenTelemetry

Auto instrumentation gets you the model layer with almost no code, and then you add the agent layer by hand because only you know what a task means in your system.

from opentelemetry import metrics, trace
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.sdk.metrics import MeterProvider
from prometheus_client import start_http_server

# Prometheus scrapes this endpoint. Start it early in the process lifecycle.
start_http_server(9464)
metrics.set_meter_provider(MeterProvider(metric_readers=[PrometheusMetricReader()]))
meter = metrics.get_meter("agent.observability")
tracer = trace.get_tracer("agent.observability")

# Explicit histogram buckets. Defaults are tuned for HTTP latency, not for
# step counts or token volumes, so set them deliberately.
steps_per_task = meter.create_histogram(
    "agent_steps_per_task",
    description="Tool calls executed before the agent reached an answer",
)
tokens_used = meter.create_counter(
    "agent_tokens_total", unit="token", description="Tokens consumed")
tool_calls = meter.create_counter(
    "agent_tool_calls_total", description="Tool invocations by name and status")
task_outcomes = meter.create_counter(
    "agent_task_outcomes_total", description="Terminal outcome of each task")

Now wrap the agent invocation. The span carries high cardinality detail, and the metrics carry only low cardinality labels, which is the split that keeps Prometheus healthy.

AGENT = "k8s-troubleshooter"

def run_task(task_id: str, prompt: str, agent) -> dict:
    """One agent task, fully instrumented."""
    with tracer.start_as_current_span("invoke_agent") as span:
        # High cardinality context belongs on the SPAN, never on a metric label.
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("gen_ai.agent.name", AGENT)
        span.set_attribute("agent.task.id", task_id)

        result = agent.invoke(prompt)
        steps = len(result.get("tool_calls", []))

        # Low cardinality labels only: agent name, model, tool name, status.
        base = {"agent": AGENT, "model": result["model"]}
        steps_per_task.record(steps, base)
        tokens_used.add(result["usage"]["input_tokens"], {**base, "direction": "input"})
        tokens_used.add(result["usage"]["output_tokens"], {**base, "direction": "output"})

        for call in result.get("tool_calls", []):
            tool_calls.add(1, {**base, "tool": call["name"],
                               "status": "error" if call.get("error") else "ok"})

        outcome = classify(result)          # "success" | "escalated" | "budget_exhausted"
        task_outcomes.add(1, {**base, "outcome": outcome})
        span.set_attribute("agent.outcome", outcome)
        span.set_attribute("agent.steps", steps)
        return result

The cardinality trap that will take down your Prometheus

This is the mistake that turns a monitoring project into an outage, and it is worth an entire section because it is so easy to make.

Prometheus creates a separate time series for every unique combination of label values. A label with a thousand possible values multiplies your series count by a thousand. Now consider what is tempting to attach to an agent metric: the prompt text, the completion, a user identifier, a session identifier, a task identifier, a trace identifier. Every one of those is effectively unbounded, and adding any of them to a metric label will inflate your series count without limit until the database falls over.

The rule is simple and absolute. Metrics get low cardinality labels only, meaning things with a small fixed set of values such as agent name, model name, tool name, direction, and outcome. Everything unbounded goes on spans and into your tracing backend, which is built for exactly that. When a metric shows you a spike, you pivot to traces to see the individual cases behind it, and that division of labour is the whole architecture.

Redaction deserves a mention alongside this. If you capture prompt or completion content on spans, run it through a redaction processor in the collector before storage, because prompts routinely contain customer data that your telemetry retention policy was never designed to hold.

Build the Prometheus foundation this assumes

Everything below is PromQL, recording rules, and dashboard design, which are learnable skills independent of AI. The Prometheus Certified Associate (PCA) prep course on KodeKloud covers observability fundamentals, PromQL, exporters, instrumentation, recording rules, alerting, and Kubernetes monitoring across roughly twelve hours of labs, which is the exact toolkit this tutorial applies to agents.

Step 2: PromQL that reveals agent behavior

Here are the queries that earn their place on a dashboard. Each one answers a question the infrastructure layer cannot.

# 1. Reasoning loop detector. A rising p95 step count means the agent is
# working harder for the same answers, which is the earliest quality signal
# you get and it precedes user complaints by hours.
histogram_quantile(0.95,
  sum by (le, agent) (rate(agent_steps_per_task_bucket[15m])))

# 2. Tool selection drift. Watch the SHAPE of this, not the total. A tool that
# was 5 percent of calls last week and is 40 percent today means the agent
# changed its mind about something, usually after a prompt or model change.
sum by (tool) (rate(agent_tool_calls_total[1h]))
  / ignoring(tool) group_left sum(rate(agent_tool_calls_total[1h]))

# 3. Tool failure rate per tool. A broken tool shows up here long before it
# shows up as a user visible failure, because agents route around errors.
sum by (tool) (rate(agent_tool_calls_total{status="error"}[10m]))
  / sum by (tool) (rate(agent_tool_calls_total[10m]))

# 4. Spend rate in dollars per hour, computed from tokens. Prometheus has no
# native cost metric, so you multiply by your price per million tokens.
sum by (model) (rate(agent_tokens_total{direction="input"}[5m])) * 3.00 / 1e6 * 3600
  + sum by (model) (rate(agent_tokens_total{direction="output"}[5m])) * 15.00 / 1e6 * 3600

# 5. Cost per SUCCESSFUL task. The metric that changes decisions, because an
# agent that retries its way to answers looks cheap per call.
(
  sum(rate(agent_tokens_total{direction="input"}[1h])) * 3.00 / 1e6
  + sum(rate(agent_tokens_total{direction="output"}[1h])) * 15.00 / 1e6
)
/ sum(rate(agent_task_outcomes_total{outcome="success"}[1h]))

# 6. Silent failure rate. Tasks that ended without success, split by how they
# ended. Escalations are healthy, budget exhaustion is not.
sum by (outcome) (rate(agent_task_outcomes_total{outcome!="success"}[30m]))
  / sum(rate(agent_task_outcomes_total[30m]))

Query five is the one to put in front of leadership. Cost per call flatters an agent that retries, because each cheap call looks efficient while the task consumes ten of them. Cost per successful outcome is the number that actually tracks value delivered, and it moves in the right direction when you improve prompts or tool design.

Wrap the expensive expressions in recording rules so dashboards stay fast.

groups:
  - name: agent-observability
    interval: 30s
    rules:
      - record: agent:steps_p95
        expr: histogram_quantile(0.95,
                sum by (le, agent) (rate(agent_steps_per_task_bucket[15m])))

      - record: agent:spend_dollars_per_hour
        expr: |
          sum by (model) (rate(agent_tokens_total{direction="input"}[5m])) * 3.00 / 1e6 * 3600
          + sum by (model) (rate(agent_tokens_total{direction="output"}[5m])) * 15.00 / 1e6 * 3600

      - record: agent:success_ratio
        expr: sum by (agent) (rate(agent_task_outcomes_total{outcome="success"}[30m]))
                / sum by (agent) (rate(agent_task_outcomes_total[30m]))

Keep your token prices in the recording rule rather than scattered across panels, so a provider price change is one edit rather than an archaeology exercise.

Step 3: Design the dashboard around questions

A dashboard organized by data source tells you nothing. Organize it by the question a responder asks, in the order they ask it.

Two panel choices make a real difference in practice.

Use a heatmap rather than a line for steps per task. An average of four steps can hide a bimodal distribution where most tasks take two and a stubborn tenth take fifteen, and the heatmap shows that split immediately while the line hides it completely.

Stack tool call share as a percentage rather than plotting absolute counts. You care about shifts in which tools the agent reaches for, and absolute counts move with traffic, which masks exactly the change you want to see.

Practice on a live Prometheus and Grafana stack

Dashboard design is a hands on skill, and the fastest way to build it is with a real stack and real metrics to query. The Prometheus and Grafana playground on KodeKloud gives you both running and ready, so you can paste the PromQL above, build the panels, and break things without any setup work.

Step 4: Alert on semantic failure, not just downtime

Your existing alerts catch outages. These catch an agent that is quietly getting worse.

groups:
  - name: agent-semantic-alerts
    rules:
      # Reasoning loops. The agent needs far more steps than it used to.
      - alert: AgentStepCountRegression
        expr: agent:steps_p95 > 12
        for: 20m
        labels:
          severity: warning
        annotations:
          summary: "Agent {{ $labels.agent }} p95 step count is {{ $value | printf \"%.1f\" }}"
          description: "Rising steps per task usually follows a prompt, model, or tool change."
          runbook_url: "https://runbooks.internal/agent-step-regression"

      # Hitting the step ceiling means tasks are being abandoned, not solved.
      - alert: AgentBudgetExhaustionHigh
        expr: |
          sum by (agent) (rate(agent_task_outcomes_total{outcome="budget_exhausted"}[30m]))
            / sum by (agent) (rate(agent_task_outcomes_total[30m])) > 0.05
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Over 5% of tasks exhausted their step budget"
          description: "The agent is giving up rather than answering. Check tool health first."

      # Spend anomaly. Token theft and a retry loop look identical on a bill.
      - alert: AgentSpendSpike
        expr: |
          agent:spend_dollars_per_hour
            > 2 * avg_over_time(agent:spend_dollars_per_hour[7d])
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Spend for {{ $labels.model }} is double its weekly average"

      # A tool that fails silently gets routed around, so watch tools directly.
      - alert: AgentToolFailureRate
        expr: |
          sum by (tool) (rate(agent_tool_calls_total{status="error"}[10m]))
            / sum by (tool) (rate(agent_tool_calls_total[10m])) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Tool {{ $labels.tool }} is failing on over 10% of calls"

The step budget alert is the one that will earn its keep. When an agent starts exhausting its budget, users experience it as unhelpful answers rather than as errors, so nothing else in your stack will tell you, and the cause is usually a broken tool the agent has been quietly routing around.

What to set as your first SLOs

Start conservative, measure for a fortnight, then tighten. These are reasonable opening positions rather than universal truths.

A note on traces and MCP

Two things trip up teams extending this to distributed agents.

Trace context does not propagate automatically from an LLM inference into tool calls made over the Model Context Protocol, so you have to inject the context into the tool call headers yourself. Without that step your traces fragment into disconnected pieces, and the reasoning path you most wanted to reconstruct is the part that breaks. The conventions do define MCP method coverage and histogram metrics, so the vocabulary exists once you handle propagation.

Async work has the same problem for the same reason. If tasks are dispatched to a queue or a worker process, inject and extract trace context across that boundary or you lose continuity precisely where debugging is hardest.

Where to start

1. Expose a Prometheus endpoint from one agent service and confirm a scrape works before instrumenting anything interesting.
2. Add auto instrumentation for your model provider, which gives you tokens and model latency with almost no code.
3. Add the four custom metrics by hand: steps per task, tool calls, tokens, and task outcomes.
4. Audit your labels against the cardinality table and move every identifier onto spans, doing this before you have months of series to clean up.
5. Build the dashboard rows in question order and use a heatmap for steps per task.
6. Add the step budget and spend spike alerts, which are the two that catch failures nothing else sees.
7. Watch cost per successful task for two weeks before setting any target, so your first SLO is grounded in reality.

Conclusion

The reason agent observability needs its own playbook is not that the tooling is exotic. It is Prometheus, Grafana, and OpenTelemetry doing what they have always done. What changes is which signals matter, because the failures that hurt most never show up as errors. A looping agent, a misused tool, and a confidently wrong answer all pass every health check you already have.

So instrument the reasoning, not just the request. Steps per task, tool call distribution, task outcomes, and cost per success are four metrics you can add this week, and together they turn a green dashboard that tells you nothing into one that notices when your agent starts getting worse. Then keep identifiers out of your labels, because the fastest way to lose a monitoring project is to take down the database it runs on.

Ready to Master Prometheus and Grafana Properly?

Every technique here is standard observability practice pointed at a new workload, which makes the underlying skills the durable investment. The Prometheus Certified Associate (PCA) prep course on KodeKloud covers PromQL, instrumentation, recording rules, and alerting in depth, the Grafana Loki course adds the logging side of the stack, and the Prometheus and Grafana playground gives you both running instantly so you can practice on real queries. Start with one today.

FAQs