Event Driven AI Agents with Prometheus Alerts, From Page to Root Cause

Alertmanager can wake a human or it can wake an agent. Here is how to wire Prometheus alerts into an AI agent that investigates before you do.

Research from incident.io puts a typical enterprise team's alert volume above 2,000 per week, and only about 3 percent of those alerts require immediate action. The other 97 percent still interrupt a human, and most of them die in the same ten investigation steps: check the pod, query Prometheus, pull the logs, look at what deployed. A CNCF case study published in April 2026 put a number on that ritual, with a two person SRE team at STCLab measuring 15 to 20 minutes of manual correlation per alert before they wired an event driven AI agent to their Prometheus alerts to run that first pass. Multiply that ritual by every page in your queue, and you have found where your on call hours actually go.

That first pass is now automatable, and the trigger is infrastructure you already run. Prometheus evaluates the rules, Alertmanager routes the event, and instead of a webhook posting a bare notification to Slack, it wakes an agent that investigates and posts the findings. This guide covers the whole shape: what event driven AI agents with Prometheus alerts actually are, the architecture and code to wire one up, the guardrails that make it safe, and the CNCF tools that already do it.

What an event driven AI agent actually is

An event driven AI agent is a language model with investigation tools that is triggered by a system event, such as a Prometheus alert delivered through an Alertmanager webhook, rather than by a human typing a question. When the alert fires, the agent autonomously queries metrics, logs, and cluster state, correlates what it finds, and produces a root cause hypothesis with evidence, all before a person engages.

The distinction from a chat assistant is the trigger and the initiative. A copilot waits for you to notice the problem, gather the context, and paste it in. An event driven agent starts from the alert payload itself, decides which tool to call next based on each result, and works the loop the way an experienced responder would: pod is crashlooping, so check the exit code, so pull the logs, so look at the last deploy. AIOps platforms have correlated and deduplicated events for years, and this is the step beyond correlation, an agent that actually runs the investigation.

Why alert fatigue is the forcing function

On call economics broke quietly, one alert at a time, and the 2026 numbers describe a system past its limit.

The NeuBird State of Production Reliability report found that 77 percent of on call teams field ten or more alerts every day, while 57 percent say under a third of those alerts are actionable. Catchpoint's SRE survey work found nearly 70 percent of SREs saying on call stress has driven burnout and attrition on their teams, with the median share of time spent on operations climbing to 30 percent. None of these figures describe a discipline problem. They describe a volume problem that threshold tuning has not fixed in a decade of trying.

Alert triage is where the volume lands, and it is the least differentiated work an engineer does. The investigation steps are nearly identical for most alerts, the context is scattered across four tools, and the outcome for the majority is "known cause, no action" or "noisy rule." Handing exactly that layer to an agent is the point of the pattern: automated incident response for the first pass, humans for the judgment calls that remain.

The architecture, from alert to answer

The pipeline has four stations, three of which you already operate.

Prometheus evaluates your alert rules and fires when an expression holds for its duration. Alertmanager groups, deduplicates, and routes the alert exactly as it does today, except one receiver is a Prometheus Alertmanager webhook pointed at your agent instead of at a chat channel. The receiver accepts the payload, extracts the alert's labels and annotations, and starts an investigation with a bounded toolset. The agent posts its findings, a root cause hypothesis, the evidence for it, and a suggested next step, into the same Slack thread the alert notification landed in.

Wiring Alertmanager to the agent is a routing change, not a platform migration.

# alertmanager.yml: send critical alerts to the agent as well as to humans.
route:
  receiver: slack-oncall
  group_by: ["alertname", "namespace"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
      receiver: ai-agent
      continue: true          # the page still reaches the on call human

receivers:
  - name: slack-oncall
    slack_configs:
      - channel: "#oncall"
  - name: ai-agent
    webhook_configs:
      - url: "http://agent.sre.svc:8080/alerts"
        send_resolved: true
        max_alerts: 5         # cap the batch so one storm cannot flood the agent

The receiver itself is small. This version accepts the webhook, runs the investigation loop against an OpenAI compatible LLM endpoint, and posts the result, with every tool the model can use defined as a safe, parameterized function.

from fastapi import FastAPI, Request
import subprocess, json

app = FastAPI()

def promql(query: str) -> str:
    """Run a PromQL instant query, read only by nature."""
    out = subprocess.run(
        ["curl", "-sG", "http://prometheus.monitoring.svc:9090/api/v1/query",
         "--data-urlencode", f"query={query}"],
        capture_output=True, text=True, timeout=30)
    return out.stdout[:4000]

def pod_logs(namespace: str, pod: str) -> str:
    """Last 200 log lines. The ServiceAccount behind this is read only."""
    out = subprocess.run(
        ["kubectl", "logs", "--tail=200", "-n", namespace, pod],
        capture_output=True, text=True, timeout=30)
    return (out.stdout or out.stderr)[:4000]

def recent_events(namespace: str) -> str:
    out = subprocess.run(
        ["kubectl", "get", "events", "-n", namespace,
         "--sort-by=.lastTimestamp"],
        capture_output=True, text=True, timeout=30)
    return out.stdout[-4000:]

TOOLS = {"promql": promql, "pod_logs": pod_logs, "recent_events": recent_events}

@app.post("/alerts")
async def receive(req: Request):
    payload = await req.json()
    for alert in payload.get("alerts", []):
        if alert.get("status") != "firing":
            continue
        context = {
            "labels": alert.get("labels", {}),          # alertname, namespace, pod
            "annotations": alert.get("annotations", {}),# description, runbook_url
        }
        report = investigate(context, TOOLS, max_steps=8)   # the agent loop
        post_to_slack(thread_for(alert), report)
    return {"ok": True}

That investigate function is the standard agentic loop: hand the model the alert context and the tool schemas, execute each tool call it makes, feed the result back, and stop at a conclusion or the step budget. Eight steps is enough for genuine root cause analysis on most Kubernetes alerts, and the cap keeps both latency and token cost bounded.

Want to build this against a real cluster?

An agent pipeline only becomes intuitive once you have watched it chase a real crashlooping pod. The Prometheus Certified Associate (PCA) course on KodeKloud covers the alerting and PromQL foundations this whole pattern stands on, and the KodeKloud playgrounds give you a disposable Kubernetes cluster with Prometheus where you can break a deployment on purpose and watch your webhook receiver investigate it.

Give the agent tools, read only first

The toolset defines the blast radius, so start with observation and add nothing else until the reports have earned trust.

A useful investigation set is small: PromQL queries against Prometheus, pod logs and describe output, Kubernetes events, and recent deployment or rollout history, since a large share of production incidents trace back to whatever changed last. Every one of those is a read. Bind the agent to its own ServiceAccount with get, list, and watch only, keep secrets access out of the role entirely, and treat the alert payload's labels and annotations as data rather than instructions, since anything that flows into a prompt deserves that suspicion.

Auto remediation is the obvious next want, and the answer is a gate, not a jump. When the agent's diagnosis implies a fix, restart the deployment, roll back the release, scale the pool, it should propose the action in the Slack thread as a button a human clicks, with the actual execution done by a separate, narrowly scoped identity. SRE automation has always worked this way, runbooks first, supervised execution second, unattended automation only for actions that are proven reversible, and an LLM in the loop does not change that ladder. It just climbs it faster.

Write Prometheus alert rules an agent can act on

The quality of the investigation is capped by the quality of the event, which makes your Prometheus alert rules part of the agent's design.

An agent receiving HighErrorRate with no annotations starts from almost nothing. The same alert carrying a description with the current value, the affected service, and a runbook link starts the investigation three steps ahead, and the runbook is not decoration, since agents like HolmesGPT explicitly ground their reasoning in the runbooks you already wrote.

groups:
  - name: checkout.rules
    rules:
      - alert: CheckoutErrorRateHigh
        expr: |
          sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="checkout"}[5m])) > 0.05
        for: 10m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "Checkout 5xx error rate above 5% for 10 minutes"
          description: "Current error ratio is {{ $value | humanizePercentage }} on the checkout service."
          runbook_url: "https://runbooks.internal/checkout-error-rate"
          dashboard: "https://grafana.internal/d/checkout"

Two routing habits keep the pattern affordable and sane. Group related alerts in Alertmanager so a node failure produces one investigation instead of forty, because with an LLM behind the webhook, deduplication is now also cost control. Route only pages worth investigating, critical severity and the warning rules you actually trust, since pointing an agent at known noisy rules automates the noise instead of fixing it.

The maturity ladder for agentic alerting

Adopt this in rungs, and be honest about which rung you are on before reaching for the next.

Most teams should live at levels 1 and 2 for a quarter, measuring accuracy against what responders actually concluded, before wiring any button at level 3.

The tools already doing this in 2026

The build or adopt question is genuine here, because the CNCF ecosystem matured fast.

Where to start

1. Pick the three alerts that page most often and add real annotations to their Prometheus alert rules: description with the live value, runbook link, dashboard link.
2. Stand up the webhook receiver at level 1, enrichment only, and route one critical alert to it with continue: true so humans still get paged.
3. Compare the agent's enrichment against what the responder actually needed for two weeks, and tune the toolset from the gaps.
4. Move to level 2 investigations on Kubernetes alerts, or deploy HolmesGPT with the Alertmanager integration and skip straight there.
5. Add a step budget, a token budget, and per alert cost logging, so the pattern's economics stay visible.
6. Only then discuss remediation buttons, one action at a time, each behind its own scoped identity.

Conclusion

Alert fatigue was never really an alerting problem. It is an investigation problem, thousands of events per week each demanding the same manual first pass, and that first pass is precisely the repetitive, tool driven, pattern matching work that agents now do well. The infrastructure to adopt it is the Prometheus and Alertmanager you already run, the integration is one webhook receiver, and the safety story is the same least privilege discipline you apply to every other identity: read only tools, scoped ServiceAccounts, human gates on anything that mutates.

Start with one alert this week. Give it real annotations, route it to an enrichment agent alongside the human page, and measure whether the context it attaches would have saved your responder the first fifteen minutes. That number, multiplied by your weekly alert volume, is the business case, and for most teams it is not a close call.

Ready to Build the Observability Skills This Runs On?

An agent is only as good as the alerts and queries beneath it, and those are learnable skills. The Prometheus Certified Associate (PCA) course on KodeKloud takes you deep into PromQL, alerting rules, and Alertmanager routing, the exact foundations this pattern is built on, and the KodeKloud playgrounds give you disposable clusters where a broken deployment costs nothing. Spin one up and wire your first alert to an agent today.

FAQs