An alert fires: the web deployment is running one replica and getting hammered. You already know the fix. You alt-tab to a terminal, type kubectl scale deployment web --replicas=3, wait for the pods, and run kubectl get to confirm. Thirty seconds of work you have done a thousand times. The knowing was instant; the typing was the slow part. So the obvious question surfaces the moment you wire up a tool like k8sgpt that can already read a cluster and tell you what is wrong: if it can see the problem, why can't it run the boring, well-understood fix itself, while you approve the one line that actually changes something?

That is the jump from a diagnostic to an agent, and it is a real jump. A diagnostic reads and explains; it cannot break anything. An agent plans and acts: it scales, patches, restarts. The capability is genuinely useful and genuinely more dangerous, and the entire engineering problem is the guardrails around it, not the AI. This guide shows you how to build one for real. In about ninety lines of Python you will build a supervised agent that reads a live cluster, proposes a change, waits for your approval, runs it, and verifies the result, all on a throwaway kind cluster you can safely hand the keys to. Every command and every transcript below came from a live run, not a mock-up. The goal is not a robot that runs your production cluster unattended. It is to understand exactly how these agents work, and exactly where the human stays in the loop.

What "An Agent" Actually Is, and How It Differs from a Diagnostic

"AI for Kubernetes" gets sold as one idea. It is really a spectrum of very different risk profiles, and building an agent means picking a spot on it deliberately.

A copilot autocompletes YAML in your editor. It never touches a cluster. A diagnostic like k8sgpt reads cluster state and explains what is broken in plain language; it is safe to point at production because the worst it can do is be wrong in a sentence. An agent is the thing that acts: it takes a goal, decides which commands to run, and runs them, in a loop, until the goal is met. That last category is the subject here, and it is a different animal precisely because it has hands.

An agent, stripped of the hype, is a small and unmysterious thing: a language model wired to a set of tools, running in a loop. You give it a goal in plain English. It picks a tool, you (or your code) run that tool, you feed the result back, and it decides what to do next. The "intelligence" is the model choosing which tool and which arguments; the "agency" is that the tools can change the world. For Kubernetes, the tool is kubectl. Everything else is plumbing and, more importantly, safety.

We are building the third row: a supervised agent that can act, but only inside a namespace you scoped it to, only with verbs you allowlisted, and only after you approve the specific command. If you want the read-only version first, the diagnostic walkthrough with k8sgpt is the safer place to start and pairs naturally with this one.

The Loop, in One Minute

Here is the whole mechanism before we write any code. You send the model a goal and a description of the tools it can call. The model replies with either a final answer or a request to call a tool ("run kubectl get deploy web"). Your code runs that tool, captures the output, and sends it back as another message. The model reads the result and either calls another tool or declares the goal met. That is the "agentic loop": think, act, observe, repeat, with a hard cap so it can never spin forever.

In production you would not hand-roll this. You would use an MCP server that exposes Kubernetes as a set of tools, and a framework like kagent, the CNCF Sandbox project that runs agents as Kubernetes resources. We build the loop by hand here for one reason: once you have written the twenty lines that turn a model's tool call into a real kubectl command, the frameworks stop being magic and start being convenience.

Before You Start

These steps were run on a single-node kind cluster on macOS. Confirm your tooling and spin up a throwaway cluster:

kind version                          # kind v0.32.0
kubectl version --client | head -1    # Client Version: v1.34.1
python3 --version                     # Python 3.14.5

kind create cluster --name ai-agent-demo

Set up an isolated Python environment with the OpenAI SDK, which speaks to any OpenAI-compatible endpoint:

python3 -m venv agent-venv
./agent-venv/bin/pip install openai   # openai 2.46.0

You need a model the agent can call. This guide used KodeKey, KodeKloud's unified key that exposes an OpenAI-compatible endpoint (https://api.ai.kodekloud.com/v1), with gpt-5.4-mini as the model. A hosted OpenAI key, an Azure OpenAI endpoint, or a local model served through an OpenAI-compatible API all slot in the same way: change the base_url and model and nothing else. Export your key so it never lands in the code:

export KUBE_AGENT_KEY="<your-api-key>"

Now deploy something for the agent to manage. Save this as workload.yaml and apply it: a single-replica web deployment, deliberately under-provisioned so there is a real change to make.

apiVersion: v1
kind: Namespace
metadata:
  name: shop
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            requests: { cpu: "50m", memory: "32Mi" }

kubectl apply -f workload.yaml
kubectl get deploy web -n shop

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    1/1     1            1           54s

One replica, running. That is the "before" state, and the number to watch.

Step 1: Give the Agent a Safe Way to Run kubectl

Notice the order. Before the model, before the loop, we build the thing that decides what the agent is allowed to do. This is the load-bearing part of the whole exercise. An agent that can call kubectl with no constraints is a script that runs LLM-generated shell against your cluster, which is exactly as reckless as it sounds.

The guardrail function does four things: it refuses any verb not on an explicit allowlist, it forces every command into the one namespace we scoped, it runs read-only verbs immediately, and it stops and asks a human before any verb that changes the cluster.

import json
import os
import subprocess
import sys

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ai.kodekloud.com/v1",
    api_key=os.environ["KUBE_AGENT_KEY"],
)
MODEL = "gpt-5.4-mini"

# --- Guardrails: the agent can only ever touch this namespace, with these verbs ---
NAMESPACE = "shop"
READ_VERBS = {"get", "describe", "logs", "top"}
WRITE_VERBS = {"scale", "rollout", "set", "patch"}
ALLOWED = READ_VERBS | WRITE_VERBS  # everything else (delete, apply, exec...) is refused


def run_kubectl(args):
    """Run one kubectl command after checking it against the guardrails."""
    if not args:
        return "refused: empty command"
    verb = args[0]
    if verb not in ALLOWED:
        return f"refused: verb '{verb}' is not on the allowlist"
    if "-n" not in args and "--namespace" not in args:
        # force every command to stay inside the one namespace we scoped the agent to
        args = args + ["-n", NAMESPACE]

    cmd = ["kubectl"] + args
    if verb in WRITE_VERBS:
        # a change to the cluster: a human approves before it runs
        print(f"\n  proposed action: {' '.join(cmd)}")
        if input("  approve? [y/N] ").strip().lower() != "y":
            return "the human declined this action"

    out = subprocess.run(cmd, capture_output=True, text=True)
    return (out.stdout + out.stderr).strip() or "(no output)"

delete is not on the list, so the agent physically cannot delete anything, no matter what the model decides. scale is on the list but classified as a write, so it never runs without a human typing y. This is an allowlist, not a blocklist, on purpose: anything you did not explicitly permit is denied by default, which is the only safe direction for a tool that can act.

Step 2: Describe the Tool to the Model

The model cannot call a Python function directly. You describe the tool as a JSON schema, and the SDK gives the model a structured way to request it. You also give it a system prompt that sets the ground rules: look before you change, make the smallest change, verify, then stop.

TOOLS = [{
    "type": "function",
    "function": {
        "name": "run_kubectl",
        "description": "Run a single kubectl command against the cluster. "
                       "Read verbs (get, describe, logs, top) run immediately. "
                       "Write verbs (scale, rollout, set, patch) need human approval.",
        "parameters": {
            "type": "object",
            "properties": {
                "args": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "kubectl arguments, e.g. ['get','deploy','web','-o','wide']",
                }
            },
            "required": ["args"],
        },
    },
}]

SYSTEM = (
    "You are a Kubernetes operations agent. You manage workloads in the "
    f"'{NAMESPACE}' namespace by calling the run_kubectl tool. Always look at the "
    "current state before you change anything. Make the smallest change that meets "
    "the goal, then verify it. When the goal is met, say so in one sentence and stop."
)

Step 3: The Agent Loop

Now the loop itself. It appends the model's messages and your tool results to a running conversation, and it caps the number of turns so a confused model can never run away.

def main():
    goal = sys.argv[1]
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": goal},
    ]

    for _ in range(8):  # hard cap on the loop so it can never run away
        resp = client.chat.completions.create(
            model=MODEL, messages=messages, tools=TOOLS,
        )
        msg = resp.choices[0].message
        messages.append(msg)

        if not msg.tool_calls:
            print(f"\nagent: {msg.content}")
            return

        for call in msg.tool_calls:
            args = json.loads(call.function.arguments)["args"]
            print(f"\n> kubectl {' '.join(args)}")
            result = run_kubectl(args)
            print(result)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": result,
            })


if __name__ == "__main__":
    main()

That is the entire agent. Guardrail, tool schema, loop. Save it as agent.py.

Step 4: Run It

Give the agent a goal in plain English and watch it work. The y you see is where the guardrail stopped and waited for approval before scaling.

./agent-venv/bin/python agent.py "The web deployment can't handle current traffic. Check its state, then give it headroom by scaling it to 3 replicas. Verify it worked."

Representative output, captured from a live run on gpt-5.4-mini. The model's exact wording and the order of its reads will vary every time you run it; the actions it takes and the guardrail behavior are the point, not the prose.

> kubectl get deploy web -n shop -o wide
NAME   READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES       SELECTOR
web    1/1     1            1           2m13s   web          nginx:1.27   app=web

> kubectl scale deploy/web -n shop --replicas=3

  proposed action: kubectl scale deploy/web -n shop --replicas=3
  approve? [y/N] deployment.apps/web scaled

> kubectl get deploy web -n shop -o wide
NAME   READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES       SELECTOR
web    3/3     3            3           2m18s   web          nginx:1.27   app=web

agent: Scaled the web deployment to 3 replicas and verified it is now 3/3 ready and available.

Read what actually happened. The agent looked first (get deploy, a read, run without asking). It then chose to scale, which the guardrail intercepted and held until approval. After the approved scale, it did not take our word for it: it ran another get to confirm the deployment reached 3/3 before declaring success. That look-act-verify pattern is behavior we asked for in the system prompt, and the model followed it. Confirm the result yourself, independently of the agent:

kubectl get pods -n shop

NAME                  READY   STATUS    RESTARTS   AGE
web-8d748fc7f-g97pq   1/1     Running   0          6s
web-8d748fc7f-qxd7d   1/1     Running   0          6s
web-8d748fc7f-vfrdx   1/1     Running   0          2m20s

One replica became three, the agent proposed the change and you approved it, and the effect is real and checkable. That is a working supervised Kubernetes agent.

The Guardrails Are the Point

The interesting part of this project is not that the model can scale a deployment. It is that it can only do things you allowed. Two guardrails are worth proving, not just asserting. First, a verb that is not on the allowlist is refused outright, before anything runs:

>>> run_kubectl(["delete", "deployment", "web"])
'refused: verb \'delete\' is not on the allowlist'

Second, a permitted write that you decline does not happen. Answer N at the prompt and the cluster is untouched:

  proposed action: kubectl scale deploy/web --replicas=9 -n shop
  approve? [y/N] n
the human declined this action

After both, the deployment is still exactly where the approved run left it: nothing deleted, no surprise scale to nine.

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     3            3           2m45s

This is why the allowlist and the approval gate are not optional garnish. They are the difference between an agent you can reason about and a model with a shell. Four controls carry the safety of this design, and none of them depend on the model behaving well.

From Toy to Production: MCP and kagent

The ninety lines you just wrote are a teaching tool, not a production system. They show you the mechanism honestly, which is the point, but a real deployment changes three things.

The tool interface becomes MCP, the Model Context Protocol. Instead of your own run_kubectl shim, you run a Kubernetes MCP server (the containers/kubernetes-mcp-server project is one) that exposes cluster operations as standard tools any MCP-aware model can call, with the read/write and RBAC boundaries defined server-side rather than in a Python if statement.

The orchestration becomes a framework. kagent, accepted into the CNCF Sandbox, is the most Kubernetes-native option: you define agents as YAML custom resources and manage them with the same GitOps tooling you already use for everything else, so an agent is version-controlled and reviewable like any other manifest. It ships tools for Kubernetes, Prometheus, Istio, and Argo out of the box.

And the guardrails become infrastructure: real RBAC scoped to a service account, admission policies, an audit trail, and a change-approval path, rather than an input() prompt. The principle does not change from our toy, which is exactly why the toy is worth building. Here is a hands-on demo of an agent debugging a real cluster with kagent:

Common Errors and Fixes

KeyError: 'KUBE_AGENT_KEY'. The script reads your API key from an environment variable and you have not exported it in this shell. Run export KUBE_AGENT_KEY="<your-key>" in the same terminal, or the SDK has no credentials to send.

The model answers in text instead of calling the tool. If the agent replies with a paragraph like "You should run kubectl scale..." instead of actually calling run_kubectl, your tools=TOOLS argument is missing from the create() call, or the model you picked does not support tool calling. Confirm both, and keep the tool description concrete.

400: tool message with no preceding tool call. Every message with "role": "tool" must carry the tool_call_id from the model's request, and it must follow the assistant message that made the call. If you filter or reorder messages, the API rejects the request. Append in order, as the loop above does.

A malformed command gets refused. You may see refused: verb '-n' is not on the allowlist when the model puts a flag before the verb (kubectl -n shop get ...). That is the guardrail doing its job: it only trusts the first token as the verb. A well-behaved model self-corrects on the next turn once it sees the refusal, which is exactly the feedback loop working.

Rate limits or a slow first call. Hosted endpoints throttle. If a call errors with a rate-limit status, wait and retry, or drop to a smaller model. The first request in a session is often the slowest.

Clean Up

Tear down everything you created so nothing lingers:

kubectl delete namespace shop
kind delete cluster --name ai-agent-demo

The cluster ran entirely in Docker, so deleting it reclaims all of it. Your API key stays in your shell session only; close the terminal and it is gone.

Conclusion

An AI agent that manages Kubernetes is not a mystery and not, today, a replacement for you. It is a language model in a loop with a tool, and the engineering that matters is the fence you build around that tool: an allowlist so it cannot run what you did not permit, a namespace scope so it cannot wander, a human approval gate on every change, and a hard cap on the loop. Build the ninety-line version once and the production stacks, MCP for the tools and kagent for the orchestration, stop looking like magic and start looking like the same idea with better plumbing. The single next move is to run this against your own throwaway cluster, then decide, guardrail by guardrail, how much rope you are actually willing to give it.

Ready to Run AI Agents on Kubernetes for Real?

Building a ninety-line agent teaches you the loop. Running agents as first-class Kubernetes resources, wiring them to MCP servers, scoping them with RBAC, and tracing what they do on live clusters are the skills that make them production-worthy, and those only come from hands-on practice. That is what KodeKloud's KAgent: Host Your AI Agents on Kubernetes course is built for: deploying and managing agentic AI on Kubernetes with GitOps, from installation to real automation, in guided labs. Pair it with the AI learning path for the model and agent fundamentals underneath, and the AI-powered roadmap for DevOps and cloud engineers to see where the skill fits in a career, then go break a throwaway cluster with an agent of your own.

Start the KAgent course →

FAQs

Sources: Model Context Protocol; kagent (CNCF Sandbox project) and kagent on GitHub; Kubernetes MCP server (containers/kubernetes-mcp-server); OpenAI API function calling; kind (Kubernetes in Docker); Kubernetes docs, kubectl scale. All commands, code, and transcripts were captured from a live run on a kind cluster (Kubernetes v1.36.1); the agent output was captured live from gpt-5.4-mini via an OpenAI-compatible endpoint.