Highlights
- What you'll build: a repeatable AI-assisted troubleshooting loop for Kubernetes, using k8sgpt against a deliberately broken cluster.
- Time required: about 30 minutes end to end.
- Prerequisites: Docker running, plus
kind,kubectl, andk8sgptinstalled. kubectl fluency assumed. - Stack: kind (local cluster), k8sgpt 0.4.36 (the diagnostic tool), and any OpenAI-compatible LLM backend for the
--explainstep. - Outcome: you diagnose a real
ImagePullBackOffand a realCrashLoopBackOff, get AI-generated fix steps, apply one, and verify recovery. - The key distinction: k8sgpt is a diagnostic that explains, not an agent that acts. It tells you what is wrong and what to do. You still run the fix.
- Current version note: k8sgpt moves fast (this used 0.4.36). Flags and backends change between releases, so check
k8sgpt versionand the built-in help rather than trusting an old blog.
The deploy went green in CI, and then the cluster told a different story. kubectl get pods comes back with a column of red: payment-api wedged on ImagePullBackOff, cart-worker cycling through CrashLoopBackOff like a bad heartbeat. So the ritual starts. Describe the worst offender. Then logs. Then logs --previous, because the container already restarted. Then back into the deployment YAML to re-read the image tag someone changed last week. Twenty minutes of muscle memory before you even have a hypothesis. None of it is hard. All of it is slow.
That slow, repetitive first pass is exactly what AI is good at compressing. Tools like k8sgpt read the same cluster state a human would, spot the broken resources, and hand back a plain-English explanation with concrete fix steps, in about ten seconds instead of twenty minutes. This guide shows you how to do it for real: you will break a cluster on purpose, point k8sgpt at it, run both its keyless detection and its AI-powered explanation, apply the fix it suggests, and watch the pod recover. Every command and every output below came from a live run on a throwaway cluster, not from a screenshot. The goal is not to hand your production incidents to a robot. It is to skip the twenty minutes of grunt work and get to the decision faster, with a human still making the call.
What AI Actually Does for Kubernetes Troubleshooting (and What It Doesn't)
Before you run anything, get the categories straight, because "AI for Kubernetes" is sold as one thing and is really three, and confusing them is how teams get burned.
k8sgpt is a diagnostic tool. It scans your cluster, finds resources in a bad state, and (optionally) asks a language model to explain the problem in plain English and suggest a fix. It reads. It explains. It does not touch your cluster. That is the whole point: it is safe to run against production because the worst it can do is tell you something you already knew.
That is different from an agent, which plans and takes actions in a loop: an agent might actually scale a deployment, patch a manifest, or restart a workload on its own. And it is different again from a copilot in your editor that autocompletes YAML. Diagnosing, acting, and assisting are three different risk profiles. k8sgpt sits firmly in the first bucket, and this guide stays there on purpose. Building an agent that acts on the cluster is a separate, more advanced job with a very different safety conversation.
So when you read "using AI to troubleshoot Kubernetes," read it precisely: AI is reading your cluster's state and the same describe/logs/events output you would read yourself, then summarizing it and proposing steps. The judgment about whether to apply those steps stays with you. That division of labor is the entire value proposition, and it is also why this is the safest, highest-return place to start with AI in your Kubernetes workflow.
Why Troubleshooting Is Where AI Earns Its Keep
Kubernetes failures are rarely mysterious. They are ImagePullBackOff because someone fat-fingered a tag, CrashLoopBackOff because a dependency is unreachable, a pending pod because no node has the resources, a failing probe because the path is wrong. The information you need is almost always sitting in kubectl describe and kubectl logs. The cost is not difficulty. The cost is the time and context-switching it takes to gather that information across a dozen resources while something is down, and the fact that a junior engineer often does not know which of the forty lines of describe output is the one that matters.
This is textbook toil: manual, repetitive, and completely automatable. A diagnostic that reads the state for you, filters the signal from the noise, and explains the failure in a sentence collapses that first pass from twenty minutes to seconds. It does not make you a better engineer. It makes your existing knowledge faster to apply, and it gives a less experienced on-call a running start instead of a blank terminal. That is a real, measurable win, and it does not require handing any authority to the machine.
The Tool Stack (and Why Each Choice)
You need four things: a cluster to break, a way to talk to it, the diagnostic tool, and a model for the explanation step. Here is what this guide uses and why.
The important design point: k8sgpt splits cleanly into a deterministic detection step that needs no model at all, and an AI explanation step that does. You can get most of the value keyless, and only reach for a model when you want the plain-English write-up.
Prerequisites and Setup
Make sure Docker is running, then confirm your tools. Installing k8sgpt is a one-liner on macOS (brew install k8sgpt) or via the release binaries on Linux; check the project's install docs for your platform.
# confirm the tooling
kind version # kind v0.32.0
kubectl version --client | head -1 # Client Version: v1.34.1
k8sgpt version # k8sgpt: 0.4.36
# create a throwaway cluster (about 30s)
kind create cluster --name ai-troubleshoot
That gives you a real single-node Kubernetes cluster running in Docker, with your kubeconfig context already pointed at it. Nothing here touches any real infrastructure, so break it freely.
Stage 1 - Break Something on Purpose
A diagnostic is only worth testing against a real failure, so create two of the most common ones. Save this as broken.yaml:
apiVersion: v1
kind: Namespace
metadata:
name: shop
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-api
namespace: shop
spec:
replicas: 1
selector:
matchLabels: { app: payment-api }
template:
metadata:
labels: { app: payment-api }
spec:
containers:
- name: payment-api
image: nginx:1.99-doesnotexist # tag that does not exist
ports:
- containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cart-worker
namespace: shop
spec:
replicas: 1
selector:
matchLabels: { app: cart-worker }
template:
metadata:
labels: { app: cart-worker }
spec:
containers:
- name: cart-worker
image: busybox:1.36
command: ["sh", "-c", "echo starting cart-worker; sleep 2; echo 'FATAL: cannot reach redis at redis:6379'; exit 1"]
Apply it and give it a minute to settle into failure:
kubectl apply -f broken.yaml
kubectl get pods -n shop
NAME READY STATUS RESTARTS AGE
cart-worker-b5448766b-624m8 0/1 CrashLoopBackOff 5 3m
payment-api-659c464f9b-92rcm 0/1 ImagePullBackOff 0 3m
Two classic failures, both real: payment-api cannot pull an image that does not exist, and cart-worker starts, fails to reach a dependency, and exits with a non-zero code, so Kubernetes restarts it on an exponential backoff. This is the column of red from the opening: two real failures, waiting for a diagnosis.
Stage 2 - Run k8sgpt Analyze (No API Key Needed)
Here is the part people miss: k8sgpt's detection is entirely deterministic and needs no model. Just run:
k8sgpt analyze
AI Provider: AI not used; --explain not set
...
11: Deployment shop/cart-worker()
- Error: Deployment shop/cart-worker has 1 replicas but 0 are available with status running
12: Deployment shop/payment-api()
- Error: Deployment shop/payment-api has 1 replicas but 0 are available with status running
13: Pod shop/cart-worker-b5448766b-624m8(Deployment/cart-worker)
- Error: the termination reason is Error exitCode=1 container=cart-worker pod=cart-worker-b5448766b-624m8
14: Pod shop/payment-api-659c464f9b-92rcm(Deployment/payment-api)
- Error: Back-off pulling image "nginx:1.99-doesnotexist": ErrImagePull: rpc error: code = NotFound
desc = ... docker.io/library/nginx:1.99-doesnotexist: not found
Notice two things. First, it found both real problems with no configuration and no key. Second, it is noisy: a plain analyze on a fresh cluster also flags a pile of unused ConfigMaps in the system namespaces (items 0 through 10, trimmed here). That is honest behavior worth knowing about, and the fix is to filter. Scope it to the namespace and resource type you care about:
k8sgpt analyze --namespace shop --filter Pod
AI Provider: AI not used; --explain not set
0: Pod shop/cart-worker-b5448766b-624m8(Deployment/cart-worker)
- Error: the last termination reason is Error container=cart-worker pod=cart-worker-b5448766b-624m8
1: Pod shop/payment-api-659c464f9b-92rcm(Deployment/payment-api)
- Error: Back-off pulling image "nginx:1.99-doesnotexist": ErrImagePull: rpc error: code = NotFound
desc = ... docker.io/library/nginx:1.99-doesnotexist: not found
That is the signal, cleanly. For the image pull failure, this alone is enough to fix it: the tag is wrong. But for the crash-loop, "termination reason is Error, exitCode=1" tells you it crashed, not why. That is where the explanation step earns its place. If you want to go deep on k8sgpt's filters, analyzers, and operator mode, KodeKloud's Introduction to K8sGPT and AI-Driven Kubernetes Engineering course walks through them hands-on.
Stage 3 - Add a Model and Use --explain
The --explain flag sends each detected problem to a language model and asks for a plain-English explanation plus remediation steps. You configure a backend once. k8sgpt supports many (openai, anthropic, ollama, localai, azureopenai, amazonbedrock, google, and more), so you can use a hosted model, a fully local one via Ollama to avoid keys and cost, or any OpenAI-compatible endpoint by pointing --baseurl at it:
# example: an OpenAI-compatible endpoint (use a local Ollama model to stay keyless)
k8sgpt auth add --backend openai --baseurl <your-endpoint>/v1 --model <model-name>
k8sgpt auth default --provider openai
The --baseurl trick means any OpenAI-compatible API works without special support in k8sgpt. This run used KodeKey, KodeKloud's unified AI key that exposes an OpenAI-compatible endpoint across multiple model providers, so you get one key you can drop straight into the --baseurl and --model flags above. A hosted OpenAI or Azure OpenAI key, or a local Ollama server, slots in exactly the same way with no other changes.
A note on secrets that matters in production: --explain sends resource metadata and error text to whatever model you configure. Use a local model or an approved internal endpoint for anything sensitive, keep the API key in the tool's config or an env var (never in a manifest or a commit), and use k8sgpt's --anonymize flag to mask resource names when you send data to a hosted provider. Now run it against the crash-loop:
k8sgpt analyze --explain --namespace shop --filter Pod
Representative output, captured live from minimax/minimax-m3 on KodeKloud's KodeKey during a real run. Your model and exact wording will differ every time. The detected problem and the fix steps are what to focus on, not the prose.AI Provider: openai
0: Pod shop/cart-worker-b5448766b-624m8(Deployment/cart-worker)
- Error: the last termination reason is Error container=cart-worker pod=cart-worker-b5448766b-624m8
Error: The container "cart-worker" inside pod "cart-worker-b5448766b-624m8" crashed and terminated
with an error status, causing the pod to fail.
Solution:
1. Run `kubectl describe pod cart-worker-b5448766b-624m8` to view termination details.
2. Run `kubectl logs cart-worker-b5448766b-624m8 -c cart-worker --previous` to check crash logs.
3. Inspect logs for errors (OOM, exceptions, config).
4. Fix the issue (resource limits, image, env vars).
5. Run `kubectl delete pod cart-worker-b5448766b-624m8` to let the controller recreate it.
1: Pod shop/payment-api-659c464f9b-92rcm(Deployment/payment-api)
- Error: Back-off pulling image "nginx:1.99-doesnotexist" ... not found
Error: The specified image "nginx:1.99-doesnotexist" doesn't exist on Docker Hub. The tag
"1.99-doesnotexist" is invalid, so Kubernetes cannot pull the image.
Solution: 1. Visit hub.docker.com/_/nginx/tags to find a valid version. 2. Update your deployment:
`kubectl set image deployment/<name> <container>=nginx:<valid-tag>`. 3. Verify:
`kubectl rollout status deployment/<name>`.
On this run the whole explanation took about ten seconds. Look at what it did: for the image pull, it correctly identified the invalid tag and gave the exact kubectl set image command to fix it. For the crash-loop, it did not hallucinate a root cause it could not know, it gave you the procedure to find it, starting with kubectl logs --previous. That is the right behavior, and it is a good example of using the model as a fast checklist, not an oracle.
Want broken clusters to practice on, without breaking your own?
KodeKloud Engineer hands you real, production-shaped Kubernetes tickets on live systems: pods that won't schedule, images that won't pull, services that won't route. It's the closest thing to on-call without the pager, and the perfect place to build the diagnostic instinct that AI accelerates but never replaces.
Try Real Tickets →Stage 4 - Apply the Fix and Verify
The AI told you what to do. Now you decide whether it is right, and you run it. This is the step that stays human. Start with the easy one, the invalid image tag, using exactly the command k8sgpt suggested:
kubectl set image deployment/payment-api -n shop payment-api=nginx:1.27
kubectl rollout status deployment/payment-api -n shop
deployment "payment-api" successfully rolled out
For the crash-loop, follow the procedure and read the actual logs, because that is where the truth is:
kubectl logs -n shop -l app=cart-worker --previous --tail=5
starting cart-worker
FATAL: cannot reach redis at redis:6379
There it is: the container is not misconfigured in Kubernetes, it is failing because a dependency (redis) is not reachable. No amount of restarting fixes that; you would deploy or fix redis, or correct the connection string. The AI got you to the log line in one hop instead of five. Confirm the state:
kubectl get pods -n shop
NAME READY STATUS RESTARTS AGE
cart-worker-b5448766b-624m8 0/1 CrashLoopBackOff 17 64m
payment-api-8d5f5f579-64742 1/1 Running 0 67s
payment-api is healthy. cart-worker is still crash-looping, correctly, because its real dependency genuinely is not there, which is a true problem the tool surfaced and a human resolves. The loop closed: detect, explain, decide, fix, verify.
Tying It All Together: The Triage Loop You Just Built
Step back and look at what you actually have. It is a four-move loop you can run against any cluster: k8sgpt analyze --filter Pod to find what is broken, --explain to turn a cryptic status into a plain explanation and a checklist, your own judgment to confirm the suggestion is sane, and kubectl plus a re-check to fix and verify. The AI owns the first two moves, the slow ones. You own the last two, the ones with consequences.
That division is the reason this is safe to adopt today, even on production, as long as you keep --explain's data handling in mind. k8sgpt never changes your cluster. It reads state you already have access to and writes you a summary. The worst outcome is a wrong explanation, which you catch because you verified against the real logs before acting, exactly as you would with a suggestion from a teammate.
Here is how the loop handled the two failures you just worked through, and where the human stays in the seat.
Where to Take It Next
Once the manual loop feels natural, there are three directions to grow. Run k8sgpt in operator mode inside the cluster so it continuously scans and exposes findings, instead of running it by hand. Wire the same idea into your observability stack so an alert arrives with a diagnosis already attached. And, when you are ready for the advanced step, move from a diagnostic that explains to an agent that acts, an LLM in a loop that can actually run remediations against the cluster, which is a genuinely different and riskier project that deserves its own guardrails and its own guide.
If you want a structured path for building the AI side of your skill set without neglecting the Kubernetes fundamentals underneath it, KodeKloud's AI-powered roadmap for DevOps and cloud engineers lays out a sensible progression, and the Kubernetes learning path keeps the core sharp, because AI accelerates the engineer who already knows what "good" looks like and quietly misleads the one who does not.
Common Gotchas
The default scan is noisy. A bare k8sgpt analyze flags unused ConfigMaps and other low-value findings. Filter by namespace and resource type (--namespace, --filter Pod) to see the incident, not the inventory.
The explanation can be confidently wrong. A model will sometimes give a clean, plausible root cause that is not the real one, especially for crash-loops where the truth is in the application logs. Treat --explain as a fast first hypothesis, always confirm against kubectl logs and describe before you act.
Mind what you send to a hosted model. --explain ships error text and resource metadata to your backend. For sensitive clusters, prefer a local model (Ollama), an approved internal endpoint, and the --anonymize flag. Never put an API key in a manifest or a commit.
Version drift is real. k8sgpt's flags, analyzers, and backend list change between releases. This guide used 0.4.36; if a flag here does not exist for you, run k8sgpt <command> --help rather than assuming the tool is broken.
Conclusion
Using AI to troubleshoot Kubernetes is not about replacing the engineer on call. It is about deleting the twenty minutes of mechanical gathering that stands between the alert and the hypothesis. k8sgpt reads the cluster, filters the noise, and explains the failure in a sentence, and you keep the judgment about what to actually do. Run the loop once against a broken kind cluster and it stops being abstract: detect, explain, decide, fix, verify. The single next move is to install k8sgpt today and point it at a cluster you can safely break, then feel how much faster the first pass gets.
Ready to Master AI-Driven Kubernetes, Not Just Read About It?
Reading a troubleshooting walkthrough is one thing. Running k8sgpt against a cluster you broke yourself, wiring up a local model backend, filtering real analyzer noise, and knowing when to override a confident-but-wrong explanation are different skills, and they only come from doing the work. That is what KodeKloud's Introduction to K8sGPT and AI-Driven Kubernetes Engineering course is built for: hands-on labs on live clusters, the tool's operator mode, and the judgment to use AI as an accelerator you control. By the end you will run the full detect-explain-fix-verify loop on real clusters, with the instinct to know where the AI stops and you take over.
FAQs
Q1: Is k8sgpt free, and do I need an API key?
The tool is open source and free, and it is a CNCF Sandbox project. The detection step (k8sgpt analyze) needs no key at all and finds the broken resources on its own. You only need a model backend for the --explain step, and even that can be a fully local model via Ollama, so you can run the entire workflow keyless and cost-free if you want.
Q2: Can I run this against my production cluster safely?
Yes, with one caveat. k8sgpt only reads cluster state, it never makes changes, so it cannot break anything by scanning. The caveat is --explain, which sends error text and resource metadata to whatever model you configure. For production, use a local or approved internal model, mask names with --anonymize, and keep keys out of manifests. The detection step alone is completely safe to point anywhere you have read access.
Q3: Does AI actually find the root cause, or just describe the symptom?
It depends on where the truth lives. For failures visible in cluster state (a bad image tag, a failed probe, a resource limit), it often nails the root cause and gives you the exact fix command. For application-level crashes, it correctly points you at the logs rather than guessing, because the real cause is in your code or a dependency, not in Kubernetes. Treat it as a very fast first hypothesis, not a verdict, and always confirm before acting.
Q4: How is this different from an AI agent that manages Kubernetes?
This is a diagnostic: it explains and recommends, and a human runs the fix. An agent plans and takes actions on its own (scaling, patching, restarting), which is a fundamentally different and riskier capability. Starting with a read-only diagnostic like k8sgpt is the safe, high-value entry point. Autonomous agents that act on production clusters are an advanced topic with a much larger safety surface, and they are worth approaching only after the assisted loop is second nature.
Sources: k8sgpt (CNCF Sandbox project) and docs; k8sgpt on GitHub; Kubernetes docs, Debug Running Pods; kind (Kubernetes in Docker); Kubernetes docs, image pull and container states. All commands and outputs in this guide were captured from a live run on a kind cluster (Kubernetes v1.36.1, k8sgpt 0.4.36); the --explain output was captured live from the minimax/minimax-m3 model.
Discussion