If the steps are fixed, you want a script. If the next step depends on what you just observed, you want an agent. Everything else is detail.
Highlights
- Knowing when to use an AI agent comes down to one question, which is whether the next action genuinely depends on what the previous action returned.
- If the sequence is identical every run, you are proposing to pay a model to follow instructions you already have, and a script will be faster and easier to debug.
- Four gates qualify a candidate, covering machine readable evidence, genuine branching, a checkable answer, and a cheap mistake.
- A pipeline failure investigation branches heavily and qualifies, while a deprecation impact question looks similar and is exhaustive enumeration, which is a search.
- Counting anything across a population is a query, and an agent asked to count will produce a confident number derived from whatever retrieval happened to return.
- The smallest useful agent is roughly thirty lines, which makes building one faster than arguing about whether you need it.
- Read only tools should always be the first version, because a confused agent with no write access cannot do damage no matter how wrong it gets.
A team spent six weeks building an agent to produce their weekly infrastructure cost report. It worked, it cost more per run than the report was worth, and the sequence of steps it followed was byte for byte identical every week, which is the definition of a script. They had a model deciding what to do next in a situation where what to do next was never in question.
That mistake is easy to make because agents are genuinely useful and the boundary is invisible from outside. This post gives you one question that resolves most candidates in a sentence, four gates for the survivors, three real candidates scored end to end from a DevOps engineer, a cloud engineer, and a platform engineer, and about thirty lines of working code. One of those three candidates fails, and the way it fails is the most useful part.
The decision rule
Build an agent when the next action genuinely depends on what the previous action returned, when the evidence it needs is machine readable, when a human can verify the result quickly, and when a wrong answer is cheap or reversible. Fail the first and you want a script. Fail the last and you want a human approving every action.
The first clause does most of the work.
When to use an AI agent, the one question
Does the next step depend on what you just observed?
Answer no and the sequence is fixed, so you want a script, a pipeline, or a workflow with branches you wrote yourself. Generating a weekly report is fixed. Rotating credentials is fixed. Deploying on merge is fixed. A model in the middle of any of those adds latency, cost, and nondeterminism to a problem that had none of them.
Say yes and you may have an agent. Investigating a pipeline failure is not fixed, because what you check second depends entirely on whether the first check found a compile error, a flaky test, or an out of memory kill. Nobody can write those branches in advance, since the branches depend on findings.
Did you know
This question works as a test because it is close to the definition. An agent is a language model in a loop with tools, where after each tool result the model chooses what to call next. Remove the loop and you have a single model call. Remove the decision and you have a script that happens to invoke a model. So asking whether the next step depends on the last is really asking whether the loop is doing any work, and if it is not, you are paying for a loop you do not need.
The four gates
Candidates surviving the one question then face four gates. The fourth changes the shape of the answer rather than the answer itself.
Evidence machine readable. An agent reads logs, queries APIs, and inspects configuration. It cannot ask the person who knows why the cluster was built that way. If the needed information lives in somebody's head, in a screenshot, or in an untagged resource, the agent will not find it and will produce something plausible instead.
The path genuinely branches. The one question restated as a check, worth applying twice because the first answer tends to be optimistic. Write out how the last five human runs actually differed.
The answer is checkable. Can somebody verify the output faster than doing the task themselves? If verification costs what the work cost, you relocated the effort rather than removing it, frequently onto someone more senior.
A mistake is cheap. This decides autonomy rather than existence. A wrong triage summary costs five minutes of reading. A wrong scaling action costs an outage.
Watch out
Gate three is the one teams wave through and the one that decides usefulness in practice. An agent producing an answer nobody can check quickly does not reduce work, it relocates it. Before building, ask specifically who verifies the output and how long that takes. If the honest answer is that they would need to redo the investigation, the candidate has failed regardless of how strongly it scores elsewhere.
Three candidates, three roles, one failure
Here are three tasks that sound equally agent shaped. Two are, and one is not.
The DevOps engineer, triaging a pipeline failure
A build fails. Somebody reads the job log, works out whether it is a compile error, a flaky test, an infrastructure timeout, or a genuine regression, then either reruns it, files a bug, or pings the author.
Verdict. The strongest candidate here. Three read only tools cover it: fetch the job log, list commits since the last green run, and fetch the failure history for the specific test. It qualifies precisely because the second tool call is unpredictable from the first, and because a human can confirm the answer against a log line rather than by repeating the work.
The cloud engineer, investigating a cost anomaly
Spend jumped thirty percent week over week. Somebody works out whether it is a new service, a region change, a runaway job, a storage class transition, or a pricing update.
Verdict. Qualifies, and the evidence gate deserves a note. Untagged resources are exactly the case where the agent produces something plausible instead of something true, because the information genuinely is not there. That is not a reason to skip the build, it is a reason to have the agent say the spend is concentrated in untagged resources rather than guessing which team owns them, which is an instruction you write into the system prompt.
The platform engineer, assessing deprecation impact
You are removing an internal API version. Which teams break, and what do they need to change?
Verdict. Fails, and it is the most instructive of the three. This is exhaustive enumeration wearing the costume of an investigation. The question requires completeness, and an agent gives you whatever its searches surfaced, confidently, with no way to distinguish nothing found from nothing there. Gate two ends it and gate three confirms it, because checking the answer means doing the search yourself.
The right tool is a code search across your repositories plus a query against request logs grouped by client. That is a Monday afternoon rather than a six week project, and it produces a complete answer rather than a plausible one.
In the wild
A team built an assistant to report how many services a vulnerability affected. It answered with a number every time, the number always looked reasonable, and it came from whichever snippets retrieval happened to return. Nobody noticed for a month, because no individual answer was obviously wrong. The lesson is not that the model failed. It is that the question was a query rather than an investigation, and the system had no way to know it was being asked something it structurally could not do. Anything requiring completeness or arithmetic across a population belongs in a query.
Problems that look like agent problems
Quick tip
A quick way to catch a disguised query is to ask what happens when the answer is nothing. A genuine investigation can conclude that it found no cause and that conclusion is useful. A query dressed as an investigation cannot distinguish nothing found from nothing there, and it will report an empty result with exactly the confidence it reports a full one. If your candidate breaks on that question, it belongs in a search index rather than a loop.
Learn the agent mechanics properly
Building the small version teaches the shape, and the parts that make an agent production worthy are state, memory, routing, and human approval gates. The LangGraph course on KodeKloud covers stateful graph based workflows with orchestration, memory, debugging, and human in the loop, as components you assemble rather than concepts you read.
Build the small one instead of arguing
Often the fastest way to answer whether you need an agent is to build one badly in an afternoon. Here is the whole shape for the DevOps candidate above, using KodeKey as the model endpoint so access is one key rather than a procurement conversation.
import json, os, subprocess, requests
KODEKEY_URL = "https://api.ai.kodekloud.com/v1/chat/completions"
HEADERS = {"Authorization": f"Bearer {os.environ['KODEKEY_API_KEY']}"}
ALLOWED = {
"get_job_log": ["gh", "run", "view", "--log-failed", "--job"],
"commits_since": ["git", "log", "--oneline", "--no-merges"],
"test_history": ["./scripts/test-history.sh", "--test"],
}
def run_tool(name: str, argument: str) -> str:
if name not in ALLOWED:
return f"refused: {name} is not an allowed tool"
out = subprocess.run(ALLOWED[name] + [argument],
capture_output=True, text=True, timeout=30)
return (out.stdout or out.stderr)[:4000]
def call_model(messages: list, tools: list) -> dict:
body = {"model": "claude-haiku-4-5", "messages": messages,
"tools": tools, "temperature": 0}
r = requests.post(KODEKEY_URL, headers=HEADERS, json=body, timeout=60)
r.raise_for_status()
return r.json()["choices"][0]["message"]
def triage(question: str, tools: list, max_steps: int = 6) -> dict:
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": question}]
trace = []
for _ in range(max_steps):
reply = call_model(messages, tools)
messages.append(reply)
calls = reply.get("tool_calls") or []
if not calls:
return {"answer": reply["content"], "trace": trace, "steps": len(trace)}
for call in calls:
args = json.loads(call["function"]["arguments"])
name = call["function"]["name"]
result = run_tool(name, args.get("argument", ""))
trace.append({"tool": name, "argument": args.get("argument", "")})
messages.append({"role": "tool", "tool_call_id": call["id"],
"content": result})
return {"answer": "inconclusive within the step budget",
"trace": trace, "steps": len(trace)}Four properties of that code are doing the qualifying rather than the investigating.
The ALLOWED dictionary is the entire security model. Three tools exist, all read only, and the command is constructed in your code while the model supplies only one argument. No amount of confusion produces a command nobody anticipated.
Returning trace alongside the answer is the deliberate choice. You will read the trace far more often than the answer, because an agent that reached the right conclusion without gathering the supporting evidence was lucky and will fail the next case where luck does not hold.
The steps count is your qualification measurement. If your task consistently finishes in one step, the loop was never needed and you have your answer without any further debate.
Temperature at zero means the same investigation twice, which is what makes evaluation possible at all.
The system prompt is short and every line earns its place.
SYSTEM = """You investigate CI pipeline failures using the read only tools provided.
Call one tool at a time and reason from what it returns before calling the next.
Stop as soon as the evidence supports a conclusion, and cite the tool output that
supports each claim you make.
If the evidence is inconclusive, say so plainly and name what you would need.
Never assert a cause you have not observed evidence for."""Instructing it to stop as soon as evidence is conclusive keeps most runs to three or four calls rather than exhausting the budget. Asking it to cite the supporting output makes the trace reviewable. The instruction to admit inconclusiveness converts a silent guess into a visible request, which is also the escalation signal you will measure later.
Watch out
The common mistake at this point is giving the agent one flexible tool that runs any command, because writing three narrow ones feels like overhead. It is simpler, and it hands unlimited capability to something you are still evaluating. A confused agent with three read only tools produces a wrong answer. A confused agent with a shell produces an incident. Write more tools, keep each boring, and construct the command in your code rather than passing through whatever the model asked for.
The qualification sheet
Two rules travel with the sheet. One or two on branching disqualifies outright, because that is the script test and strength elsewhere cannot rescue it. One or two on reversibility disqualifies nothing, it fixes the design as read only tools plus a human approving any write, permanently.
The volume row is the one people dismiss. A task passing all four gates and occurring twice a month is a poor first project, because twenty real cases is roughly the minimum for a judgment you can trust, so a weekly task gives you an answer in five months while a daily one gives you an answer in a fortnight. Choose the frequent candidate even when a rarer one scores marginally higher.
Practise against real environments
Every judgment here gets easier once you have watched an agent choose a tool for a reason you can inspect. The Introduction to OpenAI course on KodeKloud covers the model, tool calling, and API layer this sits on, and the KodeKloud playgrounds give you clusters and an AI playground where experiments cost nothing.
Run it on your own ideas
Thirty minutes, and do the first part before you write any code.
Write down the three agent ideas currently circulating on your team. Ask the one question of each and expect at least one to turn out to be a script or a query, because that is the base rate. For the survivors, do not trust your instinct on branching. Go and find the last five times a human performed the task, and write down what they actually did in what order. If the five sequences look alike, you have your answer and you have saved yourself six weeks.
For whatever survives that, name the person who would verify the output and ask them how long checking would take. If they say they would need to redo the work, stop there.
Then build the smallest version with read only tools and a step budget, and run it against twenty real historical cases where you already know what the answer turned out to be. Record three things per case: whether the conclusion matched, whether the trace shows it gathered the evidence rather than guessing, and whether it said so when it could not tell. That third number is the one that will surprise you.
What you should be able to answer now
If this landed, four questions have answers.
Which of your team's agent ideas is actually a query? There is usually one, and it is usually the one with the most enthusiasm behind it, because completeness questions feel impressive and sound investigative.
For your strongest candidate, how did the last five human runs differ? If you cannot answer from memory, that is not a gap in your recall, it is evidence that the branching gate needs checking properly before anyone writes code.
Who checks the output, and how long does that take them? An agent whose answers take a senior engineer twenty minutes to verify has not saved twenty minutes, it has moved them onto someone more expensive.
What would your first version be unable to do? If the answer is nothing, the tools are too broad. The first version should be constitutionally incapable of causing harm, not merely instructed against it.
The cost report from the opening passed three gates comfortably. It failed one, the branching gate, and nobody asked. Ten minutes with the one question would not have made the agent better. It would have revealed, before six weeks of work, that the thing they were describing already had a name and it was a cron job.
Ready to Build the One That Qualifies?
Once a candidate passes the gates, the rest is ordinary engineering: a loop, some narrow tools, and the discipline to read the trace. The LangGraph course on KodeKloud covers agent loops, state, memory, and human approval as things you build, the Introduction to OpenAI course covers the model and tool calling layer underneath, and the KodeKloud playgrounds include an AI playground reachable through a single key. Start with the smallest one.
FAQs
Q1: How do I tell whether I need an AI agent or just a script?
Ask whether the next step depends on what you just observed. If the sequence is the same every run, you want a script, a pipeline, or a workflow with branches you wrote yourself, because a model choosing the next action adds latency, cost, and nondeterminism to a problem that had none. Generating a weekly report is fixed. Rotating credentials is fixed. Deploying on merge is fixed. If the sequence genuinely varies because what you check second depends on what the first check returned, you may have an agent, and triaging a pipeline failure is the clearest example since a compile error and an out of memory kill lead to entirely different follow up. The test works because it is close to the definition, as an agent is a model in a loop with tools where the model decides what to call after each result. If the loop is not doing work, you are paying for a loop. Check honestly by finding the last five human runs and writing down what actually happened in each.
Q2: What are the four gates, and which one do teams skip?
Evidence machine readable, meaning the agent can reach the information through logs, APIs, or version controlled config rather than needing to ask a person, because otherwise it produces something plausible instead of finding the answer. Path branches on findings, which is the script test restated. Answer is checkable, meaning somebody can verify the output faster than doing the task, and this is the gate teams wave through. Mistake is cheap, which changes the design rather than the decision, since failing it means read only tools and a human approving any write, permanently rather than temporarily. Gate three deserves the attention because an agent producing answers nobody can quickly verify has relocated work rather than removed it, and usually onto a more senior person. Before building, ask specifically who verifies and how long it takes them. If the honest answer is that they would need to redo the investigation, the candidate has failed no matter how well it scores elsewhere.
Q3: Can you give a concrete example of a candidate that fails?
Deprecation impact assessment, which is a common platform engineering proposal. You are removing an internal API version and want to know which teams break. It scores well on evidence, since the answer lives in source code and request logs, and well on mistake cost, since a wrong list wastes an announcement. It fails on branching, because the procedure is identical every time: search everywhere, list what you find. And it fails on checkability, because verifying completeness means running the search yourself. The deeper problem is that the question requires completeness while an agent returns whatever its searches surfaced, with no way to distinguish nothing found from nothing there. It is exhaustive enumeration wearing the costume of an investigation. The right tool is a code search across repositories plus a query against request logs grouped by client, which is an afternoon rather than a project and produces a complete answer instead of a plausible one.
Q4: What do I need to know before building the first one?
Ordinary software skills rather than machine learning ones. A language you are comfortable in, the ability to call an HTTP API, and enough familiarity with JSON to read a tool call. Conceptually, four things matter: the loop, tool definitions and the fact that the model requests rather than executes, the context window as a shared budget, and permissions. The single most useful fact is that the model never runs anything itself, it emits a structured request that your code decides whether to honour, which is why security belongs in the tool layer rather than the prompt and why a tool you never implemented cannot be called regardless of what the model asks for. Build the smallest version first, roughly thirty lines with two or three read only tools and a step budget, because watching your own agent choose a tool for a reason you can inspect is worth more than any amount of reading. The LangGraph course on KodeKloud covers loops, state, and approval gates as buildable components.
Q5: Why must the first version be read only?
Because it is the cheapest way to learn what the thing is actually good at, and because a confused agent with no write access cannot cause damage regardless of how wrong it becomes. A read only agent produces a hypothesis, a human reads it and acts if they agree, and their agreement rate across twenty real cases is a genuine evaluation metric rather than an impression. That gives you evidence before you make any risk decision. Adding write access afterwards is a small change to your tool dictionary and a large change to your exposure, which makes it worth deciding separately and with data rather than assuming it into the first version. It also matters that the tools be narrow rather than general, since one flexible tool that runs arbitrary commands hands unlimited capability to something you are still evaluating. Write more tools, keep each one boring, and construct the command in your code rather than passing through whatever the model supplied.
Q6: How do I measure whether the agent is worth keeping?
Three numbers, and the second is the one people forget. Measure whether it reached the correct conclusion, against twenty or so real historical cases where you already know how things turned out. Measure whether it reached that conclusion for a checkable reason, by reading the tool trace and confirming it gathered the evidence supporting the answer, because a lucky guess and a sound investigation look identical in the output and behave very differently on the next case. And measure the escalation rate, meaning how often it says it cannot determine something, since an agent that never admits uncertainty has not recognised the limits of what it can see and a healthy escalation rate signals honesty rather than weakness. Track cost per completed task alongside those, because the loop resends the whole conversation at every step, so a task taking eight tool calls costs considerably more than the step count suggests.
Discussion