What Is an AI Agent and How Does It Work

What is an AI agent, mechanically? A model in a loop with tools. It never runs a command, it asks, and your code decides.

Gartner expects more than 40 percent of agentic AI projects to be cancelled by the end of 2027. That number is not a verdict on the technology, it is a verdict on how the technology gets adopted, because the teams that struggle are almost always the ones who never learned what an agent actually is. They bought a framework, wired it to production, and discovered too late that an agent is not a smarter chatbot with better manners. It is a loop, a toolbox, and a set of decisions about what software is permitted to do on its own.

The good news for anyone from an infrastructure background is that the mechanism is much simpler than the vocabulary suggests. There is no magic in the middle. There is a model that reads text and writes text, a list of functions you wrote, and a loop that runs until something says stop. This guide takes that apart properly: what an agent is, the four parts every one of them has, a real task traced turn by turn so you can see exactly what the model receives and what it sends back, and why the failures cluster where they do.

What is an AI agent

An AI agent is a large language model placed inside a control loop and given a set of tools, so that instead of answering a question in one turn it can pursue a goal across several turns, deciding at each step which tool to call, reading the result, and continuing until it reaches an answer or hits a limit you set.

Two ideas in that definition do the real work. The loop is what lets the model take a second step informed by the first. The tools are what let it affect anything outside its own text output. Take either away and you no longer have an agent.

Start by ruling things out

The fastest route to understanding agents is knowing what they are not, because the word gets attached to four quite different things.

That fourth row is the definition in practice. If a human decides every step, it is a chatbot. If you decided every step in advance, it is a workflow. If the model decides the next step from what it just observed, it is an agent.

The four parts of every agent

Once you can name these four, every agent framework you meet becomes readable, because they are all assembling the same components with different opinions about the plumbing.

The model is the reasoning engine. It reads everything it has been given and produces text, and that text may be an answer for the human or a request to use a tool. It has no memory of previous conversations, no ability to browse, and no way to run anything.

The tools are functions you wrote, described to the model in a structured format including a name, a description, and the arguments they accept. A tool might read a log file, query a database, call an API, or restart a service. The descriptions matter enormously, because the model chooses between tools by reading those descriptions.

The memory is everything the agent knows right now. Short term memory is the conversation so far, which includes the original goal, every tool call made, and every result observed. Long term memory, when present, is an external store the agent can search for things it learned in earlier sessions.

The loop is the control flow you write around all of it. Send the conversation to the model, read what comes back, execute any tool it requested, append the result, and repeat. Stop when the model produces a final answer, when a step limit is reached, or when a rule you defined says to stop.

Watch one task run

Here is the part that makes agents click. Below is a real task traced through the loop, showing what the model receives and what it sends back at each turn. The task is one an on call engineer knows well: an alert fires saying a service is returning errors.

The setup gives the model three tools. One fetches pod status for a namespace, one fetches recent log lines for a specific pod, and one fetches the deployment history for a service. All three are read only, which matters and we will come back to it.

Turn one. Your code sends the model the goal and the tool list. The model has no information yet, so it does the obvious first thing and asks for pod status.

{
  "role": "assistant",
  "tool_calls": [
    {
      "name": "get_pods",
      "arguments": { "namespace": "prod" }
    }
  ]
}

Your code receives that, checks it is an allowed tool with sensible arguments, runs the actual function, and appends the output as a new message. Notice what happened there and did not: the model did not run anything. It produced a piece of JSON describing what it would like to happen, and your code made the decision.

Turn two. The model now sees the pod list, which shows a pod named checkout-7d9f in a crash loop with 14 restarts. A human would immediately want the logs from the container that crashed rather than the one currently starting, and the model, having been told what the previous flag means, asks for exactly that.

{
  "role": "assistant",
  "tool_calls": [
    {
      "name": "get_logs",
      "arguments": {
        "namespace": "prod",
        "pod": "checkout-7d9f",
        "previous": true
      }
    }
  ]
}

Turn three. The logs come back showing a connection refused error against a database host. The model could stop and report that, but a database connection failing on a service that worked yesterday suggests something changed, so it asks for the deployment history.

Turn four. The history shows a deployment 40 minutes ago, which lines up with when the alert fired. The model now has enough and produces a final answer with no tool call attached, which is the loop's signal to stop.

That trace is the entire mechanism. Four turns, three tool calls, one conclusion, and at no point did the model do anything other than read text and write text. What made it look intelligent was the sequencing, and what made the sequencing possible was the loop.

The rule that explains everything

Here is the single most important thing on this page, and it is the fact that most explanations skip.

The model cannot execute anything. It has no shell, no network access, no file system. When an agent restarts a deployment, what actually happened is that the model produced text saying it would like to call a function named restart_deployment with certain arguments, and code that you wrote parsed that text, decided it was acceptable, and called the real function.

Every important consequence follows from that one fact.

Security lives in your code, not in your prompt. Telling a model in its instructions never to delete anything is a request, and a request can be misread, ignored, or overridden by text the model encounters later. Not implementing a delete function is a rule, and there is no prompt on earth that can call a function which does not exist.

The blast radius is exactly the union of your tools. An agent with three read only tools cannot damage anything no matter how confused it becomes. The same model with a tool that runs arbitrary shell commands can do whatever that shell can do, and the model has not changed at all between those two situations. Only your code did.

Untrusted input becomes a real concern. If your agent reads a log file, and that log file contains a line that looks like an instruction, the model sees it as just more text in its context. This is called prompt injection, and it matters precisely because the model cannot distinguish between the instructions you gave it and interesting looking text it encountered along the way.

Memory, and why agents forget

Memory confuses newcomers more than any other part, largely because the word suggests something the system does not have.

A model has no memory between calls. Each time your loop sends a request, it sends the entire conversation so far, and the model reads all of it fresh. What feels like memory is really your code resending the transcript every single turn.

That gives you three distinct kinds of memory in practice.

Working memory is the current conversation, meaning the goal, every tool call, and every result. It is the only thing the model can actually see, and it is limited by the context window, which is the maximum amount of text the model can process at once.

Short term memory is what survives within one session once the conversation grows too long. Because everything cannot fit forever, agents summarise older turns, drop tool outputs that no longer matter, or keep only the most recent exchanges. Something is always being discarded, and choosing well is a real design decision.

Long term memory is an external store, usually a database, holding facts from previous sessions. The agent searches it and adds relevant results into the current conversation. This is not the model remembering anything. It is your code doing a lookup and pasting the answer in.

Planning, and the ReAct pattern

Agents need some way to decide what to do next, and the dominant approach is refreshingly simple.

The ReAct pattern, short for reason and act, interleaves thinking with doing. The model reasons briefly about what it knows and what it needs, then acts by calling a tool, then observes the result, then reasons again. That is exactly the trace shown earlier, and its strength is that each decision is informed by real evidence rather than by a plan written before any evidence existed.

A second approach, plan and execute, has the model write out all the steps first and then work through them. That suits tasks with a known shape, such as a deployment checklist, and it struggles when reality diverges from the plan, because the agent is following a script written in ignorance.

Most practical agents use ReAct with a step budget, which caps how many tool calls one task may use. The budget matters more than it sounds, because a confused agent without one will keep calling tools indefinitely, and every call costs money and time.

Why agent projects fail

The failure data is unusually clear, and it points away from where most people look.

Research analysing more than 1,600 agent execution traces across seven frameworks found failure rates between 41 and 86 percent, and grouped the causes into three categories. Roughly 42 percent came from specification and design problems, meaning the agent was never told clearly enough what its job was, when to stop, or what to do when stuck. About 37 percent came from misalignment between steps or between agents, where context got lost in a handoff. The remaining 21 percent came from weak verification, meaning nobody checked whether the output was actually right.

Notice what is absent from that list. Model capability is not one of the three categories, and the researchers concluded that stronger base models alone would not resolve these failures. They are engineering problems, and specifically they are the engineering problems that infrastructure people are already good at: unclear requirements, lossy interfaces, and missing tests.

Build the agent foundations properly

Reading about the loop is useful, and building one yourself is where it becomes real. The LangGraph course on KodeKloud covers stateful agent workflows, routing, memory, and human feedback, which are the four things this guide has described in prose, taught as something you assemble and run.

How agents reach your systems

An agent is only as useful as the things it can touch, so at some point every team faces the integration question, and there are three answers.

Write the tools yourself. You define each function directly in your code, which is what the traced example earlier showed. This gives you complete control over arguments, validation, and scope, and it is the right choice for anything specific to your environment or anything that changes state.

Use an existing integration. Most agent frameworks ship connectors for common systems, which saves time and costs you visibility, because a connector written by somebody else may expose far more capability than your task needs.

Connect through a protocol. The Model Context Protocol, introduced in late 2024, standardises how an agent discovers and calls tools exposed by an external server. Instead of writing an integration per framework, a system publishes its capabilities once and any compatible agent can use them. Google's Agent2Agent protocol, which followed in 2025, does the equivalent job for agents talking to each other rather than to tools.

Protocols matter for the same reason any interface standard matters, which is that they turn many bespoke integrations into one published contract. They also introduce a question worth asking early, because connecting an agent to a third party server means trusting both the code and whatever that server chooses to return.

How to tell whether your agent is working

This is the step most teams skip, and skipping it is why so many agents feel impressive in a demo and disappointing in production.

The problem is that an agent's output is fluent whether or not it is correct, so reading a few answers and feeling satisfied tells you almost nothing. What you need is a fixed set of cases where you already know the right answer, run repeatedly, scored the same way each time.

Build that set from your own history. Take ten or twenty past incidents, tickets, or requests where the outcome is documented, feed each one to the agent, and record two things rather than one.

Did it reach the right conclusion? This is the obvious measure and the less informative of the two.

Did it get there for the right reason? Check which tools it called and in what order. An agent that guessed the answer without gathering the supporting evidence was lucky, and it will be confidently wrong on the next case where luck does not hold.

The escalation row is the one people find surprising. An agent that never says it is stuck is not confident, it is unable to recognise uncertainty, and that is a worse property than a slightly lower success rate. Instruct it explicitly to report when the evidence is inconclusive and to name what it would need, then treat a healthy escalation rate as a sign the system is behaving honestly.

Guardrails, in the order they matter

Guardrails get discussed as though they were a single feature. They are a stack, and the lower ones are stronger than the higher ones.

The strongest guardrail is a tool that does not exist. If your agent has no way to delete, it cannot delete. Start every design by deciding what capabilities the agent genuinely needs and implementing only those.

The next is scope inside the tools it does have. A tool that reads logs from an allowed list of namespaces is safer than one that reads logs from anywhere, and the check belongs in the function rather than in the description.

Then comes identity. The agent should run as its own service account or role with its own permissions, so that even a bug in your tool code cannot exceed what that identity is allowed to do. This is ordinary least privilege applied to a new kind of caller.

Above that sits human approval. Any action that changes something should pause and wait for a person to approve it, at least until the agent has earned trust on that specific action. Modern frameworks support this directly, and it is a configuration choice rather than a rewrite.

Weakest of all is the prompt. Instructions in the system prompt shape behaviour and do not constrain it. They are worth writing carefully, and they should never be the only thing standing between an agent and something irreversible.

Where agents genuinely help in DevOps today

Being specific here is more useful than enthusiasm, and the honest picture is good rather than miraculous.

The pattern across the strong rows is worth naming. Agents do well when the evidence is machine readable, the procedure is repetitive, the correct answer is checkable, and a mistake is cheap. They do poorly when the task needs judgment about tradeoffs, when the information lives in people's heads, or when being wrong is expensive.

Practise with real models and real environments

Concepts land faster when you can call a model, watch it request a tool, and see the loop turn. The Introduction to OpenAI course on KodeKloud covers working with models and APIs from first principles, and the KodeKloud playgrounds give you environments where you can wire an agent to a live cluster and break it safely.

Multi agent systems, and when to ignore them

The advice here is short, because the common mistake is reaching for this too early.

A multi agent system is several agents with different roles, tools, and often different models, coordinated by a supervisor or by talking to each other. It sounds like the natural next step and it usually is not, since every handoff between agents is a new place for context to get lost, which is the 37 percent failure category described earlier.

Add a second agent only when something forces it. Permissions genuinely differ, so an agent that reads audit data and one that changes infrastructure should never share a credential. Context would overflow if one agent held every domain at once. The work is truly parallel across many independent targets. Or one domain needs a different model to perform acceptably.

If none of those apply, give your single agent more tools instead. It will be cheaper, easier to trace, and more likely to work.

Getting started without building anything

You can understand agents properly before writing a line of code, and this sequence takes about an afternoon.

Use an agentic coding tool and watch its output carefully rather than skimming it. Notice how it reads a file, thinks, then reads another, then edits. That is the loop, running in front of you with a good implementation behind it.

Read the tool definitions of any open source agent you can find. The tools tell you what the agent can and cannot do far more accurately than its documentation does, and this habit transfers to evaluating vendor products.

Then build the smallest useful thing. One model, two read only tools, a loop with a step limit, and a printed trace. That is under a hundred lines in most languages, and having watched your own agent choose a tool for a reason you can inspect is worth more than any amount of reading.

Where to start

1. Write down one repetitive task on your team where the evidence is machine readable and the procedure is nearly identical every time, since that is your best first candidate.
2. List the tools it would need, then cross out every one that changes something, and confirm the task is still useful read only.
3. Write the tool descriptions before writing the tools, because the descriptions are what the model actually reads.
4. Build the loop with a hard step limit and print every tool call and result, so you can review the reasoning rather than just the conclusion.
5. Run it against ten real past examples and count how often it reached the same conclusion your team did.
6. Only after that, consider one write action, gated behind human approval and running under a separate identity.
7. Write down the agent's job, its stop condition, and what it should do when stuck, because that document is the fix for the largest failure category.

Conclusion

An AI agent is a language model in a loop with tools, and that plain description is more useful than any of the metaphors around it. The model reads text and writes text. Your tools decide what it can touch. Your loop decides how long it may keep going. Your guardrails decide what happens when it asks for something consequential.

Once that is clear, the industry's failure statistics stop being mysterious. Projects get cancelled because nobody wrote down what the agent's job was, because context vanished between steps, or because no one verified the output. Those are ordinary engineering problems with ordinary engineering fixes, and infrastructure people are unusually well equipped to solve them.

So start small and read the traces. Two read only tools and a step limit will teach you more in an afternoon than a month of reading, and the habit of asking what tools does this thing have will make you the person in the room who can evaluate any agent, including the ones being sold to you.

Ready to Build Your First Agent?

The concepts in this guide are best learned by assembling them. The LangGraph course on KodeKloud covers agent loops, state, memory, and human feedback as things you build rather than read about, Introduction to OpenAI covers working with models and APIs from the ground up, and the KodeKloud playgrounds give you environments where an experiment costs nothing. Pick one and start today.

FAQs