Building an AI Agent That Writes and Validates Ansible Playbooks

Ansible has an objective correctness test that no model can talk its way past. Here is how to build an agent around it.

Everyone who has asked a language model for an Ansible playbook has received something that looks right. It has tasks, it has names, the YAML is valid, and it probably even runs. Then you run it a second time and it reports four changed tasks, because the model reached for the shell module where a proper module existed, and your supposedly declarative automation is now a script that mutates a server every time it executes. Practitioner write ups consistently describe AI generated playbooks as roughly eighty percent correct, with the missing twenty percent concentrated in idempotence guards, secret handling, and environment specific edge cases.

That eighty percent figure is the whole opportunity and the whole problem. Boilerplate generation genuinely saves time, and unreviewed generation genuinely breaks servers. What makes Ansible an unusually good target for an agent is that the missing twenty percent is largely machine detectable: ansible-lint catches deprecated patterns and missing qualifications, check mode catches undefined variables against a real inventory, and an idempotence run catches the exact shell module mistake described above. This tutorial builds the generation and validation loop around those gates.

What an AI agent for Ansible playbooks is

An AI agent for Ansible playbooks is a language model wrapped in an automated validation loop that converts a natural language request into playbook YAML, runs it through linting, syntax checking, dry run, and idempotence testing against a disposable environment, feeds each failure back to the model for repair, and produces a reviewed pull request rather than applying anything to real infrastructure.

The loop is what distinguishes this from pasting a prompt into a chat window. A model asked to fix its own output without evidence guesses, while a model handed the specific ansible-lint rule identifier it violated fixes the actual problem, usually on the first attempt.

Why idempotence is the perfect gate

This deserves its own section because it is the strongest verification signal available in any infrastructure as code agent, and Ansible hands it to you free.

Idempotence means running the same playbook twice produces no changes on the second run. It is not a style preference, it is the property that makes Ansible safe to run repeatedly, and it is trivially machine checkable: run the playbook, run it again, and assert that the second run reports changed=0.

Now consider what language models get wrong. Trained on blog posts, Stack Overflow answers, and quick fixes, they reach readily for ansible.builtin.shell and ansible.builtin.command because those appear constantly in that corpus. Both are non idempotent by default, since Ansible cannot know whether an arbitrary command changed anything, so it reports changed every single time unless you supply creates, removes, or changed_when. A playbook full of shell tasks passes syntax checking, passes most linting, runs successfully, and fails an idempotence test immediately.

That is a rare gift in agent design: an objective, automated test that catches the model's most characteristic mistake, with no human judgment required.

The validation ladder

Order matters, because each rung costs more than the last and rejecting early saves the expensive stages for candidates that deserve them.

Molecule's default test sequence already runs these in a sensible order, moving through syntax, create, prepare, converge, idempotence, side_effect, verify, and destroy. One thing it does not include is a lint step, so run ansible-lint yourself before invoking Molecule rather than assuming it is covered.

Step 1: The generation prompt that prevents most failures

Prompt engineering will not replace validation, but a good prompt reduces how many repair cycles you pay for.

SYSTEM_PROMPT = """You write Ansible playbooks that pass ansible-lint in its
production profile and are strictly idempotent.

Hard requirements:
1. Use fully qualified collection names for every module, such as
   ansible.builtin.package rather than package.
2. Never use ansible.builtin.shell, command, or raw when a purpose built module
   exists. Prefer package, service, file, copy, template, lineinfile, user.
3. If shell or command is genuinely unavoidable, you MUST supply creates,
   removes, or changed_when so the task is idempotent.
4. Give every task a descriptive name starting with a capital letter.
5. Never write a plaintext password, key, or token. Reference a vaulted
   variable and add no_log: true to any task that handles a secret.
6. Set state explicitly on every module that accepts it.
7. Use handlers for service restarts rather than restarting unconditionally.

Return only the playbook YAML, with no explanation and no code fences."""

Requirement three is the one that pays for itself. Told merely to be idempotent, a model produces shell tasks it believes are harmless, whereas told exactly which fields make a shell task idempotent, it usually supplies them.

Step 2: Deterministic gates before the model gets another turn

Some checks should never depend on a model's cooperation. Parse the generated YAML and enforce policy in code, because a rule in a prompt is advice and a rule in a validator is a rule.

import re
import yaml

BANNED_BARE = {"ansible.builtin.shell", "ansible.builtin.command",
               "ansible.builtin.raw", "shell", "command", "raw"}
IDEMPOTENCE_ESCAPES = {"creates", "removes", "changed_when"}
SECRET_HINT = re.compile(r"(password|secret|token|api[_-]?key|private_key)", re.I)
PLAINTEXT_SECRET = re.compile(r":\s*[\"']?[A-Za-z0-9!@#$%^&*+/=]{8,}[\"']?\s*$")

def gate(playbook_yaml: str) -> list[str]:
    """Return a list of violations. Empty list means the playbook may proceed."""
    problems: list[str] = []
    try:
        plays = yaml.safe_load(playbook_yaml)
    except yaml.YAMLError as exc:
        return [f"invalid YAML: {exc}"]
    if not isinstance(plays, list):
        return ["playbook must be a list of plays"]

    for play in plays:
        for task in play.get("tasks", []) or []:
            name = task.get("name", "<unnamed>")
            if "name" not in task:
                problems.append("task is missing a name")

            modules = [k for k in task
                       if k not in {"name", "when", "become", "notify", "tags",
                                    "register", "loop", "vars", "no_log",
                                    "changed_when", "creates", "removes"}]
            for module in modules:
                # Gate 1: shell and friends need an idempotence escape hatch
                if module in BANNED_BARE:
                    args = task.get(module) or {}
                    keys = set(args) if isinstance(args, dict) else set()
                    if not (keys & IDEMPOTENCE_ESCAPES) and not (set(task) & IDEMPOTENCE_ESCAPES):
                        problems.append(
                            f"'{name}' uses {module} with no creates, removes, "
                            f"or changed_when, so it cannot be idempotent")
                # Gate 2: fully qualified collection names only
                elif "." not in module and module not in {"block", "rescue", "always"}:
                    problems.append(f"'{name}' uses '{module}' instead of an FQCN")

            # Gate 3: secrets must be vaulted and logged carefully
            flat = yaml.safe_dump(task)
            if SECRET_HINT.search(flat):
                if not task.get("no_log"):
                    problems.append(f"'{name}' handles a secret without no_log: true")
                for line in flat.splitlines():
                    if SECRET_HINT.search(line) and PLAINTEXT_SECRET.search(line) \
                       and "{{" not in line:
                        problems.append(f"'{name}' appears to contain a plaintext secret")
                        break
    return problems

Three properties make this worth writing rather than trusting the prompt. It is deterministic, so the same playbook always produces the same verdict. It runs in milliseconds, so it can reject a bad candidate before you spin up a container. And its output is specific enough to feed straight back to the model as a repair instruction, which is what makes the loop converge.

Step 3: The validation runner

Wrap the toolchain so each rung reports cleanly and the loop can decide whether to continue.

import subprocess
from dataclasses import dataclass

@dataclass
class Result:
    rung: str
    passed: bool
    output: str

def run(cmd: list[str], rung: str, timeout: int = 600) -> Result:
    """Run one validation command inside the working directory."""
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
    combined = (proc.stdout + proc.stderr)[-4000:]
    return Result(rung=rung, passed=proc.returncode == 0, output=combined)

def validate(path: str = "generated.yml") -> list[Result]:
    """Walk the ladder, stopping at the first failure to save time."""
    ladder = [
        (["yamllint", "-f", "parsable", path], "yamllint"),
        (["ansible-playbook", path, "--syntax-check"], "syntax"),
        (["ansible-lint", "--profile", "production", "-f", "pep8", path], "lint"),
        (["ansible-playbook", path, "--check", "--diff",
          "-i", "inventory/test"], "check-mode"),
        (["molecule", "test", "-s", "default"], "molecule"),
    ]
    results = []
    for cmd, rung in ladder:
        result = run(cmd, rung)
        results.append(result)
        if not result.passed:
            break            # no point converging a playbook that will not lint
    return results

Using ansible-lint --profile production rather than the default matters more than it looks. The production profile enforces the stricter rules that separate a playbook that runs from a playbook you would put in a repository, including naming conventions and fully qualified module names, which is exactly the class of thing generated code tends to skip.

Step 4: The Molecule scenario that does the real work

Molecule is the rung that turns opinion into evidence, and the idempotence stage is why it is in this tutorial.

# molecule/default/molecule.yml
---
driver:
  name: docker
platforms:
  - name: ubuntu-test
    image: ubuntu:24.04
    pre_build_image: false
    command: /lib/systemd/systemd            # so service tasks behave realistically
    privileged: true
    cgroupns_mode: host
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
provisioner:
  name: ansible
  config_options:
    defaults:
      callbacks_enabled: profile_tasks
verifier:
  name: ansible

# molecule/default/verify.yml
---
- name: Verify the requested end state
  hosts: all
  become: true
  gather_facts: true
  tasks:
    - name: Gather service facts
      ansible.builtin.service_facts:

    - name: Confirm nginx is enabled and running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services['nginx.service'].status == 'enabled'
        fail_msg: "nginx is not running and enabled as requested"

    - name: Confirm the configured port is listening
      ansible.builtin.wait_for:
        port: 8080
        timeout: 10

Running the systemd init inside the container is a deliberate choice. Service tasks behave very differently without a real init system, and a playbook that appears to work in a plain container while failing on a real host is worse than no test, because it produces false confidence.

Practice the validation toolchain properly

Molecule, roles, and Vault are the substance of this pipeline, and they reward hands on time. The Ansible Advanced Course on KodeKloud covers roles, vaults, dynamic inventory, cloud integration, and Molecule testing through a real deployment project, which is the toolchain this agent automates rather than replaces.

Step 5: The repair loop

Now assemble it. The loop generates, gates, validates, and feeds specific failures back, with a hard attempt limit so a confused model cannot spin forever.

MAX_ATTEMPTS = 4

def build_playbook(request: str, llm) -> tuple[str | None, list[str]]:
    """Generate and validate until the playbook passes or attempts run out."""
    transcript: list[str] = []
    feedback = ""

    for attempt in range(1, MAX_ATTEMPTS + 1):
        prompt = f"Request: {request}"
        if feedback:
            prompt += (f"\n\nYour previous attempt failed these checks. "
                       f"Fix each one specifically:\n{feedback}")

        candidate = llm(SYSTEM_PROMPT, prompt)
        open("generated.yml", "w").write(candidate)

        # Cheap deterministic gates first: milliseconds, no containers needed.
        violations = gate(candidate)
        if violations:
            feedback = "\n".join(f"- {v}" for v in violations)
            transcript.append(f"attempt {attempt}: {len(violations)} policy violations")
            continue

        # Then the real toolchain, cheapest rung first.
        results = validate()
        failed = [r for r in results if not r.passed]
        if not failed:
            transcript.append(f"attempt {attempt}: all rungs passed")
            return candidate, transcript

        feedback = f"The {failed[0].rung} stage failed:\n{failed[0].output}"
        transcript.append(f"attempt {attempt}: failed at {failed[0].rung}")

    return None, transcript          # exhausted: hand the transcript to a human

Notice that failure returns the transcript rather than the best guess. An agent that surrenders its last broken attempt as though it succeeded is worse than one that reports honestly, because a human reviewing a labelled failure spends five minutes while a human debugging a silent one spends an hour.

Note also what the loop never does. It never runs ansible-playbook against real inventory, only against inventory/test in check mode and against disposable Molecule containers. Generation speed is worth having, and generation autonomy against production is not.

Step 6: Measure it against a fixed task set

Before trusting this on anything that matters, find out how often it actually succeeds. Build a fixed set of requests with known correct shapes and score the agent on first pass and eventual success.

Score every run on three numbers and watch them over time: how often the first attempt passes everything, how often the loop eventually succeeds within its attempt limit, and which rung fails most. That third number is the useful one, because a persistent cluster of idempotence failures tells you the prompt needs the specific fix rather than the model needs replacing.

Practise this on a real Ansible control node

Every rung above is best learned by running it and watching what it rejects. The AI Assisted Ansible course on KodeKloud covers authoring, validating, and securing playbooks with AI tooling and linters, and the KodeKloud playgrounds give you an environment with a control node and managed nodes ready to break.

Where the agent helps and where it does not

Be candid about the right column when you show colleagues. An agent pitched as an Ansible expert loses trust the first time it mangles a multi host rollout, while one pitched as a fast, verified first draft for routine tasks keeps it.

Where to start

1. Pin the toolchain first with pip install ansible-dev-tools, then confirm ansible-lint --profile production runs clean against an existing playbook you trust.
2. Write the deterministic gate function and unit test it against a deliberately bad playbook full of bare shell tasks.
3. Add generation with the system prompt above and run the cheap rungs only, so you can see the failure pattern before paying for containers.
4. Stand up one Molecule scenario with systemd and a real verify file, since converge without verification only proves nothing crashed.
5. Add the repair loop with a hard attempt limit and log which rung fails most often.
6. Score the agent against the fixed task set and record first pass and eventual success rates before anyone relies on it.
7. Wire the output to a pull request, never to an inventory, and keep human review on the merge.

Conclusion

The reason Ansible is such a good fit for a generation agent is not that language models are especially good at it. They are roughly eighty percent good at it, which on its own is a liability rather than an asset. What makes it work is that Ansible ships an objective test for the exact thing models get wrong, since a playbook that reaches for shell where a module existed will fail an idempotence run every time, no matter how confident the explanation accompanying it was.

So build the ladder before the agent. Deterministic gates for policy, lint for convention, check mode for the real inventory shape, and Molecule for the truth, with every failure fed back as a specific instruction rather than a vague complaint. Then let the agent write the boring eighty percent quickly, and keep the pull request review that catches the rest.

Ready to Build Real Ansible Depth?

An agent is only as useful as the engineer reviewing its pull requests, and reviewing Ansible well means knowing what idempotence, handlers, and Vault actually require. The Ansible Advanced Course on KodeKloud covers roles, vaults, dynamic inventory, and Molecule testing on a real project, AI Assisted Ansible covers combining AI tooling with linters to author and secure playbooks, and the KodeKloud playgrounds give you a control node to practise on. Start with one today.

FAQs