How to Build a Simple Evaluation Harness for Your LLM Prompts

Writing the loop takes ten minutes. Deciding what counts as a correct answer is the actual work.

You changed a prompt. The outputs look different. Are they better?

Most teams answer that by reading a few responses and forming an impression. That works until the change helps four cases and quietly breaks two others, which is the normal outcome and the one an impression cannot detect.

A prompt is a code change with no test suite. This post builds the missing test suite, and spends most of its time on the part that actually matters, which is not the harness.

What an eval harness is

An eval harness has three parts, and only one of them is difficult.

You need a set of test cases, which are questions paired with the answers you know to be correct. You need a scorer, which looks at what the model actually said and decides whether it was right. And you need a loop that runs every case through the model, scores each answer, and counts how many passed.

The loop is the easy part. It comes to about ten lines, you write it once, and it never changes again.

Scoring is where the difficulty lives. Your model replies in ordinary English, and any correct answer can be written a hundred different ways, so deciding whether one of them counts as right turns out to be much harder than running the test.

That is why this post gets the loop out of the way early and then spends its time on scoring. Pick the wrong scorer and you still get a number, neatly formatted and completely meaningless.

What you need before you start

Everything here runs on one machine. A Linux box with root, which a KodeKloud playground gives you, or your own laptop.

You need two models. A small local one to produce answers, and a stronger one to judge them, which is the usual arrangement because judging is harder than answering.

Install Ollama for the answers

sudo apt-get update && sudo apt-get install -y zstd python3 python3-venv
curl -fsSL https://ollama.com/install.sh | sh
curl http://localhost:11434/
ollama pull llama3.2:1b

Ollama is running from that third command means the installer already started the service. A connection refused means you need ollama serve & first.

Get a key for the judge

The judge needs to be better than the model it grades. KodeKey gives you one key reaching Claude, GPT and Gemini through a single OpenAI compatible endpoint.

export KODEKEY_API_KEY="your key here"

Set up the project

mkdir evals && cd evals
python3 -m venv .venv
source .venv/bin/activate

Nothing to install. Everything below uses the standard library.

You will create four files.

Step 1: Write the test cases

Create cases.py. This is the part worth doing properly, because everything downstream measures against it.

CONTEXT = """
Restarting a stuck pod
When a pod enters CrashLoopBackOff, Kubernetes waits longer between each restart,
doubling up to five minutes. Delete the pod with kubectl delete pod and the
controller recreates it within seconds.

Memory limits and exit code 137
Exit code 137 is 128 plus signal 9, which is kill. The kernel out of memory killer
stopped the process. The limit is enforced per container, not per pod.

Rotating the database password
Rotate the credential in the secret store first, then update the Kubernetes secret,
then restart the deployment. The application does not watch the secret for changes.

Draining a node
Cordon the node first so the scheduler stops placing new pods on it, then drain it.
A deployment with one replica and a budget requiring one available blocks the drain.
"""

CASES = [
    {"q": "What does exit code 137 mean?",
     "expect": "the out of memory killer stopped the process",
     "keyword": "137"},
    {"q": "How long does Kubernetes wait between restarts at most?",
     "expect": "five minutes",
     "keyword": "five minutes"},
    {"q": "In what order do I rotate the database password?",
     "expect": "secret store first, then the Kubernetes secret, then restart",
     "keyword": "secret"},
    {"q": "Is the memory limit per pod or per container?",
     "expect": "per container",
     "keyword": "container"},
    {"q": "Why is my drain hanging?",
     "expect": "a disruption budget is blocking it",
     "keyword": "budget"},
    {"q": "What is the capital of France?",
     "expect": "the context does not cover this",
     "keyword": "not"},
    {"q": "Should I cordon before or after draining?",
     "expect": "cordon first",
     "keyword": "cordon"},
    {"q": "Does the app reload a changed secret on its own?",
     "expect": "no, it needs a restart",
     "keyword": "restart"},
]

Three things about that list are deliberate.

Each case carries an expect field in plain language rather than the exact words you want back. You are recording what a correct answer means, not what it should say.

The keyword field exists so we can compare scoring methods later. You would not normally maintain both.

And one case asks something the context cannot answer. A system that invents an answer there is broken in a way none of the other cases would reveal, so a test set without at least one of these is missing its most important row.

Step 2: Generate the answers

Create generate.py. It runs each case once and saves the results, so you score the same answers repeatedly without paying to regenerate them.

import json
import urllib.request

from cases import CASES, CONTEXT

PROMPT = ("Answer using only the context below. If it does not contain the answer, "
          "say so.\n\nContext:{context}\n\nQuestion: {question}\n\nAnswer briefly.")


def ask(question, model="llama3.2:1b"):
    body = json.dumps({
        "model": model,
        "prompt": PROMPT.format(context=CONTEXT, question=question),
        "stream": False,
        "options": {"num_predict": 70, "temperature": 0},
    }).encode()
    request = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(request).read())["response"].strip()


answers = []
for case in CASES:
    answer = ask(case["q"])
    answers.append({**case, "answer": answer})
    print(f"Q: {case['q']}")
    print(f"A: {answer[:100]}\n")

json.dump(answers, open("answers.json", "w"), indent=1)

python3 generate.py

python3 generate.py

Q: What does exit code 137 mean?
A: Exit code 137 means the kernel out of memory killer stopped the process.

Q: Why is my drain hanging?
A: The pod is hanging because it is waiting for Kubernetes to drain the
node, which is taking longer than expected.

Q: What is the capital of France?
A: I do not have enough context to provide a correct answer.

Read those answers before going further, because you need to know which are right in order to judge the scorers. Seven of the eight are correct here. The drain question is wrong, since the model described the drain being slow rather than a disruption budget blocking it.

Note the temperature is 0. You want the same answers each run, so that a score change means your prompt changed rather than the dice.

Step 3: Write four scorers

Create scorers.py. Each function takes a case and returns True or False.

import json
import os
import urllib.request


def exact(case):
    """The strictest possible check."""
    return case["answer"].strip().lower() == case["expect"].strip().lower()


def keyword(case):
    """Does the answer contain the word we expect."""
    return case["keyword"].lower() in case["answer"].lower()


def always_pass(case):
    """A deliberately useless scorer, included to expose a bad metric."""
    return True


JUDGE_PROMPT = """You are grading an answer against a reference.

Question: {q}
Reference answer: {expect}
Answer to grade: {answer}

Does the answer convey the same thing as the reference? Ignore wording and style.
Reply with exactly one word, PASS or FAIL."""


def judge(case, model="claude-haiku-4-5-20251001"):
    """Ask a stronger model whether the answer means the same thing."""
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": JUDGE_PROMPT.format(**case)}],
        "max_tokens": 5,
        "temperature": 0,
    }).encode()
    request = urllib.request.Request(
        "https://api.ai.kodekloud.com/v1/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ['KODEKEY_API_KEY']}",
        },
    )
    reply = json.loads(urllib.request.urlopen(request).read())
    return reply["choices"][0]["message"]["content"].strip().upper().startswith("PASS")


SCORERS = {
    "exact match": exact,
    "keyword": keyword,
    "LLM judge": judge,
    "always pass": always_pass,
}

The always_pass scorer is not a joke. It is the control, and the next section is why it is there.

Three details in the judge matter more than they look.

Temperature is 0. A judge that grades differently on reruns makes every comparison meaningless.

It returns one word. Asking for a paragraph gives you something to parse and something to argue with. PASS or FAIL parses in one line.

It sees the reference answer. Asking a model whether an answer is good in the abstract gets you an opinion. Asking whether it matches a known reference gets you a comparison.

Step 4: The loop, and the comparison that matters

Create harness.py.

import json

from scorers import SCORERS, exact, keyword, judge

ANSWERS = json.load(open("answers.json"))

# What a human says after reading each answer against the context.
# Index 4 is the drain question, which the model got wrong.
HUMAN = [True, True, True, True, False, True, True, True]

print(f"{'scorer':14s}{'agrees with human':>19s}{'catches the failure':>21s}")

for name, score in SCORERS.items():
    verdicts = [score(case) for case in ANSWERS]
    agreed = sum(1 for verdict, human in zip(verdicts, HUMAN) if verdict == human)
    caught = verdicts[4] is False
    print(f"  {name:12s}{agreed}/{len(ANSWERS)} = {agreed / len(ANSWERS):>7.0%}"
          f"{str(caught):>21s}")

print(f"\n  {'question':42s}{'human':>7s}{'exact':>7s}{'keyword':>9s}{'judge':>7s}")
for case, human in zip(ANSWERS, HUMAN):
    print(f"  {case['q'][:40]:42s}{str(human):>7s}{str(exact(case)):>7s}"
          f"{str(keyword(case)):>9s}{str(judge(case)):>7s}")

python3 harness.py

python3 harness.py

scorer          agrees with human  catches the failure
  exact match 1/8 =     12%                 True
  keyword     6/8 =     75%                 True
  LLM judge   7/8 =     88%                 True
  always pass 7/8 =     88%                False

  question                                    human  exact  keyword  judge
  What does exit code 137 mean?                True  False     True   True
  In what order do I rotate the database p     True  False     True  False
  Why is my drain hanging?                    False  False    False  False
  What is the capital of France?               True  False    False   True

That output is the reason this post exists.

The trap in that table

Look at the last two rows. The LLM judge agreed with the human 88 percent of the time. A scorer that passes everything, unconditionally, also agreed 88 percent of the time.

They tie on the headline number. One catches the failure and one catches nothing.

The reason is that the test set is unbalanced, with seven correct answers and one wrong one. Anything that says PASS is right seven times out of eight before it does any work at all.

So agreement rate on its own is not a measure of a scorer. It rewards guessing the majority class, and real eval sets are always unbalanced, because most of your outputs are fine most of the time.

The column that separates them is the third one. A scorer is only worth anything if it catches the failures you already know about.

Why the other scorers fail

Both failures are worth understanding, because you will reach for both.

Exact match scored 12 percent. It requires the answer to be character identical to your reference, which free text never is. It is genuinely useful when your output is structured, a classification label or a JSON field, and useless the moment a sentence is involved.

The keyword check failed in two directions. Look at the France question. The model correctly declined, saying it lacked enough context, and the check looked for the word "not" which never appeared, because the model used a contraction instead. A correct answer marked wrong.

Then look at the password rotation question. The answer contained the word "secret" and got a pass, while being subtly wrong about the order. A wrong answer marked right.

That is the shape of keyword scoring. It is fast, free, and deterministic, and it measures vocabulary rather than meaning.

The judge is not ground truth either

One row deserves an honest look. On the password rotation question, the human said the answer was acceptable and the judge failed it.

The judge was being stricter about the ordering than the human was. Neither is obviously wrong, and that is the point. A judge is another model with an opinion, calibrated by your prompt.

Which is why the human column exists in that table. You are not using humans to grade every run, which would defeat the purpose. You are using thirty to fifty human labelled examples once, to check whether your judge agrees with you often enough to be trusted, and then the judge grades everything after that.

Expect around 75 to 90 percent agreement from a well written judge prompt. Perfect agreement is not a realistic target, and chasing it usually means you have written a rubric so narrow it no longer measures what you care about.

Practise the whole pipeline

Evaluation is one part of getting an AI feature to production standard. The AI Agents Fundamentals course covers tokens, embeddings, retrieval, orchestration and MCP through an end to end project.

Step 5: Use it to make a decision

Now the payoff. The harness exists so you can answer a question you would otherwise argue about.

Here are two prompts. One just hands over the context, the other adds rules about staying grounded and admitting ignorance.

import json
import urllib.request

from cases import CASES, CONTEXT
from scorers import judge

PROMPTS = {
    "bare": "Context:{context}\n\nQuestion: {question}",
    "with rules": ("Answer using only the context below. If it does not contain "
                   "the answer, say so.\n\nContext:{context}\n\n"
                   "Question: {question}\n\nAnswer briefly."),
}


def ask(prompt, model="llama3.2:1b"):
    body = json.dumps({
        "model": model, "prompt": prompt, "stream": False,
        "options": {"num_predict": 70, "temperature": 0},
    }).encode()
    request = urllib.request.Request(
        "http://localhost:11434/api/generate", data=body,
        headers={"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(request).read())["response"].strip()


for name, template in PROMPTS.items():
    passed, failures = 0, []
    for case in CASES:
        answer = ask(template.format(context=CONTEXT, question=case["q"]))
        if judge({**case, "answer": answer}):
            passed += 1
        else:
            failures.append(case["q"][:38])
    print(f"  {name:12s}{passed}/{len(CASES)}  {passed / len(CASES):>6.0%}")
    print(f"    failed: {failures}")

python3 compare.py

prompt           passed   score
  bare        4/      8     50%
  with rules  6/      8     75%

  bare failed on: ['What does exit code 137 mean?', 'Why is my drain
  hanging?', 'What is the capital of France?', 'Does the app reload']
  with rules failed on: ['In what order do I rotate the database',
  'Why is my drain hanging?']

Fifty percent against seventy five. The rules are worth keeping, and now you know that rather than believing it.

The failure list matters as much as the score. The bare prompt failed on the France question, which is exactly what those rules were added to fix, and it also failed on exit code 137 for an unrelated reason. Two prompts scoring the same can still be failing on completely different cases, and only the list tells you that.

What to do next with it

Five things, roughly in the order they pay off.

Grow the case list from real failures. Every time something goes wrong in production, add it. That is how the set becomes worth trusting rather than worth arguing with.

Run it on every prompt change. Not because you will always act on the number, but because a silent regression is the failure this whole exercise exists to prevent.

Label thirty to fifty cases by hand, once. That is what tells you whether to trust your judge, and it is the only human work you need to repeat rarely.

Track the two per class numbers, not the overall agreement rate, for the reason the table above demonstrates.

Version the case list with your code. It defines your baseline, and a baseline that changes silently is worse than none.

What you should be able to answer now

Which part of an eval harness is hard? The scorer. The loop is ten lines and you write it once.

Why is exact match unsuitable for free text? Because a correct answer can be phrased a hundred ways, and it scored 12 percent agreement on real output here.

Why can agreement rate make a useless scorer look good? Because eval sets are unbalanced. Passing everything scored 88 percent here, the same as a real judge, while catching nothing.

What number should you look at instead? Whether the scorer catches the failures you already know about, measured per class rather than overall.

The harness in this post is about sixty lines and it turns "does this prompt look better" into a number with a list of failures attached. The number is not the valuable part. The list is.

Ready to Build on Top of This?

Evaluation is what turns a prompt that works in a demo into one you can change safely. The AI Agents Fundamentals course covers the surrounding pieces end to end, and the AI Learning Path sequences it alongside retrieval, vector databases and agents. Start by writing down five failures you have already seen.

FAQs