Fine Tuning vs Prompting, When to Use Each

Fine tuning changes how a model behaves. Retrieval changes what it knows. Prompting changes what you asked. Choosing wrong is where budgets disappear.

A team wanted their assistant to answer questions about internal runbooks, so they fine tuned a model on the runbooks. Three weeks on, with a compute invoice in five figures, the model was still inventing procedures, because the training had absorbed the shape and cadence of their documentation without reliably absorbing what it said. They rebuilt the same thing with retrieval in four days and it worked.

That failure repeats constantly and it comes from one misunderstanding. Fine tuning changes how a model behaves. Retrieval changes what it knows in the moment. Prompting changes what you asked for. This post walks the ladder in order, shows what each rung actually fixes, works three real candidates from DevOps, cloud, and platform engineering, and gives you runnable evaluation code, because the thing that decides whether tuning was worth it is a measurement most teams never build.

The distinction that decides everything

Fine tuning updates a model's weights so it produces a consistent form of output, meaning tone, structure, schema, or a specialised reasoning pattern. Retrieval inserts current documents into the prompt so the model reads rather than recalls. Prompting states what you want from the model you already have.

The industry shorthand that stuck in 2026 is worth memorising: tune the interface, retrieve the content. If your problem is that the model does not know something, no amount of training reliably fixes it. If your problem is that the model knows enough and keeps producing the wrong shape, training is exactly the tool.

Fine tuning vs prompting, what each one actually does

Look at the last but one row, which ends most arguments. Requirements change, and a prompt changes in seconds while a tuned model changes in a training run. If your team is still working out what good output looks like, freezing today's preference into the weights is premature by definition.

The four rung ladder

Climb in order, and prove each rung insufficient before moving up. The discipline is the whole method.

Rung one, prompting. Supply the context the model cannot have, name the output format, show one example, state the constraints, and tell it what to do when unsure. Costs minutes, and closes the gap more often than teams expect.

Rung two, retrieval. Search your own documents at question time and insert what you find. This is the answer whenever the problem is that the model does not know your specifics. Costs days and updates instantly when a document changes.

Rung three, fine tuning. Train a small adapter so the model behaves consistently without a long prompt carrying the instructions every time. Costs weeks including evaluation, and it is a commitment rather than a change.

Rung four, distillation. Tune a small open model to match a larger one on one narrow task, then run the small one instead. This is where the money is in 2026, since a job the base model already performs passably can move to a model that costs a tenth as much per call.

What fine tuning genuinely fixes

Being specific here prevents most wasted projects.

A rigid output schema. Every response in the same JSON structure, every time, without a prompt restating the schema on every call. Prompting gets you most of the way and tuning gets you the last stretch of reliability.

A consistent tone or persona. House voice for generated documentation, a specific register for customer facing text. Style is a behaviour, and behaviours are what weights hold well.

A specialised reasoning pattern. A domain specific way of working through a problem that general prompting handles unreliably across thousands of cases.

Uniform refusals. Every declined request phrased identically, which matters more in regulated settings than most engineers expect.

Cost and latency. The distillation case, where a tuned small model matches a large one on one narrow job and runs for a fraction of the price.

What it does not fix is facts. Any fact that changes, any fact specific to your organisation, and anything you would need to cite belongs in retrieval, because training is an expensive and unreliable way to inject knowledge and it offers no source attribution at all.

Build the retrieval rung properly first

Rung two resolves more problems than rung three and it is the one worth being good at, because most requests for fine tuning turn out to be knowledge problems. The Fundamentals of RAG course on KodeKloud covers architecture, document ingestion and chunking, keyword against semantic search, vector databases, and building a complete pipeline.

What tuning actually costs, honestly

The compute number is the one people quote and the smallest part of the bill.

Full fine tuning updates every parameter and is almost never the right answer now, because it is expensive, it risks the model losing capabilities it previously had, and it ties you to one base model checkpoint. Parameter efficient methods freeze the base model and train a small set of additional parameters, which for most product use cases matches full tuning closely at a fraction of the cost. LoRA and its quantised variant are the practical default.

A parameter efficient run on a mid sized open model typically lands in the tens to low hundreds of dollars and completes in hours on a single GPU. That number is genuinely small, and it is why the real costs are worth naming separately.

The dataset size question deserves correcting because outdated guidance still circulates. The old rule of needing at least a thousand examples came from full fine tuning, and parameter efficient methods changed it. For classification and extraction, a few hundred well curated examples is frequently enough, and teams stall chasing volumes that the method does not require. Five hundred good examples beat five thousand mediocre ones.

Three candidates, three roles, three rungs

Same question, three real situations, and they resolve at three different rungs.

The DevOps engineer, summarising incidents

You want an assistant that reads an incident's alerts, chat, and timeline and produces a summary in your team's postmortem format, with the same sections every time.

The gap. The model writes competent summaries in an inconsistent structure, sometimes omitting the contributing factors section, sometimes merging timeline and impact.

The rung. Start at one. Name the sections explicitly, supply one worked example of a good postmortem, and state that every section must appear even if it says nothing occurred. This is a form problem, and prompting handles form well when the format is stated rather than implied.

When to climb. If you are running hundreds of these a week and the format still drifts, rung three is justified, because you are paying for a long format instruction on every call and getting inconsistent results anyway. That is exactly what tuning fixes: the schema moves into the weights and the prompt gets short.

The cloud engineer, answering questions about spend

Somebody wants to ask which service drove last week's cost increase and get an answer grounded in your actual billing data.

The gap. The model knows nothing about your account, so it produces plausible generic answers about common cost drivers.

The rung. Two, unambiguously. This is a knowledge problem and no amount of training fixes it, because the data changes weekly and the correct answer for last week is wrong for this week. Retrieval over your cost reports, filtered by date and account, gives you a grounded answer with the line items to verify it.

The trap. This is the exact shape of the runbook failure in the opening. It feels like the model needs to learn your environment, and what it needs is to read your environment at question time. Anything that changes on a schedule belongs in retrieval permanently, not in weights temporarily.

The platform engineer, classifying incoming requests

Your platform team receives hundreds of tickets a week. You want each one classified into a category, an urgency, and a target team, in a fixed structure, cheaply and fast.

The gap. A frontier model does this well and the volume makes the per call cost noticeable, and the latency shows up in your ticket routing.

The rung. Four, and this is the clearest case for tuning on the list. The task is narrow, the output is a fixed schema, the correct answer is verifiable against how tickets were actually routed, and thousands of historical examples already exist, labelled by where each ticket really went. Tune a small open model on that history and run it instead, which typically takes both cost and latency down by a factor of ten or so.

Why it qualifies where the others did not. The task is stable, meaning your categories are not changing weekly. The output is form rather than fact. The training data already exists as a byproduct of doing the work. And you have a natural evaluation set, since every historical ticket has a known correct answer.

Build the measurement first

Here is the part teams skip and the part that decides everything. Before any training run, you need a held out set and a baseline score, because otherwise you cannot tell improvement from impression.

This code scores any candidate against a fixed set using KodeKey as the model endpoint, so you can baseline a prompted model today and compare a tuned one later.

import json, os, statistics, requests

KODEKEY_URL = "https://api.ai.kodekloud.com/v1/chat/completions"
HEADERS = {"Authorization": f"Bearer {os.environ['KODEKEY_API_KEY']}"}

def classify(ticket: str, model: str, system: str) -> dict:
    body = {"model": model, "temperature": 0,
            "messages": [{"role": "system", "content": system},
                         {"role": "user", "content": ticket}]}
    r = requests.post(KODEKEY_URL, headers=HEADERS, json=body, timeout=60)
    r.raise_for_status()
    raw = r.json()["choices"][0]["message"]["content"]
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return {"_schema_error": raw[:200]}

def evaluate(cases: list, model: str, system: str) -> dict:
    correct = schema_ok = 0
    for case in cases:
        out = classify(case["ticket"], model, system)
        if "_schema_error" not in out:
            schema_ok += 1
            if all(out.get(k) == v for k, v in case["expected"].items()):
                correct += 1
    n = len(cases)
    return {"model": model, "n": n,
            "schema_compliance": round(schema_ok / n, 3),
            "exact_match": round(correct / n, 3)}

SYSTEM = """Classify the support ticket. Reply with only a JSON object and no other text.

Use exactly these keys: category, urgency, team.
category must be one of: access, capacity, deployment, networking, other.
urgency must be one of: low, normal, high.
team must be one of: platform, infra, security, appdev.
If the ticket does not clearly fit a category, use other with urgency normal."""

if __name__ == "__main__":
    cases = json.load(open("heldout.json"))
    for model in ("claude-haiku-4-5",):
        print(evaluate(cases, model, SYSTEM))

Four properties of that script are what make it a decision tool rather than a demo.

Two metrics are reported rather than one, and the separation matters. Schema compliance asks whether the output parsed at all, which is the thing fine tuning most reliably improves. Exact match asks whether the answer was right. A tuned model frequently lifts the first number dramatically and the second barely, and if you only track accuracy you will conclude tuning did nothing when it in fact fixed your parsing failures.

Temperature is zero so the same case scores the same way twice, which is the minimum requirement for comparing two models at all.

The system prompt is the rung one attempt, written properly with an explicit enum for every field and a stated fallback. Score this first. If it hits your bar, you have finished, and you found out in an afternoon rather than three weeks.

Keeping the held out cases in a file rather than in the code means so the same set scores the prompted baseline today and the tuned candidate later. That file is the durable asset here, more than any model you produce.

Learn the model and API layer underneath

Every rung on this ladder sits on the same foundation of models, prompts, and API calls, and being fluent there is what lets you move between rungs quickly rather than committing early. The Introduction to OpenAI course on KodeKloud covers working with models, prompts, and APIs from first principles.

When the answer is both

The framing as a versus is useful for deciding and misleading as a destination, because most production systems in 2026 run both.

Retrieval keeps the knowledge current and citable. Tuning locks in the behaviour that should not change. A support assistant might use a tuned small model to guarantee the response format and a refusal style, with retrieval supplying the current policy text that the tuned model then formats. Each does what it is good at.

The sequencing still matters even when you end up with both. Prompt first to find out what the base model can already do, add retrieval for the knowledge gap, and only then tune the residual behaviours that retrieval could not fix, because those residual behaviours are only visible once the retrieval baseline exists.

Try this on your own use case

An afternoon, and the first half is free.

Take one thing your team wants a model to do and write down what a wrong answer looks like. Not what a right answer looks like, the wrong one, because that is the question that places you on the ladder. Wrong because it did not know, wrong because it expressed it badly, or right but too slow.

Then build twenty held out cases with known correct answers, drawn proportionally from real traffic and deliberately topped up with the ambiguous ones. Write the rung one prompt properly, with explicit enums and a stated fallback, and score it with the code above.

Look at the two numbers separately. If schema compliance is high and accuracy is low, you have a knowledge problem and rung two is next. If schema compliance is low, you have a form problem and better prompting will move it before any training would. If both are high, you have finished and the remaining question is only whether it is fast and cheap enough, which is a rung four question rather than a rung three one.

Whatever you find, you now have a baseline number and a reusable set, which is the thing that makes the next decision an argument about evidence rather than about preferences.

What you should be able to answer now

Four questions, and each one points back at a section.

For your current use case, is the gap form or facts? This single distinction resolves most of the argument, and the test is whether a wrong answer is wrong because the model lacked information or because it expressed what it had badly.

What does your rung one attempt score? If you cannot answer, you do not yet know whether the more expensive rungs are needed, and finding out costs an afternoon.

Do you have twenty held out cases with known answers? Without them, any claim that tuning improved something is a belief. With them, it is a measurement, and the set costs less to build than a single training run costs to waste.

If requirements changed next month, what would it cost you? A prompt changes in seconds, a retrieval index changes when you re index, and a tuned model changes when you retrain and re evaluate. If your requirements are still moving, that answer tells you which rung you belong on today.

The team from the opening did not fail because fine tuning is bad. They failed because their gap was facts and they reached for the tool that fixes form. One question, asked before the first training run, would have sent them to retrieval in week one and saved three weeks and a compute bill. The question was only ever whether a wrong answer was wrong because the model did not know, or because it did not behave.

Ready to Build Each Rung Properly?

Knowing which rung your problem sits on is half the work, and the skills for each rung are learnable in order. The Introduction to OpenAI course on KodeKloud covers prompting and the API layer that rung one lives on, the Fundamentals of RAG course covers the retrieval rung that resolves most real requests, and the KodeKloud playgrounds include an AI playground reachable through a single key so you can score a baseline today. Start at rung one.

FAQs