Understanding LLMs for DevOps Beginners

The model does not read your YAML. It reads a list of numbers standing in for chunks of it, and that changes what you should expect.

There is a party trick that reveals more about language models than any architecture diagram. Ask one how many times the letter r appears in a word, and watch a system that can write a working Terraform module get a question a child could answer wrong. It is not a gap in intelligence, it is a gap in perception, because the model never saw the letters at all. It saw a handful of numbered chunks standing in for the word, and counting letters inside a chunk it cannot look into is genuinely impossible for it.

That single mechanical fact is the best entry point into large language models for anyone from an infrastructure background, because almost every behaviour that seems strange traces back to it. Costs are quoted per million tokens because tokens are the unit of work. Your YAML burns through a context window faster than prose because structured text tokenises badly. Long sessions lose track of earlier decisions because older tokens fall out of the window. This guide starts with what the model actually sees and builds outward until the behaviour you have already noticed makes sense.

What a large language model is

A large language model is a neural network trained on enormous amounts of text to predict the next token in a sequence, which, repeated one token at a time, produces text that appears to answer questions, write code, and follow instructions, without the model retrieving facts from a database or executing anything.

Two phrases in that definition are load bearing. Next token means it generates one small chunk at a time rather than composing a whole answer and then writing it. Predict means it produces what is statistically plausible given everything before, which is not the same thing as what is true.

What the model actually sees

Start here, because every later section depends on it.

Before a model processes your text, a tokeniser splits it into pieces and replaces each piece with a number. A token is roughly three to four characters of English, so a useful rule of thumb is about 100 tokens per 75 words, meaning a thousand word document lands near 1,300 tokens. Common words are often a single token, while unusual ones get split into several.

The model then works entirely with those numbers. It has no access to the original characters, which is exactly why letter counting fails, and why asking a model to reverse a string or check spelling gives unreliable results while asking it to explain a Kubernetes controller works fine.

For infrastructure work, the more useful consequence is that different content tokenises very differently.

How it generates an answer

Once your text is tokens, generation is a loop that is simpler than most people expect.

The model reads everything it has been given and produces a probability distribution over every token it knows, essentially a ranked list of what could plausibly come next. One token is selected from that list. That token is appended to the input, and the whole thing runs again. This repeats until the model produces a token meaning stop, or until it reaches a length limit.

Two practical points follow immediately.

Generation is sequential, so a long answer takes longer than a short one in a way that is roughly linear. That is why streaming exists, and why asking for concision genuinely saves time and money rather than just being polite.

Selection involves randomness, controlled by a setting called temperature. At a temperature near zero the model almost always picks the highest probability token, making output close to deterministic. Higher temperatures let it pick lower ranked tokens more often, producing more varied and more surprising text.

The context window

The context window is the maximum number of tokens the model can process in one request, and it is the constraint that shapes most real applications.

Everything shares that one budget. Your system instructions, the conversation so far, any files you pasted, any tool results, and the model's own response all come out of the same allowance. A model with a 200,000 token window that receives 190,000 tokens of input has only 10,000 left for its answer.

Windows have grown enormously, from 512 tokens in early models to hundreds of thousands or millions today, which is roughly a twenty thousand fold increase since 2018. That growth is real and it has not made the limit irrelevant, for two reasons.

The first is that the model has no memory outside the window. It does not remember your previous conversation, and it never will, unless your tooling explicitly resends that information. What feels like memory in a chat product is the product replaying the transcript to the model every time you send a message.

A second reason is that filling a large window does not mean the model uses all of it equally well. Recall of details buried in the middle of a very long context is measurably weaker than recall of things near the beginning or end, and a model that handles 50,000 tokens cleanly may start making mistakes at 120,000. More room is not the same as more attention.

Why models hallucinate

Hallucination sounds like a malfunction. It is closer to the system working exactly as designed, in a situation where the design does not fit.

The model's objective is to produce a plausible next token, and plausibility is judged against the patterns in its training data rather than against reality. When you ask about something well represented in that data, plausible and true usually coincide. When you ask about something rare, internal to your company, or newer than the training data, they come apart, and the model produces confident text that follows the right shape while being wrong.

That shape matching explains the specific flavour of hallucination engineers meet most often. Ask for a command line flag that does not exist and you will frequently get one that looks exactly like a real flag for that tool, because the model is generating something that fits the pattern of flags rather than recalling a documented list.

Four things reduce it meaningfully.

Give the model the facts in the prompt, since a model reading your actual documentation is far more accurate than one recalling something similar. Ask for sources or reasoning so you have something checkable. Verify mechanically whenever the output has a checkable form, which for infrastructure work is often, because YAML parses or it does not and a command exists or it does not. And keep temperature low, since higher randomness increases the chance of an unusual and wrong token being chosen.

Four ways to change what a model does

This is the decision that beginners most often get wrong, usually by reaching for the most expensive option first.

The most common mistake is jumping to fine tuning when retrieval was the answer. Fine tuning changes how the model behaves and does not reliably teach it new facts, so if the problem is that the model does not know your runbooks, feeding it those runbooks at query time works better and costs a fraction as much. Fine tuning suits the case where the model already knows enough but keeps producing the wrong shape of output.

Retrieval augmented generation, usually shortened to RAG, is the pattern behind most useful internal AI tools. Your question is used to search a document store, the most relevant passages are inserted into the prompt, and the model answers from those passages. Nothing about the model changed, and the accuracy improvement can be dramatic because the model is now reading rather than recalling.

Build the model and API foundations

Everything above becomes concrete once you have called an API yourself, watched token counts move, and seen temperature change the output. The Introduction to OpenAI course on KodeKloud covers working with models, prompts, and APIs from first principles, which is exactly the layer this guide describes.

Where LLMs for DevOps genuinely help, and where they do not

Being specific is more useful than being enthusiastic, and the honest picture is genuinely good in places and genuinely poor in others.

The pattern is consistent. Models are strong where the task is pattern shaped, where the answer is checkable, and where being wrong is cheap. They are weak where the task needs exact recall, precise counting, or knowledge of your specific environment.

How a model got the way it is

You do not need the mathematics, and knowing the three stages explains a surprising amount of everyday behaviour.

Pretraining is where the model reads an enormous quantity of text and learns to predict the next token. This is where essentially all of its knowledge comes from, it is by far the most expensive stage, and it is why models have a training cutoff date beyond which they know nothing. A model that has never seen your company's internal tooling did not skip it, it simply was not there.

Instruction tuning comes next, training the model on examples of requests paired with good responses. A model straight out of pretraining continues text rather than answering questions, so if you typed a question it might produce three more questions, since that is a plausible continuation. Instruction tuning is what turns a text continuation engine into something that behaves like an assistant.

Preference tuning refines it further using human judgments about which of two responses is better. This shapes tone, helpfulness, and refusal behaviour, and it is why different models feel noticeably different to work with even when their raw capability is similar.

Embeddings, the other thing models produce

Most beginners meet only text generation, and the second output type is worth knowing because it underpins search, retrieval, and most useful internal AI tooling.

An embedding is a list of numbers representing the meaning of a piece of text, produced by a model built for that purpose. Text with similar meaning produces numerically similar lists, and that similarity can be measured. The consequence is that you can search by meaning rather than by keyword, so a query about a service failing to start can retrieve a runbook titled crash loop troubleshooting even though the two share almost no words.

This is the machinery underneath retrieval augmented generation described earlier. Your documents are split into chunks, each chunk is embedded once and stored in a vector database, and at query time your question is embedded and compared against the store to find the closest matches, which are then inserted into the prompt.

Three practical points matter more than the theory.

Chunking quality dominates the results, because a chunk that splits a procedure in half retrieves badly no matter how good the embedding model is. Chunking by natural document structure, meaning sections and headings, consistently beats chunking by fixed character counts.

Embedding models are separate from generation models, they are much cheaper, and you must use the same one for storing and querying, since embeddings from different models are not comparable.

Retrieval quality is the ceiling on answer quality. If the right passage is not retrieved, no amount of prompting will help, so when a retrieval system gives poor answers, check what it actually retrieved before blaming the model.

Choosing a model without overthinking it

Model names change constantly and the selection criteria do not, so learn the axes rather than memorising a leaderboard.

Two habits make this easier than it looks. Start with a mid tier model and only move up when you can point at a specific failure it produced, since a great deal of money is spent on frontier models doing work a cheaper one handled fine. And use different models for different jobs in the same system, because summarising log lines and diagnosing a subtle race condition are not the same task and do not need the same engine.

Prompting that actually works for infrastructure

Prompting is the cheapest lever and the one most often used badly. Five habits cover the majority of the improvement.

Give the model the context it cannot have. It does not know your Kubernetes version, your cloud provider, your conventions, or your constraints. A prompt that states these produces dramatically better output than one that assumes them, and this single habit fixes more bad answers than any other.

Ask for the format you want. If you need YAML with no commentary, say so. If you want a table, ask for a table. Models follow format instructions well, and much of what feels like a bad answer is a good answer in an inconvenient shape.

Show one example when the shape is unusual. For anything with house conventions, pasting one correct example teaches more than a paragraph of description, because the model is very good at pattern continuation.

Ask it to work through the problem before answering on anything with multiple steps. Because generation is sequential, text the model produces becomes input to what it produces next, so reasoning written out first genuinely improves the conclusion rather than merely documenting it.

Structure the input, not just the request. Models parse clearly delimited input far more reliably than a wall of pasted text, so labelling sections with something like an error section, a config section, and a question section produces better answers than running them together. This costs a few seconds and removes an entire class of confusion where the model answers about the wrong part of what you sent.

Tell it what to do when unsure. Left unspecified, a model will produce its best guess with full confidence. Instructing it to say when something is uncertain, or to name what extra information it would need, converts silent guesses into visible ones.

One more habit is worth naming because it saves the most time overall. When an answer is wrong, resist the urge to retry the same prompt hoping for better luck, and instead work out which of the four causes applies: the model lacked context you could have supplied, the request was ambiguous about format or scope, the task genuinely needs exact recall the model cannot have, or the temperature is high enough that you are sampling noise. Each of those has a different fix, and retrying blindly addresses none of them.

Cost, latency, and how to reason about them

Two numbers govern almost everything practical.

Cost is charged per token, usually quoted per million, and input and output are priced differently with output typically much more expensive. That asymmetry matters because it means a long question with a short answer is often cheaper than a short question with a long answer, which is the opposite of most people's intuition.

Latency comes mostly from generating output tokens, since input is processed in parallel while output is produced one token at a time. A request that reads a large document and returns three sentences will usually feel faster than one that reads a sentence and returns three pages.

Three habits keep both under control. Send only what is needed, since filtering logs before pasting them reduces cost and improves accuracy at the same time. Ask for concise output explicitly, as it is the single biggest lever on both bills and waiting. And match the model to the task, because a small fast model handles classification and summarisation perfectly well while a larger one is worth its cost for genuinely difficult reasoning.

Using a model through an API

Nothing here requires a framework, and seeing the raw shape of a request demystifies a lot.

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "temperature": 0,
    "max_tokens": 500,
    "messages": [
      {"role": "system", "content": "You are a Kubernetes assistant. Answer concisely and say when you are unsure."},
      {"role": "user", "content": "A pod is in CrashLoopBackOff. What should I check first?"}
    ]
  }'

That request contains everything this guide has described. The system message sets behaviour for the whole conversation and is where instructions about tone, format, and uncertainty belong. The user message is the actual question. Temperature at zero makes the answer close to deterministic, which is what you want for infrastructure guidance. And max_tokens caps the length of the response, which protects you from an unexpectedly long and expensive answer.

The response comes back with the generated text and, importantly, a usage block reporting how many input and output tokens the request consumed. Reading that number a few times is the fastest way to build intuition about what things cost, and it is worth deliberately trying the same question with a small pasted log and a large one to watch the difference.

Running models yourself

Using a hosted API is the right default, and running models locally is worth understanding because it fits some situations well.

Local models run on your own hardware, which means no data leaves your environment, no per token cost, and no dependency on a provider's availability. They cost you hardware, operational effort, and capability, since a model small enough to run on a laptop will not match a frontier model on difficult reasoning.

The decision usually comes down to three questions. If your data cannot leave your environment for regulatory reasons, local becomes attractive regardless of other factors. If your volume is enormous and the task is simple, owning the hardware may cost less than paying per token. And if the task genuinely needs frontier reasoning quality, a hosted model is usually the pragmatic answer.

For most teams starting out, hosted APIs with a clear policy about what may be sent is the right first step, and local models are worth revisiting once you know what you are actually running and why.

Practise with real environments

Reading about tokens and temperature is one thing, and watching a token count climb as you paste a bigger file is quite another. The KodeKloud playgrounds give you environments where you can call models, run experiments, and see the behaviour described here with your own inputs rather than taking anyone's word for it.

Where to start

1. Run a paragraph of prose and an equivalent length of YAML through a token counter, and see for yourself how differently they consume budget.
2. Ask a model the same question at temperature zero and at temperature one, twice each, and watch what changes.
3. Take a question your team asks often and write a prompt that supplies the context the model cannot know, then compare answers with and without it.
4. Deliberately ask about something internal to your company and read how confidently it answers, so you experience the hallucination mode rather than only reading about it.
5. Check the usage numbers on a few API calls until you have intuition for what your typical request costs.
6. Write down what may and may not be pasted into a prompt, and share it with your team, because that policy is easier to agree before an incident than after one.
7. Pick one repetitive writing task, such as drafting runbook sections, and use a model on it for a week to build a realistic sense of where it helps.

Conclusion

Language models stop being mysterious the moment you accept what they are, which is systems that read numbered chunks of text and produce whatever chunk is most plausible next. Everything that seems strange follows from that. They miscount letters because they cannot see letters. They fill context windows fast on YAML because structured text tokenises poorly. They hallucinate confidently because plausibility is the objective and there is no separate check for truth.

Understanding LLMs for DevOps well does not require any mathematics. It requires knowing that tokens are the unit of everything, that context is a budget shared by all parts of a request, that generation is sequential and randomised, and that the model knows nothing about your environment unless you tell it. Those four facts are enough to use these tools deliberately rather than hopefully.

So spend an afternoon on the experiments above, particularly the one where you ask about something only your company knows. Watching a model be fluent and wrong about your own systems is the most useful lesson available, and it will make you far better at knowing when to trust the answer and when to check it.

Ready to Go Deeper on AI Foundations?

The concepts here reward hands on time far more than further reading. The Introduction to OpenAI course on KodeKloud covers models, prompts, and APIs from first principles, the LangGraph course takes you into building systems on top of them, and the KodeKloud playgrounds give you somewhere to run every experiment in this guide. Start with one today.

FAQs