How AI Chatbots Are Built

A chatbot is a list of messages you resend in full on every turn. The model remembers nothing, and continuity is something you construct.

Ask a chatbot your name, tell it, then ask again three turns later and it answers correctly. That looks like memory. It is not. The model got your name again in that third request, along with every other message in the chat. It kept nothing at all between calls. Continuity is a trick your app pulls off by resending everything, every single time.

Once that lands, the rest follows. Why long chats cost more. Why they eventually hit a wall. Why a bot sometimes forgets something you said. And why every real system has a plan for throwing messages away. This tutorial builds a working chatbot in five stages. Each stage adds one thing and explains what it costs. We use KodeKey, so you need one key rather than an account per provider.

How AI chatbots are built, the core idea

An AI chatbot is a loop. It keeps a list of messages. It sends the whole list to a model each turn. It adds the reply to the list. Then it repeats. The model holds no state at all, so your app holds it instead. Memory is just history, resent.

That is the entire architecture. Everything else in this post is a refinement of it.

What you need

Python 3.10 or later, the requests library, and a KodeKey API key. Grab one from the KodeKey playground. It gives you several models through one OpenAI compatible endpoint.

python3 -m venv .venv && source .venv/bin/activate
pip install requests
export KODEKEY_API_KEY="your-key-here"

A virtual environment keeps this project's packages away from your system Python. That matters more than it seems, the first time two projects disagree about a version. And export the key rather than pasting it into the file. A key committed to a repo is the most common leak in AI projects, so build that habit now.

Stage one, the smallest possible chatbot

Here is a working chatbot. It is twenty lines and it demonstrates the entire mechanism.

import os, requests

URL = "https://api.ai.kodekloud.com/v1/chat/completions"
HEADERS = {"Authorization": f"Bearer {os.environ['KODEKEY_API_KEY']}"}
MODEL = "claude-haiku-4-5"

def chat(messages: list) -> str:
    body = {"model": MODEL, "messages": messages, "temperature": 0.7}
    r = requests.post(URL, headers=HEADERS, json=body, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

def main() -> None:
    messages = []
    while True:
        user_input = input("\nYou: ").strip()
        if user_input.lower() in {"exit", "quit"}:
            break
        messages.append({"role": "user", "content": user_input})
        reply = chat(messages)
        messages.append({"role": "assistant", "content": reply})
        print(f"\nBot: {reply}")

if __name__ == "__main__":
    main()

Run it and it works. Ask it your name, tell it, ask again, and it remembers.

The important bit is the messages list. Every question and every reply gets added to it. The whole list goes to the model on every call. Nothing is kept on the model's side. Send only the latest message and the bot would have no idea what you meant.

Two other choices are worth a mention. The raise_for_status call turns an HTTP error into a proper exception. Without it, a failed request gives you a confusing key error when the code hunts for a message that is not there. And temperature at 0.7 keeps replies varied. A task needing the same answer twice would set it near zero.

Stage two, the system prompt

The bot works. It has no personality yet. A system prompt fixes that, and it is just the first message in the list.

SYSTEM = """You are a support assistant for a Kubernetes platform team.

Answer concisely, in at most three short paragraphs.
Use British spelling.
When a question involves running a command, show the exact command in a code block.
If you do not know something specific to this environment, say so and name
what information would be needed rather than guessing.
Never invent cluster names, namespaces, or version numbers."""

def main() -> None:
    messages = [{"role": "system", "content": SYSTEM}]
    ...

Four things in that prompt do real work. Naming the domain gives the model a frame for vague questions. Capping the length stops the wall of text you get by default. The line about unknown details turns silent invention into a visible ask. That is the best line you can write for any bot near a real system. And the last line closes off a specific failure, where a model invents a namespace that sounds right and does not exist.

The system prompt goes out again on every request, as the first message. So it costs tokens every turn. A five hundred token prompt across fifty turns is twenty five thousand tokens of instruction. Worth knowing before you write a two page persona.

Stage three, streaming

Waiting eight seconds in silence feels broken. Streaming sends tokens as they are made. That changes nothing about the answer and a lot about how it feels.

import json

def chat_stream(messages: list):
    body = {"model": MODEL, "messages": messages,
            "temperature": 0.7, "stream": True}
    with requests.post(URL, headers=HEADERS, json=body,
                       stream=True, timeout=120) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line or not line.startswith(b"data: "):
                continue
            payload = line[6:]
            if payload == b"[DONE]":
                break
            delta = json.loads(payload)["choices"][0].get("delta", {})
            if "content" in delta:
                yield delta["content"]

The reply arrives as server sent events. That is why each line starts with data: , and why the code strips those six bytes before parsing. The [DONE] marker ends the stream. Skipping blank lines matters too, because the protocol uses them to separate events.

Written as a generator, the function yields each fragment as it arrives, which lets the caller print progressively.

        print("\nBot: ", end="", flush=True)
        chunks = []
        for piece in chat_stream(messages):
            print(piece, end="", flush=True)
            chunks.append(piece)
        reply = "".join(chunks)
        messages.append({"role": "assistant", "content": reply})

Collecting the pieces matters as much as printing them. You still need the full reply to add to the history. Here is a common first bug. You stream to the screen and forget to store the result. Now the bot answers, then has no memory of having done so.

The flush=True argument pushes output straight out rather than waiting for a newline. Leave it off and buffering swallows the whole point of streaming.

Learn the model and API layer properly

Everything in this build sits on the chat completions API. Understand that layer and each choice here stops looking like magic. The Introduction to OpenAI course on KodeKloud covers models, prompts, and APIs through hands on projects, using the same request shape this tutorial uses.

Stage four, memory and the context window

Here is where the fact from the top becomes a real problem.

Every turn resends the whole chat, so the request grows. Eventually it passes the model's context window and fails outright. Before that, it gets costly. And the growth is not a straight line.

Turn one sends one message. Turn ten sends nineteen. Turn fifty sends ninety nine. Total tokens grow with the square of the chat length. That is why a long support session costs far more than the message count suggests.

Four strategies exist. Each one throws away something different.

The sliding window is the simplest. It works well for short chats.

MAX_TURNS = 10

def trim(messages: list, max_turns: int = MAX_TURNS) -> list:
    system = [m for m in messages if m["role"] == "system"]
    rest = [m for m in messages if m["role"] != "system"]
    return system + rest[-(max_turns * 2):]

Pulling the system message out before trimming is the part that matters. It sits first in the list. So a lazy trim that keeps the last twenty entries will eventually throw it away. Then the bot loses its instructions and turns into a plain assistant. That one is nasty to debug, because nothing errors. The personality just quietly evaporates.

Multiplying by two reflects that each turn contributes a user message and an assistant reply.

For longer chats, a summary works better. It keeps the shape of what came before.

SUMMARISE = ("Summarise this conversation in under 150 words. "
             "Preserve decisions made, specific names and numbers mentioned, "
             "and any unresolved questions. Omit pleasantries.")

def compress(messages: list, keep_recent: int = 6) -> list:
    system = [m for m in messages if m["role"] == "system"]
    rest = [m for m in messages if m["role"] != "system"]
    if len(rest) <= keep_recent + 4:
        return messages
    old, recent = rest[:-keep_recent], rest[-keep_recent:]
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    summary = chat([{"role": "user", "content": f"{SUMMARISE}\n\n{transcript}"}])
    return system + [{"role": "assistant",
                      "content": f"Summary of earlier conversation: {summary}"}] + recent

Telling it to keep decisions, names, numbers, and open questions is what makes this usable. A generic summary drops exactly the details a later turn will need. Keeping the newest messages word for word matters too, since the current thread should not be paraphrased.

Note that a summary costs its own model call. That is why it only fires past a threshold, not every turn.

Stage five, what it costs

Cost catches people out, because it does not track the number of messages.

Those figures assume a 200 token system prompt and about 200 tokens per exchange. The shape matters more than the exact numbers. A fifty turn chat processes roughly twenty times the input of a ten turn one, despite being only five times longer.

Four levers control this. Trim hard, because most chats do not need thirty turns of history. Keep the system prompt tight, since you pay for it on every request. Match the model to the job, because a small fast model handles most turns just fine. And cache where you can. Several providers discount repeated prefixes, which suits a stable system prompt exactly.

def estimate_tokens(messages: list) -> int:
    return sum(len(m["content"]) for m in messages) // 4

def main() -> None:
    messages = [{"role": "system", "content": SYSTEM}]
    total_input = 0
    while True:
        user_input = input("\nYou: ").strip()
        if user_input.lower() in {"exit", "quit"}:
            print(f"\nApproximate input tokens this session: {total_input:,}")
            break
        messages.append({"role": "user", "content": user_input})
        messages = trim(messages)
        total_input += estimate_tokens(messages)
        ...

Dividing characters by four roughly matches English tokens. Close enough to watch a trend, without adding another package. Printing the running total on exit is the part worth keeping. Watching that number climb across a real chat makes the growth concrete in a way the table above cannot.

The complete build

Stitch the stages together and you get a chatbot with a persona, streaming output, memory handling, and cost tracking. About eighty lines.

import json, os, requests

URL = "https://api.ai.kodekloud.com/v1/chat/completions"
HEADERS = {"Authorization": f"Bearer {os.environ['KODEKEY_API_KEY']}"}
MODEL = "claude-haiku-4-5"
MAX_TURNS = 10

SYSTEM = """You are a support assistant for a Kubernetes platform team.

Answer concisely, in at most three short paragraphs.
When a question involves running a command, show the exact command in a code block.
If you do not know something specific to this environment, say so and name
what information would be needed rather than guessing.
Never invent cluster names, namespaces, or version numbers."""

def trim(messages: list) -> list:
    system = [m for m in messages if m["role"] == "system"]
    rest = [m for m in messages if m["role"] != "system"]
    return system + rest[-(MAX_TURNS * 2):]

def estimate_tokens(messages: list) -> int:
    return sum(len(m["content"]) for m in messages) // 4

def chat_stream(messages: list):
    body = {"model": MODEL, "messages": messages, "temperature": 0.7, "stream": True}
    with requests.post(URL, headers=HEADERS, json=body, stream=True, timeout=120) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line or not line.startswith(b"data: "):
                continue
            payload = line[6:]
            if payload == b"[DONE]":
                break
            delta = json.loads(payload)["choices"][0].get("delta", {})
            if "content" in delta:
                yield delta["content"]

def main() -> None:
    messages = [{"role": "system", "content": SYSTEM}]
    total = 0
    print("Type exit to finish.")
    while True:
        try:
            user_input = input("\nYou: ").strip()
        except (EOFError, KeyboardInterrupt):
            break
        if not user_input:
            continue
        if user_input.lower() in {"exit", "quit"}:
            break

        messages.append({"role": "user", "content": user_input})
        messages = trim(messages)
        total += estimate_tokens(messages)

        print("\nBot: ", end="", flush=True)
        chunks = []
        try:
            for piece in chat_stream(messages):
                print(piece, end="", flush=True)
                chunks.append(piece)
        except requests.HTTPError as exc:
            print(f"\n[request failed: {exc}]")
            messages.pop()
            continue
        print()
        messages.append({"role": "assistant", "content": "".join(chunks)})

    print(f"\nApproximate input tokens this session: {total:,}")

if __name__ == "__main__":
    main()

Two error handling details are worth copying. Catch the HTTP error and drop the message that failed. Otherwise you leave an unanswered question in the list, the next request has two user turns in a row, and the model gets confused. And catch the keyboard interrupt so the token summary prints on exit rather than dumping a stack trace. A small courtesy that makes a command line tool feel finished.

Where each design choice bites, by role

The same twenty line loop, deployed for three different purposes, and the constraint that dominates changes each time.

The DevOps engineer building an incident helper. The bot answers questions during an incident, which makes latency the dominant constraint, since a responder will not wait eight seconds. Streaming is mandatory rather than nice here, and the model choice should favour speed over depth. The design detail that matters most is a short system prompt, because during an incident the questions are short and the prompt would otherwise dominate every request.

The cloud engineer building a cost query assistant. Questions arrive infrequently and each one needs current data, which flips the priorities entirely. Latency barely matters. Memory barely matters either, since each question stands alone. What matters is that the bot reads real numbers rather than inventing ones that sound right. This is the case where a chatbot alone is the wrong shape and retrieval is required, which is the next step described below.

The platform engineer building an internal documentation bot. Many users, long sessions, and a corpus that changes weekly. Every constraint in this tutorial applies at once. Memory strategy matters, because sessions run long. Cost matters, because usage multiplies across teams. And the system prompt has to be strict about not inventing internal details, since the audience will believe it. This is the deployment where summarisation earns its extra model call and where pinning key facts outside the summary becomes necessary rather than optional.

Where to take it next

Four directions, roughly in order of use.

Give it your documents. Search your own content and drop the useful bits in before the question. Now a general assistant knows your systems. This is retrieval augmented generation, and it is the usual next step.

Give it tools. Let the model ask for an action, like fetching a status or hitting an API. That turns a chatbot into an agent, and it changes the security questions a lot.

Put a web front end on it. The console loop shows the mechanism. A small web framework with a streaming endpoint is a short step from here.

Save the chats. Store the message list against a session ID and a chat survives a restart. That is a database write, not an AI problem.

Every one of those keeps the same core loop. The list of messages, resent each turn, is still the whole design.

Practise with several models through one key

Different models handle the same prompt in different ways. Comparing them side by side is the fastest way to learn which to use where. The KodeKey playground on KodeKloud gives you access to several through a single key, so swapping the MODEL constant is the whole experiment.

Try this yourself

An afternoon. Each step teaches something the one before set up.

Build stage one exactly as written and check it works. Then delete the line that adds the reply to the message list, and talk to it again. The bot forgets everything at once. That shows you the core idea better than any explanation.

Next, write a system prompt for a domain you know well. Watch how much the same questions change. Then make it contradict itself. Ask for brevity and full detail at once. Watch how the model settles the clash, because that tells you a lot about how it weighs instructions.

Add streaming. Time the gap between the first visible token and the full reply. Total time barely moves. Felt speed changes hugely. That is the whole argument for it.

Finally, set MAX_TURNS to two and have a longer chat. Watch the bot forget something you said four turns back. Every memory strategy in this post suddenly feels necessary rather than academic.

What you should be able to answer now

Four questions, each pointing at a stage.

Why does a chatbot forget things? Because your app threw them away, not the model. The model never remembered anything. So every forgetting bug lives in your trim or summary code.

Why does a long chat cost more than one long question? Because every turn resends everything before it. Total input grows with the square of the chat length, not in a straight line.

What happens if your trim function drops the system message? The bot loses its instructions and turns generic. There is no error to point at. That is why you pull the system message out before trimming.

Where would you enforce a rule the bot must never break? In your code. Check the output and limit what it can do. A system prompt is an instruction, not a wall, and users can ask a model to ignore it.

Come back to the trick from the top. The bot knew your name because you told it again, in that same request, along with everything else. Every chatbot you have used works this way. Every design choice in this tutorial exists because of it.

Ready to Build Something Bigger?

What you built here is the foundation for retrieval, tools, and agents, all of which keep the same message list at their centre. The Introduction to OpenAI course on KodeKloud covers the model and API layer through projects. The Fundamentals of RAG course covers giving your bot your own documents. And the KodeKey playground gives you several models through one key, so you can compare them yourself. Start by extending what you just built.

FAQs