You ask a chat assistant a question about your own infrastructure. The answer comes back fluent, confident, and thin. You suspect a better model would do better, so you open a second tab, log into a different provider, and start re-pasting: the manifest, the error, the three clarifications you already gave. By the time the second model has the context the first one already had, you have spent four minutes retyping your own meeting.

That friction is not a property of AI. It is a property of the product you are using, which is owned by a company that sells exactly one family of models. This post fixes it with about seventy lines of Python. You will build a chat app that runs on your machine, puts every model the gateway offers in a dropdown, and lets you switch model in the middle of a conversation without losing a single line of history. Ask a cheap model, get an answer, switch to a stronger one in the same thread, and watch it pick up where the other left off.

You need Python 3.9 or newer, a terminal, and about twenty minutes. No database, no framework, no front-end code.

Why a Dropdown Is the Whole Point

A chat window is not interesting. You have used a hundred of them. The interesting part is the list of models next to it, because that list is normally impossible to build.

Normally, every entry in that dropdown is a separate company: a signup, a key, a billing relationship, and an SDK with its own parameter names. Building a five-model dropdown means doing that five times before you write a line of UI, which is why almost nobody builds one and why every chat product you have used locks you to one lab. The previous post in this series is about how that sprawl happens and why it is structural rather than a discipline problem.

When many models sit behind one OpenAI-compatible endpoint, the dropdown stops being an integration project and becomes what it looks like: a list of strings. That is the entire reason this app is short.

Before You Start

Create a folder, a virtual environment, and install the two dependencies.

mkdir private-chat && cd private-chat
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install streamlit openai

Check what you got. Version numbers matter later when something behaves differently than this post describes.

python -c "import streamlit, openai; print(streamlit.__version__, openai.__version__)"

1.61.1 2.53.0

Now the key. Generate one from your KodeKloud account on the KodeKey page, then put it in your environment rather than in your code. A key that lives in a file gets committed eventually.

export KODEKEY_API_KEY="your-key-here"     # Windows: setx KODEKEY_API_KEY "your-key-here"

Step 1: Prove the Key Works Before You Build Anything

Do not start with the UI. If the key is wrong, you want to find out now, from three lines, not from a browser tab showing a stack trace.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["KODEKEY_API_KEY"],
    base_url="https://api.ai.kodekloud.com/v1",
)

reply = client.chat.completions.create(
    model="claude-haiku-4-5-20251001",
    messages=[{"role": "user", "content": "Say OK."}],
)
print(reply.choices[0].message.content)

OK.

Two things are worth noticing in that snippet, because they are the only two things that make this whole post possible. The base_url points at a gateway instead of a single provider, and the model is a string. Everything else is the standard openai SDK doing what it always does. If you want the full mechanics of what just happened, what a token is, and where the money goes, that is the first post in this series.

Step 2: Ask the Gateway What It Has

The dropdown should not be a list you typed by hand, because that list goes stale the week a new model ships. Ask the endpoint instead.

models = sorted(m.id for m in client.models.list().data)
print(len(models))
print("\n".join(models[:8]))

39
alibaba/qwen3-coder-plus
claude-fable-5
claude-haiku-4-5-20251001
claude-opus-4-8
claude-opus-5
claude-sonnet-4-6
claude-sonnet-5
deepseek/deepseek-V3.2

A few dozen models from about ten labs, in one list, from one key. That list is your dropdown, and it updates itself.

One caveat that will save you a confusing five minutes later: this endpoint lists what exists, not what your particular key is allowed to call. Which models your plan includes is a property of your subscription, so a model can appear in the list and still refuse the call. Step 6 handles that properly instead of pretending it cannot happen.

Step 3: The Smallest Thing That Runs

Create chat_app.py with just the shell: a title, the dropdown, and an input box. No API calls yet.

import os

import streamlit as st
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["KODEKEY_API_KEY"],
    base_url="https://api.ai.kodekloud.com/v1",
)

st.title("My Private Chat")


@st.cache_data(ttl=3600)
def load_models():
    return sorted(m.id for m in client.models.list().data)


model = st.sidebar.selectbox("Model", load_models())
prompt = st.chat_input("Ask anything")

Run it.

streamlit run chat_app.py

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501

A browser tab opens with a sidebar dropdown listing every model, and a chat box at the bottom that does nothing yet. The @st.cache_data(ttl=3600) decorator matters more than it looks: Streamlit re-runs your entire script top to bottom on every single interaction, so without it you would call the models endpoint again every time you typed a character.

Step 4: Make It Answer, and Understand Why It Forgets

Add the call. This version works, and it is also subtly broken in a way worth seeing for yourself.

if prompt := st.chat_input("Ask anything"):
    with st.chat_message("user"):
        st.markdown(prompt)

    with st.chat_message("assistant"):
        reply = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],   # the bug
        )
        st.markdown(reply.choices[0].message.content)

Ask it "my name is Priya", then ask "what is my name?". It has no idea.

This is the single most important thing to understand about these APIs, and it trips up everyone once: the endpoint is stateless. It does not remember your last message, because there is no "your" and no "last". Each request is complete and isolated. Every chat product that appears to remember you is re-sending the entire conversation on every message, and so must yours.

Step 5: Give It a Memory, and a Receipt

Two changes. Keep the conversation in st.session_state, which is the only thing that survives a Streamlit re-run, and send all of it every time. While you are there, print what each answer cost.

Here is the finished file, 68 lines, 48 of them actual code.

import os

import streamlit as st
from openai import OpenAI, OpenAIError

client = OpenAI(
    api_key=os.environ["KODEKEY_API_KEY"],
    base_url="https://api.ai.kodekloud.com/v1",
)

st.set_page_config(page_title="My Private Chat", page_icon="*")
st.title("My Private Chat")


@st.cache_data(ttl=3600)
def load_models():
    """Ask the gateway which models exist, instead of hardcoding a list."""
    return sorted(m.id for m in client.models.list().data)


model = st.sidebar.selectbox("Model", load_models())
if st.sidebar.button("New chat"):
    st.session_state.messages = []

if "messages" not in st.session_state:
    st.session_state.messages = []

# Streamlit repaints the page top to bottom on every interaction, so the
# conversation has to be redrawn each time from session_state.
for turn in st.session_state.messages:
    with st.chat_message(turn["role"]):
        st.markdown(turn["content"])
        if turn["role"] == "assistant":
            st.caption(turn["receipt"])

if prompt := st.chat_input("Ask anything"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.markdown(prompt)

    # The whole conversation goes with every request. This is the line that
    # makes switching models mid-thread work: the model you just picked has
    # never seen this chat, so you hand it the entire history as if it had.
    history = [
        {"role": t["role"], "content": t["content"]}
        for t in st.session_state.messages
    ]

    with st.chat_message("assistant"):
        try:
            reply = client.chat.completions.create(model=model, messages=history)
        except OpenAIError as err:
            st.error(str(err))
            st.session_state.messages.pop()  # drop the question, keep the thread
            st.stop()

        answer = reply.choices[0].message.content
        # Not every model reports a cost, so fall back to the token count.
        cost = getattr(reply.usage, "cost", None)
        receipt = f"{model} - " + (
            f"${cost:.5f}" if cost else f"{reply.usage.total_tokens} tokens"
        )
        st.markdown(answer)
        st.caption(receipt)

    st.session_state.messages.append(
        {"role": "assistant", "content": answer, "receipt": receipt}
    )

That history list comprehension is the app. Everything else is presentation.

Note the getattr on the cost. When this post was written, models from most labs returned a per-call cost on the usage object and the Anthropic models did not, returning cache token counts instead. Reaching for reply.usage.cost directly would crash the app on half the dropdown, which is the kind of thing you only learn by running it.

Step 6: Switch Models Mid-Conversation

Now the part the whole app exists for. Start a conversation on a cheap model and ask it something with real money attached.

This is a genuine transcript from the finished app, captured on 11 August 2026. Long answers are trimmed where marked, and nothing else is edited.

YOU  -> A pod requests 250m CPU and runs 24/7. A second pod requests 1.5 CPU
        but only runs a 12-minute job every 6 hours. At $0.096 per vCPU-hour,
        which costs more over 30 days? Keep it under 120 words.

BOT  [google/gemini-3.1-flash-lite - $0.00041]
     Pod 1 (24/7):
     *   Usage: 0.25 vCPU x 720 hours = 180 vCPU-hours.
     *   Cost: 180 x $0.096 = $17.28.
     Pod 2 (Intermittent):
     *   Total vCPU-hours: 1.5 vCPU x 24 hours = 36 vCPU-hours.
     *   Cost: 36 x $0.096 = $3.46.
     Conclusion:
     Pod 1 costs significantly more ($17.28 vs. $3.46) because its continuous
     uptime outweighs Pod 2's higher resource request.

That answer is correct. The arithmetic checks out and the conclusion follows. If you were going to act on it, you would go and look at Pod 1.

Now open the dropdown, pick a model from a different lab, and keep typing in the same thread.

YOU  -> Is there anything in that answer that could still bite us in production?

BOT  [claude-opus-5 - 1952 tokens]
     [...] the math is right but the model behind it is optimistic:

     1. You probably don't pay per pod-hour at all.
     On standard EKS/GKE/AKS you pay for *nodes*, not requests. [...] The
     per-vCPU-hour model only holds for Fargate, Cloud Run, or GKE Autopilot.

     2. The 1.5 CPU request reserves capacity even when idle.
     If it's a long-lived Deployment that's idle 99% of the time, you're paying
     for 1.5 CPU of reserved headroom 24/7 (~$103/mo), not $3.46. Only a
     CronJob that actually terminates gets you the cheap number.

     3. Autoscaler tax.
     [...] a 12-min job can hold a node for 25+ min. Plus you pay for the
     *whole node*, so a 1.5 CPU request on a 4-vCPU node bills ~4 vCPU. [...]

Read what actually happened there. The second model did not correct an error, because there was no error. It answered a question about an answer it never produced, which means the entire prior conversation reached it intact. And the thing it added is the thing that matters: if that second pod is a long-running Deployment rather than a terminating job, the real number is about $103 a month, not $3.46. The cheap answer was right and would still have sent you to optimize the wrong pod.

One more turn, to make the continuity impossible to argue with.

YOU  -> So which of the two pods should I look at first on Monday?

BOT  [claude-opus-5 - 1756 tokens]
     Pod 2 - the 1.5 CPU one. It's the bigger blast radius and the bigger
     unknown. [...] if that 1.5 CPU request lives in a long-running Deployment
     rather than a CronJob that terminates, you're at ~$103/mo of reserved
     headroom instead of $3.46. That's a 30x error [...]
     Pod 1 can wait. It's $17/mo, the request is small, and it's steady-state.

Nothing in that question names a pod, a CPU figure, or a price. "The two pods" only means something to a model holding the whole thread. That is the payoff, and it costs you one list comprehension.

A note on honesty, because it would be easy to fake this section: the first draft of this tutorial was going to show a cheap model getting a question wrong and a strong one fixing it. Four candidate questions were tried first, on Kubernetes eviction deadlocks, HPA replica math, CronJob restart policies, and quota admission. The cheap model got all four right. Modern small models are genuinely good at questions with well-known answers, and the useful difference now is usually depth rather than correctness. That is a more useful thing to know than a staged failure.

What Just Happened

Three ideas carried the whole build, and they transfer to anything else you make with these APIs.

The endpoint has no memory. Conversation is an illusion you maintain by re-sending everything. Once that is obvious, a lot of odd behavior stops being odd, including why long chats get more expensive with every message: you are paying to re-read the whole thread each time.

The model is a parameter, not an architecture. Because model is a string in the same request shape, switching costs one dropdown selection rather than a second SDK, a second key, and a second invoice.

Every response carries a receipt. Printing the cost next to each answer changes how you use the app within a day. You stop reaching for the expensive model by reflex, because you can see what the reflex costs.

Common Errors and Fixes

These are the three that actually came up while building this, with their real messages.

The 403 is the one worth internalizing, because the naive version of this app crashes on it. A model appearing in GET /v1/models tells you the gateway knows about it, not that your subscription includes it. Handle the error and the dropdown stays usable; ignore it and one wrong selection takes the page down mid-conversation.

Clean Up

There is nothing running in the cloud and nothing to tear down. Stop the app with Ctrl+C, and if you are finished with the project entirely:

deactivate
rm -rf private-chat

Your conversations were never written to disk. They lived in st.session_state, which is memory attached to the browser session, and they are gone when you stop the app. That is a feature for anything you would not want in someone else's product, and a limitation the moment you want history tomorrow, which is the natural next thing to add.

Next Steps

The app is deliberately minimal so the ideas stay visible. Four upgrades, roughly in order of value:

1. A system prompt box in the sidebar, prepended to history as a {"role": "system"} message. This is the fastest way to feel how much behavior is controlled by instructions rather than model choice.
2. Persistence, by writing st.session_state.messages to a JSON file and reloading it on start. Twenty lines, and it turns a demo into something you keep using.
3. Streaming, using stream=True so tokens appear as they are generated instead of after a pause.
4. A running total of the session cost in the sidebar, which is the feature most likely to change your habits.

If you want the concepts under the code rather than more features, the AI Learning Path covers prompting, retrieval, and agents in order, and the official KodeKey examples repo has working starting points in Python, JavaScript, and curl.

Conclusion

The dropdown is the product. A chat window is a solved problem, but a chat window where the model is a runtime choice rather than a vendor commitment changes how you work: you stop treating "which model" as an architecture decision and start treating it like a font size, something you change when the current one is not working.

Sixty-eight lines, no database, no framework. The next post in this series takes the same one-key idea somewhere less obvious, using two models from two different labs on a single meeting transcript, because it turns out that where you point the second model matters more than which model you pick.

Build the Dropdown You Cannot Normally Build

The reason this app is seventy lines instead of a weekend is that every model in the dropdown answered to one key and one endpoint. KodeKey gives you that: a few dozen models from about ten labs behind one OpenAI-compatible URL, included with your KodeKloud subscription, and built for exactly this kind of learning and prototyping rather than production traffic. Generate a key, paste the file from step 5, and have it running before your coffee goes cold.

Then push past the tutorial. Put your own recurring question in the box, ask it on the cheapest model in the list, and switch to the most expensive one in the same thread. The gap between those two answers, on work you actually understand, will teach you more about model selection than any comparison table. The AI Learning Path is where to go when you want to build the judgment behind the choice.

FAQs