How to Run an Open Source LLM Locally With Ollama

Whether a model runs well on your machine is a question you can answer in ten minutes.

Search for how to run a model locally and you will find a table like this one. Eight gigabytes gets you these models. Sixteen gets you those. Expect thirty tokens a second.

Those numbers came from someone else's laptop. Different silicon. A different amount of memory free. A different quantization. Something else open, or not. None of that is in the table.

So this post does something different. Instead of another table, it builds a small benchmark that measures your machine, then explains how to read the result. The numbers below come from a deliberately small machine. Yours will differ, which is the point.

The one constraint that decides everything

Local inference is a memory problem before it is anything else.

The model weights have to sit in memory while the model runs. If they fit, you get a usable speed. If they nearly fit, everything slows as the system starts swapping. If they do not fit, the process is killed.

That is the whole shape of it, and it means you can predict a lot with arithmetic.

Working out what fits

A model's memory need comes from two numbers: how many parameters it has, and how many bits each parameter uses.

At full precision each parameter takes 2 bytes. So a 7 billion parameter model needs about 14GB. That is more than most laptops have spare.

Quantization fixes that. It stores each parameter with fewer bits, trading a little accuracy for a lot of memory. The common choice is 4 bit, usually labelled Q4. That cuts the need to roughly half a byte per parameter.

The overhead column matters. Beyond the weights you need room for the context window, which grows with how much text you send. You also need room for the runtime. A rough rule is weights plus twenty five percent.

So the useful question is not how much RAM you have. It is how much you have free, with your browser and editor open, because that is what the model gets.

Install and confirm

sudo apt-get update && sudo apt-get install -y zstd
curl -fsSL https://ollama.com/install.sh | sh

The zstd line is not optional on a fresh image. The installer extracts a zstd archive and stops with an error that never names the cause.

Now check whether it is already running:

curl http://localhost:11434/

Ollama is running

Ollama is running means the installer registered a systemd service and started it. That is what happens on a normal virtual machine. A connection refused means there is no systemd, which is common in containers, so start it yourself with ollama serve &.

Running ollama serve when it is already up gives you address already in use. That reads like a failure and is really the port being held by the copy that is already working.

Pull something small to start with:

ollama pull qwen3:0.6b

That is 522MB. It downloads in under a minute on most connections and runs almost anywhere.

Your first response

ollama run qwen3:0.6b "Name three reasons a Linux service fails to start."

You get an answer, probably slower than you expected. That is worth understanding rather than shrugging at, and the next section measures why.

For anything beyond a one off question, the HTTP API is easier to work with.

import json
import urllib.request

def generate(model, prompt, num_predict=60):
    body = json.dumps({
        "model": model,
        "prompt": prompt,
        "stream": False,
        "options": {"num_predict": num_predict, "temperature": 0},
    }).encode()
    request = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(request).read())

result = generate("qwen3:0.6b", "Name three reasons a Linux service fails to start.")
print(result["response"])

No API key, no account, and nothing left the machine.

Build the benchmark

Here is the part that replaces the table you would otherwise be copying.

Every response from Ollama carries timing fields alongside the text. They are in nanoseconds. They also separate the two things a stopwatch runs together.

Generation speed is eval_count divided by eval_duration. That is the number worth comparing, because it holds true whatever your prompt length.

import json
import subprocess
import time
import urllib.request

MODELS = ["qwen3:0.6b", "llama3.2:1b", "qwen3:4b"]
PROMPT = "Name three reasons a Linux service fails to start."


def generate(model, prompt, num_predict=60):
    body = json.dumps({
        "model": model,
        "prompt": prompt,
        "stream": False,
        "options": {"num_predict": num_predict, "temperature": 0},
    }).encode()
    request = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    started = time.time()
    payload = json.loads(urllib.request.urlopen(request).read())
    return payload, time.time() - started


def resident_size(model):
    listing = subprocess.run(["ollama", "ps"], capture_output=True, text=True).stdout
    family = model.split(":")[0]
    for line in listing.splitlines():
        if line.startswith(family):
            parts = line.split()
            return f"{parts[2]} {parts[3]}"
    return "not loaded"


print(f"{'model':14s}{'resident':>10s}{'load s':>8s}{'tok/s':>8s}{'total':>9s}")

for model in MODELS:
    try:
        payload, elapsed = generate(model, PROMPT)
        speed = payload["eval_count"] / (payload["eval_duration"] / 1e9)
        print(f"  {model:12s}{resident_size(model):>10s}"
              f"{payload['load_duration'] / 1e9:8.1f}{speed:8.1f}{elapsed:8.1f}s")
    except Exception:
        print(f"  {model:12s}{'failed':>10s}   out of memory, process killed")
    time.sleep(2)

Run it and you get your own numbers rather than mine.

python3 bench.py

model           resident  load s   tok/s    total
  qwen3:0.6b      1.0 GB     4.4    15.2     8.6s
  llama3.2:1b     2.0 GB    10.7    17.0s
  qwen3:4b        failed   out of memory, process killed

Reading your results

That output came from a machine with one CPU and four gigabytes of memory. Roughly what a small cloud instance or a modest playground gives you. Three things in it apply to your results too.

The 4B model failed rather than slowed. It downloaded without complaint, then the process was killed when it tried to load. So the boundary is a cliff and not a slope, and you cannot feel it coming.

The load time dwarfs the generation time. Twenty two seconds to load a model that then produced sixty tokens in five. You pay that once per model, and Ollama keeps it resident for about five minutes, so a second question is far quicker. Timing a single request tells you almost nothing.

The smaller model was faster per token. Fourteen tokens a second against ten. That is the trade you are making, since the larger model gives better answers when it fits.

Practise on a machine you can throw away

Trying a model that turns out to be too big is much less annoying on something disposable. The KodeKloud playgrounds give you a Linux box, so you can install Ollama, pull a few models, find your cliff, and close the tab.

The diagnostic to reach for

When something is slow, ollama ps tells you more than any log line.

ollama ps

NAME        ID              SIZE      PROCESSOR          CONTEXT    UNTIL
qwen3:4b    359d7dd4bcda    3.6 GB    29%/71% CPU/GPU    4096       4 minutes from now

Two columns matter. SIZE is what the model actually occupies, usually more than the download because it includes the context window. PROCESSOR tells you where it runs, so a split like 29%/71% CPU/GPU means most of the work is on the GPU and the rest is not.

If you have a GPU and see 100% CPU, the model did not fit in video memory and fell back. That one line explains most "why is this so slow" questions.

Choosing a model

Once you can measure, choosing gets simpler. Start smaller than you think, then move up until it stops being comfortable.

Those are starting points rather than recommendations. The reason to run the benchmark is that your machine will disagree with the table in ways that matter.

Two things beyond size are worth knowing when you pick.

Licences differ. Qwen3 and Mistral are Apache 2.0, Phi is MIT, and both let you do essentially anything. Llama and Gemma have their own licences with conditions worth reading if this is going anywhere near a product.

Some models think before answering. Reasoning models produce internal tokens you never see, so they take longer than the visible answer suggests. Fine for hard questions, wasteful for simple ones.

A small project worth keeping

Something local can do that a hosted model cannot, at least not without sending your code somewhere.

This reads your staged git diff and writes a commit message. It all runs on your machine, so your unreleased code stays there.

import json
import subprocess
import sys
import urllib.request

MODEL = "llama3.2:1b"

PROMPT = """Write a git commit message for this diff.

Rules:
- One short summary line, under 60 characters, in the imperative mood.
- Then a blank line, then two or three bullet points explaining what changed.
- Describe what changed and why, not how the code works.
- Output only the commit message, no preamble.

Diff:
{diff}
"""


def clean(message):
    """Strip the wrapping small models add.

    Asking for the message alone gets you most of the way. Small models still
    wrap the result in quotes or a code fence often enough to be annoying, and
    those characters end up in your commit if you do not remove them.
    """
    fence = "`" * 3
    message = message.strip()
    if message.startswith(fence):
        message = message.split("\n", 1)[-1].rsplit(fence, 1)[0]
    message = message.strip()
    if len(message) > 1 and message[0] == message[-1] and message[0] in "\"'":
        message = message[1:-1]
    return message.strip()


def staged_diff():
    result = subprocess.run(
        ["git", "diff", "--staged"], capture_output=True, text=True
    )
    if result.returncode != 0:
        sys.exit("Not a git repository, or git is unavailable.")
    return result.stdout


def write_message(diff, model=MODEL):
    body = json.dumps({
        "model": model,
        "prompt": PROMPT.format(diff=diff),
        "stream": False,
        "options": {"temperature": 0.2},
    }).encode()
    request = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    message = json.loads(urllib.request.urlopen(request).read())["response"].strip()
    return clean(message)


if __name__ == "__main__":
    diff = staged_diff()
    if not diff.strip():
        sys.exit("Nothing staged. Run git add first.")
    if len(diff) > 12000:
        diff = diff[:12000] + "\n\n[diff truncated]"
    print(write_message(diff))

git add app.py && python3 commit.py

Remove disk usage from app.py

git commit -m "$(python3 commit.py)"
[main 8f2a1c9] Fix disk usage calculation

Three decisions in there are worth naming, because each one came from the thing failing first.

The diff is cut at 12,000 characters. A large diff overflows a small model's context window, and what happens then is worse than an error. The model sees only part of the change and describes it confidently.

Temperature sits at 0.2 rather than 0. Commit messages benefit from a little variation, and at 0 you get the same phrasing for similar diffs.

And the prompt says what not to do. Without "output only the commit message", small models add a friendly preamble you then delete every time.

Even with that instruction, the first run here came back as "Add disk usage calculation", quotes included, which would have gone straight into the commit. That is why clean exists. Small models follow instructions most of the time, and a tool you use daily needs the other times handled.

Pipe it straight into git once you trust it:

git commit -m "$(python3 commit.py)"

When local is the right answer

Local is not simply cheaper hosted. It is a different set of trade offs, and it is worth being honest about which way they run.

What decides it is usually volume and sensitivity rather than cost. High volume mechanical work on data you would rather not send anywhere is where local wins. A handful of hard questions a day is where a hosted model wins, because your waiting time is worth more than the pennies saved.

Plenty of people run both, which the code above makes easy since Ollama speaks the same API shape as most hosted providers.

Try this on your own machine

An hour, and you end up knowing something specific rather than something general.

Run the benchmark with three models. One clearly small, one you expect to fit, one you expect to be too large. Finding your cliff on purpose beats finding it during real work.

Then run each model twice and compare. The gap between the first and second run is your load cost. Knowing it stops you misreading a cold start as a slow model.

Now point the commit message tool at a real repository. Try your small model and your larger one, and see whether you can tell the difference on a task this mechanical. Most people cannot, which is the useful discovery.

Finally, pull something slightly too big and watch what happens. Either it is killed or it crawls, and either way you will recognise it instantly next time.

What you should be able to answer now

Why is a table of local model speeds not much use? Because it was measured on different hardware, with different memory free and a different quantization. Your machine will disagree, and only your machine matters.

Why does a model download fine and then fail? Because ollama pull checks disk and running needs memory. Nothing warns you at pull time.

What does the first request cost that later ones do not? The model load, which was twenty two seconds in the measurement here against five seconds of actual generation.

Where do you look first when generation is slow? ollama ps, for the resident size and whether the model ended up on CPU when you expected GPU.

The benchmark here is about thirty lines and replaces every table you would otherwise copy. That is the shape of getting good at local models. Stop reading other people's numbers and spend an hour making your own.

Ready to Build on Top of This?

Running a model locally is the foundation for the rest. The AI Agents Fundamentals course covers tokens, embeddings, retrieval, orchestration and MCP through an end to end project. The AI Learning Path sequences that alongside vector databases and agents. Start by finding your own cliff.

FAQs