How to Generate and Compare Text Embeddings With Ollama

Text embeddings turn meaning into distance. Run one locally and the whole idea fits in about forty lines.

Try this in any search box. Type "the service will not start" and see whether it finds the page called "crash loop troubleshooting".

It will not, because those two phrases share no words at all. A keyword search compares letters, and by that measure they have nothing in common. Anybody reading them knows they are about the same thing.

Text embeddings close that gap. They turn text into numbers in a way that puts similar meanings near each other, and once you can measure that distance you have semantic search. This tutorial runs a model locally with Ollama, generates your first embedding, compares two of them, then builds a small search over your own files. No API key, no vector database, and nothing leaves your machine.

Why keyword search runs out

Keyword search matches letters. That works when people type the same words your documents use, and it fails the moment they do not.

Think about who writes each side. Your docs were written by whoever built the thing, in the words that team uses. Your search box is used by someone who does not know those words yet. That is why they are searching.

So the mismatch is structural rather than occasional.

Every row is a real question about a real document. Every one comes back empty. The reader decides the answer does not exist and opens a ticket.

The usual patches help a little. None of them fix it. Synonym lists need a human to think of every synonym up front. Fuzzy matching catches typos, not different words. Tags help until somebody forgets one.

What you actually want is search that matches meaning rather than spelling. That is what an embedding gives you.

What text embeddings are

A text embedding is a list of numbers that captures meaning. Send text to a model, get back a list of maybe 1,024 numbers. Send similar text and the numbers come back similar.

That is the whole idea. Meaning becomes position, and position can be measured.

Where the numbers come from

An embedding model is a neural network trained on a huge amount of text. It has one job. Put things that mean similar things near each other.

Picture a map. Every piece of text you embed gets a pin. Sentences about failing containers land in one area. Sentences about billing land somewhere else. Sentences about both sit in between. Nothing labelled those areas. They formed because the training data used those words in similar ways.

The list of numbers is the coordinates of that pin. With 1,024 numbers you get 1,024 coordinates, so the map has 1,024 directions rather than two. That sounds strange. It is the same idea with more room. Extra directions let the model separate things that would collide on a flat map, like a database lock and a door lock.

What the dimension count means

Your model decides how many numbers you get. You do not.

More dimensions give the model more room for fine distinctions. They also cost more memory and more time to compare. mxbai-embed-large returns 1,024. nomic-embed-text returns 768. Neither is better in the abstract, and you cannot compare across them anyway.

One thing surprises people. No single number means anything on its own. There is no dimension for "technical" or "urgent" to read off. The meaning lives in the whole pattern. That is why the only useful operation is comparing one list against another.

Choosing the right environment

You need two things: somewhere to run Python, and an embedding model for the code to call.

One Linux playground covers both. Ollama installs on it and serves the model on a local port, and Python is one apt command away. So the whole tutorial runs on one machine, with nothing to sign up for. Your own laptop works just as well.

Why run the model locally? Embeddings sit on a different endpoint from chat. They use a different family of models too. So an AI gateway built around chat often serves no embedding models at all. Running your own removes that question, and it costs nothing per request.

Install Ollama:

sudo apt-get update && sudo apt-get install -y zstd
curl -fsSL https://ollama.com/install.sh | sh

That first line is not optional on a fresh image. The installer extracts a zstd archive. Without the tool it stops with an error that never names the cause.

Now check whether it is already running:

curl http://localhost:11434/

Ollama is running

Ollama is running means the installer registered a systemd service and started it for you, which is what happens on a normal virtual machine. You are done, so skip the next line.

A connection refused means nothing is listening, which is what happens inside a container where there is no systemd for the installer to use. Start it yourself:

ollama serve &

Checking first is worth the extra command. Running ollama serve when the service is already up gives you address already in use, which reads like a failure and is really just the port being taken by the copy that is already working.

Pull the model this tutorial uses:

ollama pull mxbai-embed-large

That is 669MB and takes a minute or so. Confirm it landed:

ollama list

NAME                        ID              SIZE      MODIFIED
mxbai-embed-large:latest    468836162de7    669 MB    4 hours ago

Now check the endpoint answers:

curl http://localhost:11434/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "mxbai-embed-large", "input": "hello"}'

{"object":"list","data":[{"object":"embedding","embedding":[0.038474035,
0.005037504,0.003819053,0.018831413,-0.015279996,-0.029728577,
0.01728224,0.04530758,0.025229545, ... ]}]}

Ollama serves an OpenAI compatible route at /v1/embeddings, so the Python below uses the ordinary openai package.

Now the Python side. A minimal Ubuntu image ships without it, so install it first:

sudo apt-get install -y python3 python3-venv

Both packages matter. python3 is the interpreter, and python3-venv is what makes python3 -m venv work, because Debian and Ubuntu split that out into its own package. Without it you get an error telling you to install the very thing you were trying to use.

python3 -m venv .venv
source .venv/bin/activate
pip install openai numpy

The virtual environment matters. Python 3.11 and later will not install into the system Python, so a plain pip install on a current image gives you error: externally-managed-environment and installs nothing. Creating the environment first avoids that entirely, and it gives you a pip of your own inside .venv.

import numpy as np
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

MODEL = "mxbai-embed-large"

That api_key is required by the library and ignored by Ollama, so any string does. It is the one odd looking line here, and it exists because the client was written for a service that needs a key.

Step 1: Generate one embedding

Start with a single call, because the shape of what comes back is the thing to understand.

response = client.embeddings.create(model=MODEL, input="The service will not start")

vector = response.data[0].embedding

print(len(vector))
print([round(v, 4) for v in vector[:5]])

python3 -c "
from openai import OpenAI
c = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')
v = c.embeddings.create(model='mxbai-embed-large',
                        input='The service will not start').data[0].embedding
print(len(v))
print([round(x, 4) for x in v[:5]])
"

1024
[-0.0475, -0.0082, -0.0205, 0.0341, 0.045]

You get 1,024 floats. The first few look like meaningless noise.

They are meaningless on their own. No single number in that list means anything you could name. The meaning is in the whole list, and it only shows up when you compare one list against another.

That is worth sitting with. People expect this part to be more magical than it is. There is no topic dimension to read. There is just a position, and positions can be near or far.

Step 2: Compare two embeddings

Comparison is one formula. Cosine similarity measures the angle between two vectors and ignores their length.

def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

That returns a number between minus one and one. Nearer one means more alike, nearer zero means unrelated.

Now try it on the pair from the opening.

pairs = [
    ("The service will not start", "Crash loop troubleshooting"),
    ("The service will not start", "Quarterly sales figures for the north region"),
    ("The service will not start", "Container keeps restarting"),
]

for a, b in pairs:
    r = client.embeddings.create(model=MODEL, input=[a, b])
    score = cosine(r.data[0].embedding, r.data[1].embedding)
    print(f"{score:+.3f}  {a}  <->  {b}")

Here is what that prints.

Read those rather than skimming. The first pair scores well above the unrelated one, despite sharing no words. That is the entire capability, in one comparison.

Notice the second detail too. Passing a list to input gets both vectors in one request rather than two. That is faster and one character of difference.

Step 3: A small semantic search

Everything you need is now in place. A search is embeddings plus that formula plus a sort.

DOCS = [
    "Restart a pod that is stuck in CrashLoopBackOff",
    "Rotate the database password and update the secret",
    "Add a new node to the cluster and drain an old one",
    "Certificate expired and TLS handshake now fails",
    "Increase the memory limit on a deployment",
    "Grant a new engineer read access to production logs",
]

doc_vectors = [d.embedding for d in
               client.embeddings.create(model=MODEL, input=DOCS).data]

def search(query, top_k=3):
    q = client.embeddings.create(model=MODEL, input=query).data[0].embedding
    scored = [(cosine(q, v), doc) for v, doc in zip(doc_vectors, DOCS)]
    return sorted(scored, reverse=True)[:top_k]

for score, doc in search("my container keeps dying"):
    print(f"{score:+.3f}  {doc}")

Run it and you get this.

+0.599  Restart a pod that is stuck in CrashLoopBackOff
+0.552  Increase the memory limit on a deployment
+0.550  Add a new node to the cluster and drain an old one

The CrashLoopBackOff line comes first, sharing not one word with the query. All six documents went in a single request, which took about a second on a modest machine.

Three things about that code are worth naming.

Passing the whole list to input embeds everything in one request. Looping instead makes six calls for no benefit.

The query uses the same model as the documents. This is not optional, and it is the most common way beginners break their own search.

And the sort is ordinary Python. There is no clever retrieval step hiding anywhere. It is a similarity score attached to each document, sorted.

From search to RAG

What you built in step 3 is the retrieval half of a pattern you have probably heard called RAG, short for retrieval augmented generation. The name is heavier than the idea.

A language model knows what was in its training data. It does not know your runbooks, your incidents, or anything written last week. Ask about your systems and you get something confident and invented. Producing plausible text is exactly what it does.

RAG fixes that by changing the question. Instead of asking what the model knows, you find the right text yourself and hand it over with the question.

Three steps, and you have already built the hard one.

Retrieve. Embed the question, compare against your documents, take the closest few. That is the search function from step 3, unchanged.

Augment. Paste those documents into the prompt, with an instruction to answer from them.

Generate. Send that to a chat model and read the answer.

def answer(question, top_k=2):
    context = "\n\n".join(doc for _, doc in search(question, top_k))
    prompt = (
        "Answer using only the context below. "
        "If it does not contain the answer, say so.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    reply = chat_client.chat.completions.create(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return reply.choices[0].message.content

Retrieval runs locally through Ollama. Generation needs a chat model, which can be another Ollama pull or any hosted endpoint. That half of the loop is an ordinary chat call.

Two lines in that prompt do the real work. Telling the model to use only the context stops it answering from training data and mixing the two. Telling it to say so when the context falls short turns silent invention into a visible gap. That is the best instruction you can give any assistant near real systems.

That is also why this post spends its time on embeddings and similarity rather than on prompts. Retrieval quality sets the ceiling for everything built on top.

Which local model should you use

Model choice matters more than tutorials admit. These three are the common Ollama picks, and they do not agree.

Run the same six documents and the same query through all three and you get different winners. Only mxbai-embed-large put the CrashLoopBackOff document first. The other two ranked an unrelated cluster document above it, by margins small enough to be noise.

That is not a reason to distrust any of them. It is a reason to test on your own text. A model that tops a public leaderboard can still be wrong for your six documents.

Learn the retrieval layer properly

Embeddings are the first step of retrieval, and chunking and search strategy decide more of the outcome than the model does. The Fundamentals of RAG course covers ingestion, chunking, keyword against semantic search, vector databases, and building a full pipeline.

What decides your results

Four things, roughly in the order they matter.

What you embed. A whole page averages out into mush. A section is far better. This is chunking, and it shapes your search quality more than the model does.

The model. It sets how many numbers you get and how well meaning is captured, as the comparison above shows.

The formula. Cosine for text, nearly always.

Whether both sides used the same model. Not a quality question. A correctness one.

Where embeddings beat keywords, and where they do not

Both approaches fail, and usefully they fail in opposite directions.

Keyword search misses anything phrased differently. Ask about a service that will not start, and it never finds your crash loop page.

Semantic search misses exact strings. Search an error code, a product SKU, or a function name, and you get things broadly about the topic. The one document holding that exact token is missed.

That table is why real systems run both and merge the results. You do not have to choose. Their failures do not overlap, which is what makes combining them work.

Where else embeddings earn their keep

RAG is the use everyone names first. It is not the only one. Some of the others are easier to build, because they need no language model at all.

The pattern underneath is always the same. Turn text into a position, then ask a question about distance.

Group things nobody labelled

You have thousands of support tickets and no categories. Embed them, cluster the vectors, and the groups fall out on their own. Nobody writes the taxonomy up front. That is the point, because a taxonomy written up front always misses the category you most needed.

The same trick finds duplicates. Two tickets about the same outage, phrased completely differently, sit close together. A distance threshold catches pairs a keyword match never would.

Route a request without writing rules

Say you want support questions to reach the right team. The rules approach is a pile of keyword matches. It grows forever and still misses things.

Instead, embed one short description per team, embed the incoming question, and send it to the closest one. Adding a team is adding a sentence.

def embed(text):
    return client.embeddings.create(model=MODEL, input=text).data[0].embedding

ROUTES = {
    "platform": "Kubernetes clusters, deployments, pods, scaling and nodes",
    "data": "Databases, backups, migrations, queries and replication",
    "security": "Access, permissions, certificates, secrets and audit logs",
}

route_vectors = {name: embed(desc) for name, desc in ROUTES.items()}

def route(question):
    q = embed(question)
    return max(route_vectors, key=lambda name: cosine(q, route_vectors[name]))

That is classification with no training, no labelled data, and no model call beyond the embedding. It will not beat a properly trained classifier. It also takes ten minutes rather than a sprint.

Answer a question you have already answered

A semantic cache stores past questions with their answers. A new question gets embedded and compared against the cache. A close enough match returns the stored answer, with no model call at all.

Ordinary caching cannot do this. It matches strings, and users never phrase things the same way twice. "How do I restart a pod" and "what is the command to restart pods" are different strings and the same question.

The catch is the threshold. Set it loose and you serve confident wrong answers to questions that were only a bit similar. Start strict, measure how often it hits, then loosen slowly.

Notice when something does not belong

Embed what normally arrives, take the average position, then flag anything landing far away. On log lines that finds unusual messages. On user queries it finds questions your system was never built for.

Here the threshold problem from earlier becomes useful rather than annoying. A search always returns its nearest neighbour, relevant or not. Measure that distance, and the low scores are themselves the signal.

Five of those seven need no language model at all. That is worth noticing. Embeddings get filed under generative AI, and most of what they do well is older and quieter. It is search, grouping and comparison. Ordinary engineering problems, made much easier once meaning has a position.

When you need a vector database

Not yet. That is the honest answer for the code above.

A list of six vectors compared in a loop is fine. So is six thousand. You are doing arithmetic over an array, and Python is fast enough that you will not notice.

You want a real store when the list stops fitting in memory. Or when you need to filter by a date or a version while searching. Or when your vectors change often enough that rebuilding the list gets annoying.

Until then, keep the loop. Understanding what a vector database does is much easier once you have felt the thing it replaces.

Practise the whole pipeline

Once embeddings make sense, the pieces around them are what decide your results. The Vector Database for GenAI course covers foundations, the embedding layer, similarity, vector storage, and database internals through hands on labs.

The complete script

Everything this post built, in one file. Seventy odd lines, and it covers search, routing and duplicate detection.

import numpy as np
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
MODEL = "mxbai-embed-large"

DOCS = [
    "Restart a pod that is stuck in CrashLoopBackOff",
    "Rotate the database password and update the secret",
    "Add a new node to the cluster and drain an old one",
    "Certificate expired and TLS handshake now fails",
    "Increase the memory limit on a deployment",
    "Grant a new engineer read access to production logs",
]

ROUTES = {
    "platform": "Kubernetes clusters, deployments, pods, scaling and nodes",
    "data": "Databases, backups, migrations, queries and replication",
    "security": "Access, permissions, certificates, secrets and audit logs",
}


def embed(texts):
    if isinstance(texts, str):
        texts = [texts]
    return [d.embedding for d in client.embeddings.create(model=MODEL, input=texts).data]


def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))


doc_vectors = embed(DOCS)
route_vectors = {name: embed(desc)[0] for name, desc in ROUTES.items()}


def search(query, top_k=3):
    q = embed(query)[0]
    scored = [(cosine(q, v), doc) for v, doc in zip(doc_vectors, DOCS)]
    return sorted(scored, reverse=True)[:top_k]


def route(question):
    q = embed(question)[0]
    return max(route_vectors, key=lambda name: cosine(q, route_vectors[name]))


def duplicates(items, threshold=0.60):
    vectors = embed(items)
    pairs = []
    for i in range(len(items)):
        for j in range(i + 1, len(items)):
            score = cosine(vectors[i], vectors[j])
            if score >= threshold:
                pairs.append((score, items[i], items[j]))
    return sorted(pairs, reverse=True)


if __name__ == "__main__":
    for score, doc in search("my container keeps dying"):
        print(f"{score:+.3f}  {doc}")

    print()
    print("routed to:", route("the database keeps timing out"))

    print()
    tickets = [
        "Pod stuck in CrashLoopBackOff after the deploy",
        "Container will not stay up following the latest release",
        "Need read access to the staging logs",
    ]
    for score, a, b in duplicates(tickets):
        print(f"{score:+.3f}  {a}  |  {b}")

Save it, run it, and you get this.

python embeddings.py

+0.599  Restart a pod that is stuck in CrashLoopBackOff
+0.552  Increase the memory limit on a deployment
+0.550  Add a new node to the cluster and drain an old one

routed to: data

+0.668  Pod stuck in CrashLoopBackOff after the deploy  |  Container will not stay up following the latest release

Three things about the shape of that file are worth naming.

The documents and route descriptions are embedded once at import, not per query. Embedding is the slow part, so doing it up front means each search is arithmetic over an array rather than a network call.

Notice that embed takes a string or a list and always returns a list. That is a small kindness to yourself, because otherwise every caller has to remember which one it wanted.

And duplicates takes its threshold as an argument rather than hardcoding it. The 0.60 works for these tickets and will be wrong for your text, so it belongs where you can change it.

Try this on your own documents

An afternoon, and it runs on files you already have with nothing to sign up for.

Take one folder of real docs. Split them on headings, not on a character count. A section holds one idea. A page holds several.

Embed them all in one request. Write down which model you used, because you cannot mix models later.

Now write ten questions you would genuinely ask, and note which chunk should answer each one. Run them and count how often the right chunk lands in the top three.

That number is your search quality, and it is more useful than any benchmark you will read, because it is measured on your own text.

Then try three things. Pull a second model and run the same ten questions, since swapping the string is the whole experiment. Embed a whole document as one chunk and compare against the split version. And ask about something your documents do not cover, to see what comes back when there is no right answer.

What you should be able to answer now

What is actually in an embedding? A list of numbers where nothing individually means anything. The meaning is in the whole list, and it only becomes visible in comparison.

Why cosine and not straight line distance? Because cosine ignores length, so a long document and a short sentence about the same thing still count as similar.

What happens if you embed queries with a different model from your documents? Your search quietly stops working, and it looks like poor quality rather than an error.

When your demo looks unconvincing, what do you check first? Your examples. Two sentences about the same subject genuinely are close, and no model will separate them much.

The two phrases at the top of this post share no words and score 0.490 against each other, against 0.293 for something genuinely unrelated. Nothing clever produced those numbers. They are the angle between two lists, and everything else in retrieval is built on top of it.

Ready to Build the Rest of the Pipeline?

Embeddings are step one. What surrounds them decides your results. The Fundamentals of RAG course covers chunking, search strategy, and vector databases end to end. The Vector Database for GenAI course goes deeper on storage and internals. And the AI Learning Path sequences both alongside agents and MCP. Start with ten questions about your own documents.

FAQs