A Beginner's Guide to Vector Databases

A team spent three weeks picking a vector database. They compared four products on how fast each one searched a hundred million vectors. Then they launched with eleven thousand documents. At that size, every option they tested would have answered in under ten milliseconds. So would a plain table with no index at all.

The comparison was careful. It just measured the one thing their problem did not have. This post starts with what a vector database really stores. Then how the search works. Then how to build one in a database you probably already run. Three worked examples follow, at three very different sizes, because size is what actually decides.

What a vector database is

A vector database stores embeddings and finds the closest ones to your query. It answers one question, fast: which of these is most like this?

It gives you a very good answer instead of a perfect one. That trade is deliberate, and we will get to why.

Start with the embedding

An embedding is a list of numbers that captures meaning.

You send text to a model. It hands back a list of maybe 1,536 numbers. Send similar text and you get similar numbers. Send something unrelated and the numbers land far away.

That is the trick. Meaning becomes distance.

Here is what it buys you. Someone searches for "service will not start." Your doc is titled "crash loop troubleshooting." Not one word matches. Keyword search finds nothing. Vector search finds it easily, because both sit close together in number space.

Why the search is approximate

To find the true closest vector, you compare against every single one.

With ten thousand vectors, fine. Milliseconds. With ten million, far too slow for a live request.

So vector indexes cheat on purpose. They arrange the vectors so a search can skip most of them. Now and then they miss a true best match.

We measure that with recall. Recall of 0.95 means you got nineteen of the top twenty. One slightly worse result slipped in.

For search, nobody notices. For anything that must be complete and exact, this is the wrong tool.

The two indexes to know

Both ship with pgvector. Both turn up in other databases under the same names. Learn them once and the knowledge travels.

HNSW builds a layered map. The top layer is sparse, so a search takes big jumps across the space. Each layer below is denser and narrows things down. Think of finding a house: first the city, then the street, then the number.

IVFFlat splits vectors into clusters, then searches only the nearest few. Faster to build, lighter on memory, and it needs your data present before you can build it.

Go with HNSW unless you have measured a reason not to. Two settings matter: m and ef_construction. Start at 16 and 64.

Build one right now

Here is the fastest way to understand any of this. Build it in Postgres.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id          bigserial PRIMARY KEY,
    document_id text NOT NULL,
    section     text,
    version     text,
    content     text NOT NULL,
    embedding   vector(1536)
);

Look at that table. One vector column, four ordinary ones. Those ordinary columns are the whole reason to do this in Postgres. You will see why shortly.

The vector(1536) type locks in the size. Send a vector of the wrong length and Postgres rejects it. That catches a model swap before it quietly ruins your data.

Now the index.

CREATE INDEX chunks_embedding_idx ON chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

The vector_cosine_ops part tells the index which distance measure you plan to use. It must match the operator in your query. If it does not, Postgres quietly ignores the index and scans everything. No error. Just a slow query and a confused afternoon.

And here is the search.

SET hnsw.ef_search = 100;

SELECT document_id, section, content,
       1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE version = '2.4'
ORDER BY embedding <=> $1
LIMIT 5;

The <=> operator is cosine distance. Order by it, take the top five, and that is a similarity search. Subtracting from one flips distance into similarity, which reads better to humans.

Now look at that WHERE clause again. Most tutorials skip it. It matters more than anything else on this page.

Filtering beats tuning

Here is a bug you will meet.

Your support bot keeps answering about the wrong product version. Docs for version 2 and version 3 use nearly identical wording, so they sit right beside each other in number space. Retrieval is doing exactly what you asked. Both passages really are about the question.

No amount of better search fixes this. A WHERE version = '2.4' fixes it instantly.

That is why keeping vectors next to normal columns is so useful. Version, date, team, document type: all just columns.

One catch. Filters and approximate search interact awkwardly. Ask for the top five, filter them, and you might end up with two. Postgres handles this with iterative scanning, which keeps walking the index until it has enough surviving matches.

SET hnsw.iterative_scan = relaxed_order;

That is one line. If you take a single setting from this post into production, take that one.

Learn vector databases properly, end to end

The idea is simple. The details are where projects stall. The Vector Database for GenAI course on KodeKloud covers foundations, embeddings, similarity, vector storage on AWS S3, the wider landscape, and how these databases work inside, all with hands on labs.

Three examples, worked through

Same question, three very different workloads. Watch the numbers, not the labels.

Fifteen thousand documents

Runbooks and postmortems in a repo. You want to ask a plain question and get the right section back.

The numbers. About fifteen thousand chunks. A few queries a minute. Content changes when someone merges a pull request.

Do the maths. Fifteen thousand vectors at 1,536 dimensions is roughly 90MB. That fits in memory without anyone noticing. HNSW answers in single digit milliseconds.

The answer. Postgres. Not as a compromise. A second database here means one more thing to back up, monitor, and keep in sync, and you would get nothing measurable back.

Do this instead. Chunk on headings, not on character counts. Store the file path and heading as columns. Then answers can cite a source, and you can search one folder at a time.

WHERE document_id LIKE 'runbooks/%'

There is your folder filter. One line, because the path is just a column.

Four million tickets

Old support tickets, growing daily. You want to find similar past incidents when a new one lands.

The numbers. Four million vectors. Tens of thousands added weekly. Every query filtered by product area, usually by date too.

Do the maths. Four million vectors at 1,536 dimensions is roughly 24GB. Memory is now a real number you have to think about. This is also where smaller embeddings start to pay for themselves.

The answer. Still Postgres, though it is closer now. The heavy filtering keeps it there. Product area and date are columns, so the planner can combine them with the vector search in one pass.

SET hnsw.iterative_scan = relaxed_order;

SELECT ticket_id, resolved_at, summary
FROM tickets
WHERE product_area = 'ingress'
  AND created_at > now() - interval '18 months'
ORDER BY embedding <=> $1
LIMIT 10;

What would change the answer. Hundreds of thousands of new rows an hour, where index maintenance becomes the bottleneck. Or growth past the point where the index stops fitting in memory. Watch for both. Do not plan for them yet.

Fifty million vectors, two hundred tenants

One search service, many teams, strict isolation between them.

The numbers. Fifty million vectors. Two hundred tenants, a few much larger than the rest. No query may ever cross tenants.

Do the maths. Here instinct and arithmetic disagree. Fifty million vectors is not a fifty million vector problem when every query is scoped to one tenant. Filter to a single tenant and you are searching two hundred thousand rows. That is small.

Measure your biggest tenant, not your total. That one reframe sends more teams back to Postgres than any benchmark.

The answer. It depends on which constraint hurts more, and pretending otherwise would be dishonest. If isolation is the hard requirement, row level security is a strong reason to stay put.

ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON chunks
    USING (tenant_id = current_setting('app.tenant_id')::uuid);

With that in place, a query that forgets its tenant filter returns nothing. It does not return someone else's data. That is the difference between a bug and an incident.

If instead one noisy tenant must never slow another, a dedicated store starts to earn its keep.

When you do need a dedicated store

Four honest reasons.

Real scale. Tens of millions of vectors and climbing, where one Postgres box starts to strain.

Global reach. Low latency search on every continent. A purpose built distributed system does this well.

Very heavy writes. Sustained high insert rates, where index maintenance becomes the bottleneck.

Your team does not run Postgres. And does not want to start. That is a fair reason, not a technical one.

Not reasons: a benchmark, a scale you have not reached, or a diagram that looks better with another box on it.

Moving between these is easier than it sounds. Your chunks, their metadata, and their embeddings are the real assets. The index is derived, so you can rebuild it anywhere in an afternoon.

Build the pipeline around it

The database is one piece. Chunking and search strategy decide more of your result than the store does. The Fundamentals of RAG course on KodeKloud covers architecture, document processing and chunking, keyword against semantic search, vector databases, and building a full pipeline.

Try it on your own documents

Give this an afternoon. Every step runs on a database you already have.

Take one folder of real docs. Split them on headings, not character counts. Embed the chunks and write down which model you used, because that is the choice you cannot casually undo.

Load them into a table with the vector column plus two or three plain columns: file path, heading, date. Query it with no index first. A full scan is exact, so those results become your baseline.

Now write twenty questions you would genuinely ask. Note which chunk should answer each one. Run them, and count how often the right chunk lands in the top five.

That number is your search quality. It is the only measurement here that predicts whether the thing will be useful.

Add the HNSW index and run the same twenty. Compare. You have now measured recall on your own data, which is what every benchmark you have read was only guessing at.

Finally, add a filter to one query. Restrict by date, or by folder. For most real apps, that single filter improves answers more than any index tuning will.

What you should be able to answer now

How many vectors will you really have next year? Most people guess an order of magnitude high. The honest number usually settles the whole question.

Do your queries need filters, and on what? Version, tenant, date, document type. If yes, that matters more than raw speed, and it favours a database that already understands columns.

How good is your search on twenty real questions? If you cannot say, you do not yet know whether your system works. No database choice will tell you.

Which embedding model are you tied to? Changing it means re embedding everything. Worth knowing the size of that job now, while your corpus is still small.

The team at the top of this post did careful work. They compared products thoroughly on the one dimension the industry advertises loudest, and their workload was never going to be limited by it. Twenty real questions against their eleven thousand documents would have told them, in one afternoon, that chunking was the interesting problem and the database was not a decision at all.

Ready to Build With Vectors Properly?

Vector search works when the pieces around it work. The Vector Database for GenAI course on KodeKloud covers foundations, embeddings, similarity, and internals with hands on labs. The Fundamentals of RAG course covers the chunking and search choices that decide your results. And the KodeKloud playgrounds give you somewhere to load your own docs and measure what comes back. Start with twenty questions.

FAQs