Chunking sets the ceiling. Nothing downstream recovers from splitting a document badly.
Highlights
- Chunk size and boundaries decide more of your retrieval quality than the embedding model does, and a bad split cannot be rescued by a better model.
- A chunk holding two topics produces an embedding that is partly about each, so it matches both queries weakly and neither one well.
- Measured on the same corpus, the gap between the best and worst strategy was 60 percentage points on whether the top result could actually answer.
- Recall alone is a misleading metric, because sending the entire document scores 100 percent and is obviously the wrong answer.
- Chunks that are too small find the right area and cannot answer from it, which is how a strategy wins on recall and loses on usefulness.
- Adding the section heading to each chunk before embedding raised the top result from 80 to 90 percent for one line of code.
- Overlap is worth less than it is usually credited with, and cost you real money, so measure before adding it.
A RAG pipeline that returns the wrong document cannot be fixed by a better model. The model only sees what retrieval handed it. If the right text never arrives, no amount of prompting brings it back.
Retrieval quality is mostly decided before any model runs, by how you split your documents. This post measures that. Same corpus, same questions, same embedding model, and eight ways of chunking. So the differences come from the split alone.
The results are more interesting than the received wisdom, and one of them is a trap.
Why chunking decides so much
An embedding is one vector for a whole chunk. That one fact explains most of what follows.
A chunk about one topic gets a vector pointing somewhere specific, and a question on that topic lands near it. A chunk covering three topics gets a vector sitting between all three. Now it is weakly similar to questions about each, and strongly similar to none.
You can watch that happen. Take one focused chunk about exit code 137, then paste three unrelated sentences after it. The fact has not moved and the words are all still there.
The question it should answer dropped from 0.812 to 0.718. The question about draining nodes, which the mixed chunk now genuinely mentions, rose by only 0.040.
So the chunk lost more on the thing it is good at than it gained on the things it merely touches. That is the whole mechanism, and it is why a big chunk does not simply waste tokens. It blurs the very thing you match against.
Where a chunk sits in the pipeline
Worth being precise about this, because chunking happens once and affects everything afterwards.
Your document is split into chunks. Each chunk is embedded, producing one vector, and that vector is stored beside the chunk text. When a question arrives it is embedded too, compared against every stored vector, and the closest chunks are returned.
So a chunk is two things at once. It is the unit you search over, meaning its vector decides whether it gets found. And it is the unit you send, meaning its text is what the model has to answer from.
Those two jobs want different sizes. Searching wants small and specific. Answering wants large and complete. Every chunking decision is a negotiation between them, which is why there is no setting that is simply correct.
Small chunks fail the other way. A single sentence gets a sharp, specific vector, and it often lacks the context needed to answer. Retrieval finds the right area, and what comes back is not enough.
So two failure modes pull opposite ways. That is why chunk size has a middle rather than a maximum.
Set up the corpus and questions
You need a document and a set of questions before any of this means anything. Use your own if you have them, and if not, these are the ones every number in this post came from.
Save this as doc.txt:
Six short sections, each about one thing that goes wrong in a cluster. Small enough to read in full, and varied enough that retrieval has to choose rather than always returning the only option.
Then the questions. Each one carries the section that should answer it, and one fact that must be present for an answer to be possible.
QUESTIONS = [
{"q": "why did my pod get killed with 137",
"section": "Memory limits and exit code 137",
"fact": "128 plus signal 9"},
{"q": "the site works in chrome but fails from a script",
"section": "Certificate expiry on the edge",
"fact": "cache intermediates"},
{"q": "what order do I rotate credentials in",
"section": "Rotating the database password",
"fact": "secret store first"},
{"q": "my container keeps dying and restarting",
"section": "Restarting a stuck pod",
"fact": "CrashLoopBackOff"},
{"q": "how do I take a node out of service safely",
"section": "Draining a node for maintenance",
"fact": "Cordon the node first"},
{"q": "my build failed but the error looks wrong",
"section": "Reading a failed pipeline",
"fact": "earliest step"},
{"q": "how long does kubernetes wait between restarts",
"section": "Restarting a stuck pod",
"fact": "five minutes"},
{"q": "a sidecar was killed not my app",
"section": "Memory limits and exit code 137",
"fact": "per container"},
{"q": "the drain command hangs forever",
"section": "Draining a node for maintenance",
"fact": "disruption budget"},
{"q": "do pods pick up a new secret automatically",
"section": "Rotating the database password",
"fact": "does not watch the secret"},
]Notice two of those questions point at the same section from different angles. That matters, because a strategy that happens to suit one phrasing should not score well unless it suits both.
The two things worth measuring
Recall at 3 asks whether the answer shows up anywhere in the top three chunks. It tells you if retrieval found the right region.
Answerable at 1 asks whether the single best chunk holds the fact. It tells you whether the model gets what it needs.
Both matter, and they can disagree. That is where the interesting findings sit.
def score(chunks, vectors, questions):
recall = answerable = 0
for item in questions:
q = embed(item["q"])[0]
ranked = sorted(((cosine(q, v), c) for v, c in zip(vectors, chunks)),
reverse=True)
if any(item["fact"].lower() in c.lower() for _, c in ranked[:3]):
recall += 1
if item["fact"].lower() in ranked[0][1].lower():
answerable += 1
return recall / len(questions), answerable / len(questions)The results
Eight strategies, same corpus, same questions, same model.
Read the top row first, because it is the trap.
The metric that misleads
Not chunking at all scores 100 percent on both measures. The whole document is one chunk. It holds every answer. It is always returned.
By the numbers it is the best strategy on the table. It is also obviously wrong. Seeing why is the most useful thing here.
Look at the last column. That strategy sends 2,356 characters per question, against 800 for the best real option. You pay to send five irrelevant sections every time. The model has to find the answer inside them. And the cost grows with every document you add.
So recall alone rewards doing nothing. Add what it costs to serve and the ranking changes completely.
Watch out
Any retrieval metric that ignores how much you send will tell you to chunk less, and following it far enough leads to sending everything. This is why teams sometimes report improving recall while their system gets worse. Larger chunks catch more answers by covering more ground, so the number climbs, and each answer arrives buried in four unrelated topics. Track what you send alongside what you find, and the trade becomes visible rather than hidden.
Chunks that are too small
Now the bottom half of the table.
One sentence per chunk finds the right region 70 percent of the time. It produces an answerable top result only 40 percent of the time. So half the successful retrievals return something too thin to use.
That gap is the clearest result here, and it explains something that confuses people in published benchmarks. Semantic chunking has scored above 90 percent on retrieval recall in one study, and 54 percent on end to end answer accuracy in another. Both numbers can be right. Small, tightly focused chunks match questions well, then hand over a fragment.
So a strategy can win the retrieval benchmark and lose the real task. When you read a chunking comparison, check which one it measured.
Where the middle sits
Fixed chunks at 400 characters scored 100 percent recall and 90 percent answerable. That matches the best result on the table, while sending a third of what the whole document costs.
Push to 800 and recall stays at 100 percent while answerable drops to 80. That looks backwards until you see why. With only three chunks, the top result often covers a neighbouring topic, and the fact sits in the second.
Go down to 200 and both fall, because the split lands mid explanation.
The usual published advice is 400 to 512 tokens, roughly 1,600 to 2,000 characters. That is larger than what won here, and both can be true. Those corpora are dense technical documents where a topic needs room. These runbook sections are short and self contained.
Which is the point. The number that works is a property of your documents, not a constant you can copy.
Learn the retrieval layer properly
Chunking is one decision inside a pipeline with several, and they interact. The Fundamentals of RAG course covers ingestion, chunking, keyword against semantic search, vector databases, and building a complete pipeline.
The line of code that paid for itself
Two rows differ by one string join.
Chunking by heading scored 90 percent recall and 80 percent answerable. Add the section title to the text before embedding, and the same chunks scored 100 and 90.
vectors = embed([f"{c['heading']}. {c['text']}" for c in chunks])A heading is usually the most topic dense sentence in a section. Including it moves the chunk vector towards what the section is about. It is already there, and it costs nothing to add.
This generalises. Anything you know that helps place a chunk can go into the text you embed. Document title, section, product version. Embed the context, not just the content.
Quick tip
Try prepending metadata before you try a different embedding model. A model swap means re embedding everything and comparing across a moving baseline. Prepending a heading is one line, costs a re index of the same corpus, and gave a bigger improvement here than most model changes would. Cheap experiments first.
What about overlap
Overlap gets recommended everywhere, usually at 10 to 20 percent, to stop an answer being cut in half at a split.
On this corpus it hurt. Fixed 400 with 20 percent overlap scored 90 and 80, against 100 and 90 without it, while producing eight chunks instead of six.
Why would it hurt rather than merely fail to help? Overlap creates chunks that share text, so several vectors point at nearly the same place. A question that should match one section now matches two half copies of it, and the specific one can lose to a duplicate.
That is not an argument against overlap. It is an argument for measuring it, because it is not free. Overlap raises your chunk count. You store more vectors, compare more per query, and pay more to embed. On a corpus where splits rarely cut through answers, you buy nothing.
Overlap earns its cost when text runs continuously, with no headings, where every split point is arbitrary. It earns much less when your documents have structure to split on.
The honest default
If you want one answer to start from, this is it.
Split on structure where you have it. Headings, sections, pages. A break the author put in is usually a real topic break.
Fall back to fixed size where you do not. Around 400 to 800 characters is a fair place to begin. Treat it as a starting point, not a setting.
Prepend the heading or title before embedding. One line, measurable gain.
Add overlap only after measuring without it. It costs storage and query time. On structured documents it often buys nothing.
Measure on your own documents. Everything above came from one corpus of runbook sections. Yours will separate somewhere else.
The complete benchmark
Everything above in one file. It reads doc.txt, runs every strategy, and prints the table.
import re
import numpy as np
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
MODEL = "mxbai-embed-large"
TEXT = open("doc.txt").read()
def embed(texts):
if isinstance(texts, str):
texts = [texts]
return [d.embedding for d in client.embeddings.create(model=MODEL, input=texts).data]
def cosine(a, b):
a, b = np.array(a), np.array(b)
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def sections(text):
out, heading, buffer = [], None, []
for line in text.splitlines():
stripped = line.strip()
is_heading = (stripped and len(stripped) < 60
and len(stripped.split()) <= 8
and not stripped.endswith((".", ":", ",")))
if is_heading:
if heading and buffer:
out.append((heading, " ".join(buffer)))
heading, buffer = stripped, []
elif stripped:
buffer.append(stripped)
if heading and buffer:
out.append((heading, " ".join(buffer)))
return out
def fixed_size(text, size, overlap=0):
joined = " ".join(f"{h}. {b}" for h, b in sections(text))
out, i = [], 0
while i < len(joined):
out.append(joined[i:i + size])
i += size - overlap
return out
def by_sentences(text, per_chunk):
out = []
for _, body in sections(text):
parts = re.split(r"(?<=[.!?])\s+", body)
for i in range(0, len(parts), per_chunk):
group = " ".join(parts[i:i + per_chunk])
if group.strip():
out.append(group)
return out
STRATEGIES = {
"whole document": lambda text: [text],
"fixed 200 chars": lambda text: fixed_size(text, 200),
"fixed 400 chars": lambda text: fixed_size(text, 400),
"fixed 400, 20% overlap": lambda text: fixed_size(text, 400, 80),
"fixed 800 chars": lambda text: fixed_size(text, 800),
"one sentence each": lambda text: by_sentences(text, 1),
"three sentences": lambda text: by_sentences(text, 3),
"by heading": lambda text: [b for _, b in sections(text)],
"by heading, title added": lambda text: [f"{h}. {b}" for h, b in sections(text)],
}
print(f"{'strategy':26s}{'chunks':>7s}{'avg':>6s}{'recall@3':>10s}"
f"{'answerable@1':>14s}{'chars sent':>12s}")
for name, split in STRATEGIES.items():
chunks = split(TEXT)
vectors = embed(chunks)
recall = answerable = 0
for item in QUESTIONS:
q = embed(item["q"])[0]
ranked = sorted(((cosine(q, v), c) for v, c in zip(vectors, chunks)),
reverse=True)
if any(item["fact"].lower() in c.lower() for _, c in ranked[:3]):
recall += 1
if item["fact"].lower() in ranked[0][1].lower():
answerable += 1
first = embed(QUESTIONS[0]["q"])[0]
top_two = sorted(((cosine(first, v), c) for v, c in zip(vectors, chunks)),
reverse=True)[:2]
sent = sum(len(c) for _, c in top_two)
avg = sum(len(c) for c in chunks) // len(chunks)
print(f" {name:24s}{len(chunks):7d}{avg:6d}"
f"{recall / len(QUESTIONS):10.0%}{answerable / len(QUESTIONS):14.0%}{sent:12d}")Three details in there are worth naming.
fixed_size joins the sections back together before splitting, on purpose. Fixed size chunking ignores structure by definition, so giving it a document with the headings already stripped out would be measuring something else.
The chars sent column uses the first question rather than an average. It is an illustration of cost rather than a precise figure, and one example keeps the table readable.
And every strategy is scored against the same questions and the same model. That is the only reason the numbers can be compared at all, and it is the thing to preserve when you swap in your own documents.
In the wild
The most common way teams tune chunking is by reading a blog post and copying its numbers, including this one. That fails quietly, because the right number depends on how dense your documents are and how your users phrase questions. A corpus of short runbook sections wants smaller chunks than a corpus of dense reference material. The fix is not a better source of numbers. It is ten of your own questions and an afternoon, after which you can measure any change instead of debating it.
Try this on your own documents
An afternoon, and every step runs on documents you already have.
Write ten questions you would really ask your corpus. For each, note the section that should answer it. Then note one fact that must appear for an answer to be possible. That fact field is what separates finding from answering.
Then run three strategies rather than one. Your current split, fixed size at two sizes, and structural if your documents have headings. Print chunk counts and average sizes next to the scores. A strategy that wins by making forty chunks per document has a cost you should see.
Add a column for how much text you send per question. That is the column that stops the whole document winning.
Finally, take your best strategy and add the section title before embedding. If it helps you as much as it helped here, that is your cheapest win.
What you should be able to answer now
Why can a better embedding model not fix bad chunking? The model never sees text that retrieval failed to return. And a chunk covering three topics has a blurred vector that no model can sharpen.
Why is recall on its own a misleading metric? Because not chunking at all scores 100 percent. Any measurement that ignores what you send rewards sending everything.
How does a strategy win on recall and lose in practice? By making chunks too small to answer from. The right region is found, and the fragment returned is not enough. That is why one benchmark shows above 90 percent recall while another shows 54 percent answer accuracy.
What is the cheapest improvement available to you? Prepending the heading to each chunk before embedding. One line, and it beat every other single change measured here.
The whole document scored 100 percent on both metrics and would make a terrible system. Every useful decision here came from asking what a number left out. That habit is worth more than any particular chunk size.
Ready to Build the Rest of the Pipeline?
Chunking matters most, and it is not the only decision. The Fundamentals of RAG course covers ingestion, search strategy, and vector databases end to end. The Vector Database for GenAI course goes deeper on storage once your chunks outgrow a Python list. And the KodeKloud playgrounds give you a machine where Ollama and your own documents can sit together. Start with ten questions.
FAQs
Q1: What is the best chunk size for RAG?
There is no single answer. It depends on how dense your documents are. Common advice lands on 400 to 512 tokens, or roughly 1,600 to 2,000 characters, with 10 to 20 percent overlap. On the corpus in this post, which is short runbook sections, 400 characters scored best and 800 was already too big. Both results are right for their material. Dense reference documents need room to hold a complete idea. Short self contained sections do not. So treat any published number as a starting point, not a setting. What matters more is having a way to check. Ten questions with known answers lets you measure a size change in minutes. That turns an argument into a measurement, which is worth more than any default.
Q2: Why does my retrieval find the right document but give bad answers?
Almost always because the chunks are too small. Retrieval and answering are different jobs. Small chunks are good at one of them. A single sentence gets a sharp, specific vector, so it matches a question well. Then it hands the model a fragment with no surrounding context. Measured here, one sentence per chunk found the right region 70 percent of the time. It gave a usable top result only 40 percent of the time. This also explains a contradiction in published benchmarks. Semantic chunking has scored above 90 percent on retrieval recall in one study, and 54 percent on end to end accuracy in another. Both can be true. The fix is to make chunks large enough to stand alone, or to retrieve the small chunk and send the section around it.
Q3: Do I need overlap between chunks?
Less often than the advice suggests, and it is not free. Overlap stops an answer being cut in half at a split. That is a real problem in flowing prose with no natural break points. On the corpus here, 20 percent overlap matched the plain version on the top result, scored slightly worse at three, and made eight chunks instead of six. Those extra chunks cost storage, embedding time, and comparison time on every query. So measure without it first. If your documents have headings, you are already splitting where the author chose to, and overlap buys little. If your text runs continuously and every split is arbitrary, overlap does real work and earns its cost.
Q4: What do I need to measure my own chunking?
Ten questions with known answers, and about forty lines of code. For each one, note the section that should answer it. Then note one fact that must appear for an answer to be possible. That second field is what most benchmarks miss. It is also what lets you tell finding the right area apart from returning enough to answer. Then measure three things per strategy. Whether the answer appears in the top three results. Whether the single best result holds the fact. And how many characters you send per question. That third column is essential, because without it the metric rewards not chunking at all. You need no extra tooling. If embeddings work, you have everything. This post continues from the RAG over PDF documents tutorial, where that pipeline gets built.
Q5: Should I use semantic chunking instead of fixed size?
Measure before you switch, because the evidence is mixed. Semantic chunking uses embedding similarity to find topic boundaries, rather than splitting at character counts. That sounds better, and sometimes is. It has scored above 90 percent retrieval recall in one benchmark. In another it scored 54 percent on end to end answer accuracy, below plain recursive splitting at 69 percent. The likely reason is chunk size. Semantic boundaries can produce fragments too small to answer from. It also costs more, because finding boundaries means embedding the document twice. Recursive or structural splitting is the sensible default. Semantic chunking is worth trying when your documents genuinely mix topics without headings. Either way, decide from your own questions rather than someone else's benchmark.
Q6: What is the cheapest improvement to retrieval quality?
Prepend the section heading to each chunk before you embed it. Measured here, chunking by heading scored 90 percent recall and 80 percent answerable. The same chunks with the title prepended scored 100 and 90. That is one string join. It works because a heading is usually the most topic dense line in a section. Including it moves the chunk vector towards what that section is about. The idea goes further than headings. Anything you know that helps place a chunk can go into the embedded text. Document title, product version, page number, owning team. Try this before changing embedding models. A model swap means re embedding everything and comparing against a moving baseline. This one costs a re index of the same corpus, and shows up straight away.
Discussion