When a RAG system underperforms, teams reach for a bigger model or a fancier reranker. More often the real culprit is upstream and unglamorous: how the documents were cut into pieces. Chunking decides what your retriever can even see. Get it wrong and the perfect answer sits in your corpus, invisible, because it was split down the middle or buried in a wall of unrelated text. Get it right and mediocre models start looking brilliant.
This is a field guide to the chunking strategies we actually use, when each one wins, and how to choose based on the structure of your own documents rather than a blog post's default of "512 tokens with 50 overlap."
Why chunking is the lever nobody adjusts
Embedding models compress a chunk into a single vector. That vector has to do two contradictory jobs: be specific enough to match a narrow query, and complete enough to answer it once retrieved. A chunk that is too large dilutes its own meaning — one vector trying to represent five topics matches nothing well. A chunk that is too small is precise but context-starved; you retrieve the sentence "This is prohibited" with no idea what "this" refers to.
Every chunk is a bet about what a future question will need to see all at once. Good chunking is really the discipline of keeping semantically complete ideas intact.
The strategies, from crude to careful
1. Fixed-size chunking
Split every N tokens with a fixed overlap. It is fast, predictable, and the right starting point for homogeneous prose. Its weakness is that it is blind to meaning — it will happily cut a table in half or sever a heading from its paragraph. Use it as a baseline, not a destination.
2. Recursive / structure-aware splitting
Split on the largest natural boundary that fits — paragraphs, then sentences, then words — so you only break mid-idea when you truly must. This respects the document's own shape and is our default for most text.
SEPARATORS = ["\n\n", "\n", ". ", " "] # coarse -> fine
def recursive_split(text, max_tokens=350, seps=SEPARATORS):
if count_tokens(text) <= max_tokens or not seps:
return [text]
sep, *rest = seps
parts, chunks, buf = text.split(sep), [], ""
for p in parts:
if count_tokens(buf + sep + p) > max_tokens and buf:
chunks.append(buf)
buf = p
else:
buf = f"{buf}{sep}{p}" if buf else p
chunks.append(buf)
# anything still too large recurses on a finer separator
return [c for ch in chunks for c in
(recursive_split(ch, max_tokens, rest)
if count_tokens(ch) > max_tokens else [ch])]3. Semantic chunking
Embed sentences, then start a new chunk when the similarity between consecutive sentences drops below a threshold — i.e. cut where the topic shifts. It produces beautifully coherent chunks but costs embeddings at index time and is sensitive to the threshold. We reserve it for high-value corpora where retrieval quality justifies the extra pipeline.
4. Document-structure chunking
For anything with real structure — Markdown, HTML, contracts, API docs — parse the hierarchy and chunk along it: one section per chunk, with its heading path prepended. This is frequently the single biggest win, because the document's authors already did the semantic grouping for you.
A worked example makes the difference obvious. In a 40-page employee handbook, fixed-size chunking will regularly merge the tail of the parental-leave policy with the head of the bereavement-leave policy into one vector — so a query about either retrieves a muddled chunk that answers neither well. Structure-aware chunking keeps each policy in its own unit, tagged with its section path, and both queries land cleanly. Same corpus, same model, dramatically different retrieval — decided entirely by where you drew the boundaries. When we audit an underperforming RAG system, this class of "two topics fused into one chunk" error is the first thing we look for, and it is present far more often than not.
The techniques that matter more than the split itself
Once your boundaries are sane, these refinements move the needle further than swapping strategies:
- Overlap, but modestly. A 10–15% overlap keeps ideas that straddle a boundary retrievable. Beyond that you just pay to store and rerank duplicates.
- Prepend context to every chunk. Store the document title and heading path inside
the chunk text:
Employee Handbook > Leave > Parental Leave: .... It disambiguates chunks that would otherwise be identical across documents. - Decouple what you embed from what you return. Embed a small, precise unit for matching, but return a larger parent window for generation. This "small-to-big" pattern gives you retrieval precision and answer completeness.
- Keep tables and code atomic. Never let a splitter cut through a table or a code block. Detect them and treat each as one indivisible chunk.
The best chunk size is the one your evaluation set votes for. Everything in this post is a hypothesis until your own queries confirm it.
How to actually choose
Don't argue about chunk size in a meeting — measure it. Build a small evaluation set of real questions with known-correct source passages, then sweep configurations and score retrieval:
- Assemble 50–100 representative questions and label the passage(s) that should be retrieved.
- Index the corpus under each candidate strategy and size.
- Score recall@k (did the right passage make the top k?) and context precision (how much retrieved text was actually relevant).
- Pick the smallest chunk that keeps recall high — smaller chunks mean cheaper, sharper prompts.
In our experience the winning configuration is almost always structure-aware splitting at 300–500 tokens, 10% overlap, heading context prepended, tables kept whole — but the point is that your corpus decides, not a default.
Conclusion
Chunking is where retrieval quality is quietly won or lost. Before you reach for a larger model or a heavier reranker, look at what your retriever is being handed. Respect document structure, keep complete ideas together, give each chunk enough context to stand alone, and let a real evaluation set settle the numbers. It is the cheapest, highest-leverage tuning you can do in a RAG system — and the one teams skip most often.