Chunking Strategies for Business Documents in RAG Systems

Why chunking quietly decides answer quality

A retrieval system does not search documents. It searches chunks, the fragments you cut documents into before embedding them. If a clause is split in half, or a table is separated from the heading that explains it, the right answer can be indexed and still never be found in a usable form.

Chunking is also the least glamorous decision in a RAG project, which is why it often gets made in one line of code with a default of 1,000 characters and never revisited. That default is rarely right for business documents, which have structure: numbered clauses, headed sections, tables, footnotes and appendices that refer back to the main text.

The main chunking strategies

For most business collections we start structure-aware, fall back to recursive where no structure exists, and add parent-child retrieval when answers need surrounding context.

Chunking by document type

Different documents want different treatment, and a single strategy across a mixed collection is usually a compromise.

• Contracts and terms. Split by clause and keep the clause number and heading in the chunk. Definitions sections deserve their own chunks, because other clauses depend on them.
• Policies and handbooks. Split by section heading, carrying the full heading path, such as 'Leave policy > Parental leave > Eligibility'.
• Technical manuals. Keep procedures whole. A 12-step procedure split into three chunks produces answers that stop at step 5.
• Spreadsheets and tables. Convert rows into readable statements with column headers repeated, or keep small tables whole. Never split a table across chunks without its header.
• Emails and tickets. One message per chunk, with sender, date and subject attached, and quoted history stripped out.
• Slide decks. One slide per chunk, with the deck title and speaker notes.

The upstream work of getting clean text out of PDFs and scans matters just as much. We cover it in preparing documents for AI retrieval.

Chunk size and overlap: sensible starting points

There is no universally correct size, but there are reasonable defaults to test from.

• Roughly 200 to 500 tokens for question-answering over policies and manuals
• Larger, perhaps 800 to 1,200 tokens, when questions need whole procedures or arguments
• Overlap of 10 to 15% with fixed-size splitting; little or none with structure-aware splitting
• Small search chunks of around 150 tokens with a larger returned parent when both precision and context matter

Smaller chunks match questions more precisely but carry less context. Larger chunks carry context but dilute the embedding, so a chunk about six topics matches none of them strongly. Test the trade-off rather than arguing about it.

Metadata: the cheapest quality gain available

A chunk that reads 'The limit is 30 days unless otherwise agreed' is useless on its own. Which limit? Which document? Which version?

Attach context to every chunk, both as filterable metadata and as a short header in the text that gets embedded: document title, section path, document type, effective date, owner and access level. Prepending 'Returns Policy, UK, section 4.2 Refund timing' to that fragment transforms how retrievable it is.

Some teams go further and have a model write a one-sentence summary of where each chunk sits in the wider document. It adds cost at indexing time only, and on long, cross-referencing documents it can be well worth it.

How to test chunking strategies

1. Collect 50 to 100 real questions with the passage that answers each
2. Index the same documents with two or three candidate strategies
3. Measure how often the correct passage appears in the top 5 retrieved chunks
4. Read the failures, which usually reveal a pattern such as split tables or lost headings
5. Adjust, re-index and measure again

Two cheap refinements are worth trying in the same round. First, check whether retrieving neighbouring chunks alongside the best match fixes answers that stop mid-procedure. Second, compare embedding the chunk text alone with embedding it plus its heading path. On structured business documents the second version often wins clearly, and it costs nothing but a re-index.

This is retrieval evaluation in its simplest form, and it takes days rather than weeks. Skipping it means tuning prompts to compensate for a search problem, which works badly and costs more.

Where chunking is not the problem

Chunking fixes fragmentation. It does not fix missing documents, contradictory versions or questions that need an aggregate across hundreds of records, such as 'how many contracts renew in March'. That last kind belongs in a database query, not a retrieval pipeline.

When SpiderHunts audits a RAG system, chunking is one of the first things we inspect, simply by printing the chunks that were retrieved for failed questions and reading them. It is a humbling exercise and often a short one. If you are planning a new assistant, our AI chatbot development work starts with the content, not the model.

Frequently asked questions