Four components
- Chunking — splitting documents into retrievable passages
- Embedding — turning passages into vectors
- Indexing — storing them so similar meaning can be found
- Retrieval — finding the most relevant for a query
Teams spend most of their attention on the third and fourth. The first is what actually determines answer quality, and it is usually done with a default setting.
Chunk on meaning
Splitting every thousand characters cuts tables in half, separates headings from their content and produces passages that make no sense in isolation.
- Split on document structure — sections, headings, natural boundaries
- Keep the heading with the content it introduces
- Do not split tables
- Overlap slightly so context is not lost at boundaries
- Carry metadata — source, section, date — with every chunk
Filter before searching
Permissions, recency and document type should filter the candidate set before similarity search, not after. Filtering afterwards is both slower and, for permissions, incorrect.
Most vector stores support metadata filtering during search. Use it rather than post-filtering results.
Combine with keyword search
| Query type | Best served by |
|---|---|
| Conceptual question | Semantic similarity |
| Exact product or part number | Keyword matching |
| A specific phrase | Keyword matching |
| Mixed | Both, combined |
Pure semantic search fails badly on exact identifiers, which is a common and surprising problem in technical domains. Combining the two handles it.
Keep the index current
Documents change, and a retrieval system that reindexes nightly can be a day out of date on the morning a price changes.
Watch the source and update affected chunks within minutes. Remove deleted documents immediately — superseded content in the index is the main cause of confidently wrong answers.