Start with what you already run
Most business retrieval systems involve tens of thousands of chunks, not hundreds of millions. At that scale, a vector extension to your existing database performs perfectly well.
Adding a dedicated vector database to a system with forty thousand chunks is adding an operational dependency for no measurable benefit.
The options
| Option | Suits | Operational cost |
|---|---|---|
| Database extension | Up to millions of chunks | None — already running it |
| Managed vector service | Larger scale, no ops capability | Subscription |
| Self-hosted vector database | Very large scale, specific needs | Real |
| In-memory index | Small, static datasets | Low, but not durable |
What actually matters
- Metadata filtering during search, not after — essential for permissions
- Update performance, since documents change
- Backup and recovery, like any other data store
- Operational familiarity within your team
Filtering during search is the requirement people discover late, usually when implementing permissions.
Keep the source of truth elsewhere
The vector store is an index, not a system of record. Documents should live in their proper store, with the index rebuildable from them.
That means a corrupted or lost index is an inconvenience rather than data loss, and it makes changing vector store later straightforward.
Plan for reindexing
- Changing embedding model requires a full reindex
- Changing chunking strategy requires a full reindex
- Both will happen at some point
- Make it a scripted operation rather than a project