Knowledge Base Freshness: Keeping RAG Up to Date

The assistant that was right in March

An assistant launches with a carefully indexed set of policies and product guides. It is accurate, people like it, and it is quietly getting worse from the day it launches. Prices change, a returns policy is updated, a product is discontinued, a procedure is rewritten. The source documents are updated in SharePoint or Confluence. The index is not.

Stale answers are more dangerous than missing ones. An assistant that says 'I could not find that' sends someone to check. An assistant that confidently quotes last year's refund window sends a customer the wrong information with a citation attached.

Freshness is an operational problem, not a model problem, and it needs designing as carefully as retrieval itself.

Four ways a knowledge base goes stale

• Changed documents not re-indexed. The source was edited; the old chunks are still being retrieved.
• Deleted documents still searchable. The page was removed from the intranet but its chunks remain in the vector index.
• Superseded versions both present. 'Pricing 2025.pdf' and 'Pricing 2026.pdf' are both indexed, and retrieval picks whichever matches the wording better.
• Sources nobody maintains. The document is technically current because nobody has touched it in three years, and it is wrong.

The first two are engineering problems with clean solutions. The third needs metadata. The fourth needs people, and no pipeline fixes it on its own.

Incremental syncing: re-index what changed

Re-indexing the entire collection nightly works for a few hundred documents and becomes slow and wasteful as the collection grows. An incremental sync scales better.

1. Pull changes from each source using change feeds, webhooks or modified timestamps where the source supports them
2. Compute a content hash per document, and skip documents whose content has not actually changed
3. For changed documents, delete all existing chunks with that source ID, then re-chunk and re-embed
4. For deleted documents, remove every chunk and any cached answers built from them
5. Log each sync: documents added, updated, removed and failed
6. Run a periodic full reconciliation, weekly perhaps, comparing index contents with the source to catch anything the incremental sync missed

Deleting old chunks in the same transaction as inserting new ones matters. A window where both versions exist is exactly when someone asks the question. The ingestion side is covered in preparing documents for AI retrieval.

Metadata that makes freshness manageable

With effective dates, the retrieval query can include a simple filter so only currently valid documents are searched. Citations can show the date too, which lets users spot problems themselves.

Some sources make this awkward. Shared drives are full of files named 'Policy FINAL v3 (2).docx' with no reliable version information. Rather than guessing, agree a simple rule with the business: one designated folder or space per content type counts as authoritative, and everything else is excluded from the index. It is less clever than trying to infer versions automatically, and far more dependable.

How fresh is fresh enough?

Different content changes at different speeds and carries different risk. Match the sync approach to both.

• Stock, prices, order status: do not put these in a RAG index at all. Query the live system through a tool at answer time.
• Support articles and product guides: sync within minutes to an hour via webhooks or frequent polling.
• Policies and contracts: daily sync is usually fine, with effective dates respected.
• Archives and reference material: weekly or on demand.

If a fact changes hourly, it belongs in an API call, not in an index.

Monitoring and ownership

Freshness fails silently, so it needs watching.

• Alert when a sync fails or the lag between source changes and indexing exceeds a threshold
• Keep a handful of test questions about recently changed content and run them after each major update
• Report documents past their review date to their owners each month
• Let users flag an answer as outdated, and route that flag to the content owner, not just the development team
• Track which documents are cited most, and review those first

The ownership point is the one most projects skip. An assistant surfaces the quality of the documents behind it, including the ones everyone had forgotten about. Someone in the business has to own that content, or the assistant will faithfully repeat whatever is out of date.

How SpiderHunts builds freshness in

When SpiderHunts builds a knowledge assistant, the sync pipeline, deletion handling and freshness monitoring are part of the first release rather than a later phase, because the first stale answer does more damage to trust than several slow ones. We also agree content ownership with the client before launch, which is often a short and slightly awkward meeting that pays for itself.

For the broader design of an internal assistant, see how to build an internal knowledge base with AI, and our automation service can connect the sync to the systems your documents already live in.

Frequently asked questions