The failure is almost always content
Businesses assume the difficulty is the AI. It is not. Retrieval over documents is well-understood and quick to build.
The difficulty is that the documentation is three years old, contradicts itself in two places, and the real answer lives in someone's head. An assistant over that produces confident wrong answers faster than a person could find them.
What to do before building
- List the twenty questions people actually ask most
- Find the document that answers each — several will not exist
- Check each answer is current with whoever owns it
- Write the missing ones, briefly
- Delete or archive superseded versions, properly
That is a fortnight of unglamorous work and it determines whether the project succeeds.
What good looks like in use
- Answers cite the document and section, always
- “I do not have information on that” when the documents are silent
- Respects permissions — HR documents do not surface for everyone
- Lives where people already are: Slack, Teams, the intranet
- A one-click route to a person when the answer is not there
Measuring it
Questions asked, answer rate, citation click-through, and the volume of the same questions still reaching people directly. That last figure falling is the real proof.
Also worth tracking: questions asked that could not be answered. That list is your documentation backlog, generated automatically.
Keeping it current
Documents change; the index must follow within hours, not months. And someone should own the twenty core documents with a review date.
Without that ownership, the assistant decays back to confidently wrong within a year, which is the most common ending for these projects.