Why it happens
A language model produces plausible continuations. Asked something it has no grounding for, plausible is all it has, and plausible reads exactly like correct.
The fix is not to ask it to try harder. It is to ensure it has the right material and a legitimate way to say it does not.
Four measures that work
- Constrain to retrieved context and instruct that answers must come from it
- Require a citation for every factual claim, and reject uncited output
- Design the refusal so “I do not know” is an approved, useful answer
- Verify against structured data — prices, stock, dates — rather than trusting prose
The third is the one businesses resist and the one that helps most. A system with no acceptable way to refuse will always invent.
Numbers should never come from prose
Prices, quantities, dates and balances should be looked up from your systems and inserted, not generated. A model reproducing a number from a document will occasionally reproduce it wrongly.
Where the answer includes a figure, fetch the figure. That single rule removes most of the errors that actually cost money.
Make wrong answers detectable
- Citations that a reader can check in one click
- A visible note when confidence is low
- One-click reporting that captures the full context automatically
- Sampled review of live answers, weekly, by someone who knows
Set expectations in the wording
“Based on our documentation” sets a different expectation from an unqualified assertion. Users read more carefully when the framing invites it.
It is free, it takes one sentence, and it measurably changes how people treat the output.