When the bot says the wrong thing
A customer screenshots your chatbot telling them returns are accepted within 60 days. Your policy says 30. Another customer is told you deliver to an area you do not cover. A third gets a confident, detailed answer about a product feature that does not exist.
The team starts double-checking everything the bot says, which defeats the point of having it. Someone suggests turning it off. You are not sure whether to fix it or scrap it.
Where wrong answers come from
A modern AI chatbot is a language model that writes plausible text. If it is not tightly held to your own information, it fills gaps with whatever sounds right, which is often what is typical for businesses like yours rather than what is true for yours.
The causes usually fall into a few groups:
| Cause | What it looks like |
|---|---|
| Answering from general knowledge | Plausible policies that are not yours |
| Outdated source content | Old prices, old policies, discontinued products |
| Conflicting sources | Website says one thing, a PDF says another |
| Poor retrieval | The right page exists but the bot does not find it |
| No rule for uncertainty | Confident answers where it should say 'I am not sure' |
| Questions needing live data | Guesses about stock, orders or availability |
Only the first is really about the AI model. The rest are about the content and design around it, which is good news, because those are fixable.
What wrong answers cost
- Customers who act on wrong information, then complain or ask for what they were promised.
- Staff time spent correcting the bot's promises.
- Loss of trust in the bot, so customers stop using it and contact your team anyway.
- Possible disputes where a customer relies on what your website told them.
That last point is why this deserves attention quickly. Customers reasonably treat what your chatbot says as what your business says.
How SpiderHunts fixes a chatbot that gets things wrong
- Audit real conversations. We review a sample of transcripts, mark each wrong or weak answer, and trace it to one of the causes above.
- Clean the sources. Remove outdated pages and documents from what the bot can read, resolve contradictions with you, and create a small set of approved answers for policies that must be exact, such as returns, delivery and pricing.
- Restrict the bot to those sources, with instructions to answer only from what it retrieves and to say plainly when it does not know.
- Improve retrieval so the right content is found: better chunking of documents, metadata such as product and region, and search tuned to how customers phrase questions.
- Connect live data for questions that depend on it, such as stock, order status or availability, instead of letting the bot guess.
- Add a confidence check. When retrieval is weak or the question is out of scope, the bot hands over to a person rather than improvising.
- Show sources in answers where it helps, so customers and staff can check.
- Build a test set of real questions with correct answers, and run it before every change to prompts, content or model, so fixes do not quietly break something else.
If your bot is built on a platform such as Intercom, Zendesk, Tidio or a Shopify app, we work within it where the platform allows these controls. Where it does not, we tell you, and explain the options.
A bot you can trust again
The bot answers from the same approved information your team uses, and says so when it cannot help. Policy answers are exact. Live questions get live data. Your team stops second-guessing it, and when content changes, you update it in one place.
Ownership becomes clear as well. Someone in your business owns the approved answers, the weekly review has a short checklist, and the conversation logs show what customers asked that the bot could not answer. Those gaps become new content, reviewed before the bot is allowed to use it.
No chatbot is perfect, and we do not claim otherwise. What changes is that mistakes become rare, visible and quickly fixed, rather than frequent, hidden and repeated. The test set keeps them from coming back.
Is your chatbot causing problems?
- Customers have quoted wrong information from your chatbot.
- The bot answers questions confidently that it should not answer at all.
- Nobody reviews its conversations regularly.
- It cannot see live order, stock or booking data but answers questions about them.
- Your team has started to distrust it.