Ten Questions to Ask Before Buying Anything Described as AI

Demos are built to succeed

Every AI demo you see has been rehearsed on data chosen to work. That is not dishonest, it is what a demo is. It simply tells you very little about performance on your material.

The questions below get at what happens afterwards, which is what you are actually buying.

The ten questions

1. What accuracy do you achieve on our data? Ask for a trial on your ten most awkward examples, not their samples.
2. What happens when it is wrong? There should be a review path and a correction mechanism.
3. How do we correct it, and does it learn? Corrections that vanish mean the same error recurs forever.
4. Where is our data processed and stored? Region, provider, and whether it leaves your jurisdiction.
5. Is our data used to train your models? Get the answer in the contract, not the sales call.
6. What does it cost at ten times our current volume? Usage pricing has a habit of surprising people.
7. Can we export everything if we leave? Including the corrections and configuration you invested in.
8. Which model does it use, and what happens when that changes? Behaviour can shift under you.
9. What is your uptime commitment and what happens during an outage?
10. Who else in our sector uses it, and can we speak to them?

Insist on a trial with your own data

The single most useful thing you can do is give three vendors the same twenty real examples with known correct answers, and compare. It takes an afternoon and it settles arguments that would otherwise run for months.

A vendor unwilling to run a trial on your data is telling you something about how their product performs outside controlled conditions.

Watch for the accuracy sleight of hand

Accuracy figures need context: on what data, measured how, at what confidence threshold, and per field or per document? A system with 95% field accuracy might route half your documents to human review, which is a very different proposition.

Ask specifically for the straight-through processing rate — the proportion needing no human touch — because that is the number that determines the saving.

Model the running cost honestly

AI products often price per document, per seat plus usage, or on a consumption model. Model it at realistic volume including seasonal peaks, and ask what happens if you exceed a tier mid-month.

Also ask about the human review time you will still be paying for. That is frequently the largest cost and it never appears on the vendor's pricing page.

Frequently asked questions