Taking an AI Pilot Into Production

Why so many pilots stall

The commonly quoted figure is that most corporate AI pilots never reach production. Our experience matches it, and the reasons are consistent — and almost never technical.

1. It was built on curated data and the real data is messier
2. Nobody agreed in advance what success would look like
3. It has no failure handling, because demos do not fail
4. It cannot integrate with the systems it would need in production
5. Nobody was named as its owner afterwards
6. The cost per transaction at real volume was never calculated

Design the pilot to graduate

The fix is to constrain the pilot up front so that success means something.

• Use real data from day one, including the ugly examples
• Write the success threshold before starting, as a number
• Pick a task narrow enough to finish in three weeks
• Include one real integration, even a crude one
• Name the person who will own it in production

A pilot with no agreed success threshold cannot succeed. It can only be liked, and liking does not survive a budget round.

The eight things a production system has that a pilot does not

1. Error handling for every failure mode, including provider outages
2. Monitoring and alerting on quality, not just uptime
3. Cost controls and a spend cap
4. Access control and audit logging
5. A rollback path — a switch that turns it off cleanly
6. An evaluation set that is re-run when anything changes
7. Documentation for whoever inherits it
8. A named owner with time allocated

These are roughly 60% of the total effort. That ratio is why a pilot built in three weeks takes three months to productionise, and why quoting the pilot cost as the project cost misleads everybody.

The threshold conversation, held early

Decide before building: what accuracy makes this worth deploying? It forces the useful question of what happens to errors, and it prevents the endless-improvement trap where a system is never quite good enough to ship.

A good threshold has three parts: the accuracy target, the volume it must sustain, and the cost ceiling per transaction. All three, agreed in writing, before anyone starts.

Shadow mode: the step that de-risks everything

Run the system on real work without acting on its output. Compare its decisions to the human ones for two to four weeks.

You get a genuine accuracy measurement on real data, a business case built from evidence, and a list of failure cases to fix — all with zero risk, because nothing the system produces is used. We do this on every AI project that touches a live process.

Who owns it afterwards

AI systems need an owner in a way that conventional software does not. Someone has to look at the quality metrics, review the corrections, notice drift and decide when to re-evaluate.

It does not need to be a full-time role — typically a couple of hours a month once things settle. But it must be somebody's name, and the handover has to include that expectation explicitly, or the system will be excellent for six months and unloved by month nine.

Frequently asked questions