Why Businesses Bring Us In After a Failed AI Pilot

The story we hear most often

It usually goes like this. Someone built an AI pilot, internally or with a supplier, and the demo was genuinely impressive. It answered questions about company documents, or read invoices, or drafted replies to customers. The board saw it. A budget was agreed. Then it met real work.

Real documents were scanned at an angle. Real customers asked questions the demo had never seen. Staff tried it for a week and went back to the old way. Monthly API bills arrived higher than anyone had estimated. The person who built it moved to another project, and the pilot has been sitting in a folder for four months while everyone avoids mentioning it.

None of that is unusual. Our longer piece on why AI pilots fail covers the general causes. This post is about what happens when a business asks SpiderHunts to look at one.

What we look at first

We do not start by reading the code. We start by finding out what the pilot was supposed to change for the business, and whether that change was ever measured.

• What task was it meant to take over or speed up, and who does that task today?
• Was there a baseline: how long the task took, how many errors, before the pilot?
• What data was it tested on, and how different is that from the data it now sees?
• Who tried to use it, for how long, and what did they say when they stopped?
• What does it cost per use, and has anyone multiplied that by real volume?
• Who can still explain how it works?

Then we run the pilot against a few hundred real, recent cases and record what happens. That single exercise settles more arguments than any number of meetings.

Wrong problem, wrong data or wrong build?

Almost every failed pilot we review falls into one of three categories, and each needs a very different response.

The fourth row turns up more than you would think. A perfectly good AI feature can fail simply because using it meant opening another tab.

What can usually be salvaged

More than people expect. A pilot that failed in production has still taught the business something expensive to learn: which inputs are hard, which questions users actually ask, which edge cases matter. We treat that as an asset.

1. Real examples collected during the pilot become the evaluation set
2. Prompts and extraction rules that worked on common cases are kept and tested
3. User complaints become a prioritised list of what the rebuild must handle
4. Integration credentials and data access already arranged save weeks

What we rarely keep is the glue: scripts that call a model with no retry, no logging and no limit on cost. That is where most pilots were thinnest, because nobody needs it for a demo.

When we recommend stopping

Sometimes the right answer is to stop, and we say it in writing. If the task requires answers that must be correct every time with no human review, such as final compliance decisions or prices sent straight to customers, a language model is the wrong core component. If the volume is a few cases a week, a rescue will never pay for itself. If nobody in the business wants the outcome enough to change how they work, rebuilding the software will not change that.

A pilot that is honestly retired is a better result than one that is rescued into something nobody needed.

Stopping is not a failure of the pilot team either. A pilot's job is to find out. Some of them find out no.

How a rescue engagement runs

We keep the first step small and fixed-price, so the business is not committing to a rebuild before anyone knows whether it is worthwhile.

1. A short review of the pilot, the data and the business goal, usually one to two weeks
2. A written finding: diagnosis, what to keep, what to rebuild, and a recommendation that may be to stop
3. If continuing, a fixed-scope plan with an evaluation set agreed before building
4. A production build with routing for low-confidence cases, logging, cost limits and monitoring
5. A staged rollout to one team, measured against the original baseline

For the production side, our AI integration service and the checklist in taking an AI pilot into production describe what we add that a pilot usually lacks.

What to bring to the first conversation

Whatever exists: the code or a link to the tool, a handful of cases where it worked and where it failed, the original goal as it was written at the time, and ideally someone who tried to use it. We have reviewed pilots built by internal teams, by freelancers and by other agencies. We are not interested in who to blame. We are interested in whether the idea deserves a second, better attempt.

Frequently asked questions