A common and expensive pattern
A pilot is run, produces good results, is presented, and is received well. Then nothing happens. Six months later it is referenced as evidence that this does not work here.
The pilot was usually fine. What was missing was a route from pilot to production that anyone had thought about.
One: integration was never scoped
The pilot ran on an extract in a notebook and produced a spreadsheet. Production needs it inside the system where decisions are made, which is a different and frequently larger piece of work.
That work is often owned by a different team with its own priorities. Discovering at the end that the necessary change is in next year's roadmap stops the project regardless of how good the model is.
Two: nobody owns it
A pilot is owned by whoever ran it. A production system needs someone accountable for it working - monitoring it, deciding when to retrain, fielding questions when a prediction looks wrong.
If that person does not exist, the system has no home. Projects frequently reach this point and discover that the data team considers it the business's, and the business considers it the data team's.
Three, four and five
- The pipeline was manual. Someone exported data by hand for the pilot. Production needs that automated, and it is often more work than the model.
- Nobody budgeted for running it. The build was funded; hosting, monitoring, retraining and support were not, and no budget line exists to create.
- The pilot proved accuracy, not value. It showed the model predicts well but never established what acting on it would be worth, so there is no business case for the next stage.
That last one is the most common and the most avoidable. A pilot should test whether the improvement is worth having, not only whether prediction is possible.
Decide the production path before the pilot
- Write down where predictions will appear and who must agree to that change, before the pilot starts.
- Name the person who will own the system in production, and involve them from the beginning.
- Establish the baseline first, so value can be measured rather than asserted.
- Estimate running costs early and get them into a budget line.
- Define what the pilot must show for production to proceed - and what would mean it should not.
None of this makes the pilot much more expensive. It changes it from a demonstration into a decision, which is what it should have been.
A pilot that only proves the model works has not answered the question anyone needed answering.