Why Most ML Proofs of Concept Never Ship, and How to Design One That Does

The proof of concept that proved nothing

A familiar story. A data team spends eight weeks on a proof of concept, presents a chart showing the model is 91% accurate, and the room is pleased. Six months later it is still not in production, and nobody quite remembers deciding to stop.

The model was fine. The proof of concept simply answered a question nobody needed answering. It showed a model could fit historical data, which is almost always true. It did not show that the model could get fresh data on time, run where it needed to, beat what the business already did, or be acted upon.

We have written separately about why AI pilots fail on the road to production. This piece is the constructive half: how to design a machine learning proof of concept so that shipping is the default outcome rather than a second project.

Why ML proofs of concept stall

• Built on an extract. Someone exported a clean CSV. Production would need a live feed nobody has scoped.
• Features that do not exist in real time. The model used month-end figures that are only finalised two weeks after the month ends.
• No agreed success number, so any result is open to interpretation and nobody feels able to say yes.
• No decision owner. The data team built it; operations never asked for it.
• Different people for the next stage. The notebook gets thrown over the wall to engineers who start again.

Principle one: agree the go or no-go number first

Before any data is touched, write down what result would justify going to production and what would not. Make it a business number where possible. 'Reduces manual invoice coding by at least 60% at no more than 2% error' is decidable. 'Shows promising accuracy' is not.

Also write down the baseline: how the decision is made today and how good that is. If the current rule of thumb gets 70% right, a model at 74% probably does not justify a production system. Knowing that in week one saves the other seven.

Principle two: use production-shaped data

The proof of concept should use data as it will look at prediction time, pulled the way it will be pulled in production, even if that is slower to set up. That means no fields that are filled in later, no manually cleaned extract, and testing on the most recent period rather than a random sample.

This usually lowers the headline score. Good. A lower, honest number is worth far more than a flattering one that collapses on contact with Monday's data.

Principle three: build the thinnest end-to-end path

A proof of concept that ships has a very rough version of every stage, not a polished version of one. The model can be simple. What matters is that a prediction travels from live data to the place where a person uses it.

Principle four: run it in shadow

The strongest evidence a proof of concept can produce is a fortnight of the model making predictions on live data alongside the people currently making the decision, with nobody acting on the model yet. At the end you compare.

Shadow mode flushes out every practical problem at once: late data, missing fields, predictions that are correct but useless because they arrive after the decision has been made. It also gives the decision owner something concrete to react to, which is how buy-in actually happens.

A realistic timeline

For a typical SME use case, such as scoring inbound leads or predicting which orders will be late, we would expect something like this:

1. Week 1: success number, baseline measured, data access agreed
2. Weeks 2 to 3: production-shaped data pipeline and data audit
3. Weeks 4 to 5: simple model, honest evaluation on recent data
4. Weeks 6 to 7: scores delivered into the working system, shadow mode
5. Week 8: compare against the baseline, go or no-go decision

A no-go at week eight is a success of a kind. It cost eight weeks, not eight months, and the pipeline work is usually reusable for the next idea.

What we insist on

SpiderHunts will not run a machine learning proof of concept without a named decision owner and a written success number. That occasionally loses us work. It has also meant that the proofs of concept we do run mostly end with a clear answer, one way or the other, which is the point of doing one.

We also ask, before starting, who would build the production version if the answer is yes. If it is a different team, they join the proof of concept from week one, even for a couple of hours a week. Handing a finished notebook to engineers who were not there is the single most common place we see a successful proof of concept quietly lose six months. The code gets rewritten, assumptions get lost, and by the time it is live the business has moved on to a different priority.

Frequently asked questions