Scoping against reality
Most disappointing projects were scoped against an expectation formed by marketing rather than by what these systems do. Stating the limits plainly is more useful than another list of possibilities.
None of what follows means machine learning is not worth doing. It means the projects worth doing look different from the ones people often ask for.
It cannot predict the unprecedented
Models learn patterns from history. An event with no precedent in the data - a novel disruption, a first-time regulatory change, a competitor doing something nobody has done - is outside what any model can anticipate.
This is worth stating because forecasting projects are sometimes justified on avoiding the last crisis. A model would not have predicted it, and building one on that basis sets up a failure.
What models do handle well is the ordinary variation that makes up most of business life, which is where the value actually is.
It cannot establish causation from observation
A model finds patterns. It cannot tell you that changing something will change the outcome, only that the two have gone together historically.
This matters because the business question is usually causal. 'Customers who receive the newsletter buy more' does not mean sending more newsletters will increase sales - it may mean engaged customers subscribe.
Answering a causal question requires an experiment or a method designed for it. Where an experiment is impossible, a simulation with honest assumptions is often the next best thing.
Five more it cannot do
- Work without recorded outcomes. If nobody wrote down what happened, there is nothing to learn from.
- Fix a process that does not work. Predicting failures in a broken process produces accurate predictions of a bad outcome.
- Replace a decision nobody has defined. If it is unclear what should happen with a prediction, the prediction is useless.
- Overcome bad data with more data. Systematically wrong records produce a model that reproduces the error faithfully.
- Tell you what your business should do. It informs decisions within a strategy; it does not supply one.
The limits are the useful part
| Ask instead | Rather than |
|---|---|
| Which of these will happen more often? | What unprecedented event is coming? |
| What would a test tell us? | Does this cause that? |
| What should we start recording? | What can we do with no outcome data? |
| Which decision will change? | What insights can we get? |
The right-hand column is what gets asked; the left is what can be answered. Translating between them is most of what a good scoping conversation does.
Knowing what it cannot do is what lets you scope a project that will work.