The notebook is the easy part
Training a model that performs well on historical data is a well-understood exercise. Running it as a dependable component of a business system is a different discipline entirely.
The gap between a promising notebook and a reliable service is where most business machine learning projects stall, and it is almost entirely engineering rather than data science.
What production requires
- Versioned models, with the training data and code recorded
- Reproducible predictions — the same input gives the same output
- Input validation, because production data differs from training data
- Monitoring for drift in inputs and in outputs
- A fallback for when the model is unavailable or unconfident
Drift is the thing that catches people
A model trained on last year's data becomes less accurate as the world changes. Nothing errors; the predictions simply get worse, and nobody notices until someone questions a result.
- Monitor the distribution of inputs against training data
- Monitor the distribution of predictions
- Track accuracy against actual outcomes where they become known
- Set a retraining trigger and a schedule
Feature consistency
The features computed at prediction time must match those computed at training time exactly. Divergence between the two is one of the most common and most subtle production failures.
Share the feature computation code between training and serving rather than reimplementing it.
Be honest about whether it is needed
| Problem | Frequently better solved by |
|---|---|
| Classification with clear rules | Rules |
| Small data volume | Human judgement |
| Forecasting stable seasonality | Statistical methods |
| Text understanding | A language model, not custom training |
| Genuine pattern recognition at volume | Machine learning |
A great deal of what is proposed as machine learning is better solved by simpler methods, and we say so reasonably often.