The prerequisite that is not one
A common sequence: a business wants a prediction, is told it needs a data warehouse first, and starts a large infrastructure project. A year later the warehouse exists and the original question is still unanswered.
Sometimes a warehouse is genuinely needed. Frequently a first project needs a well-defined extract of a few tables, which is weeks of work rather than a year.
What the model actually requires
- The specific fields it uses, not everything in every system
- Enough history for the patterns that matter
- Outcomes recorded reliably so there is something to learn from
- A repeatable way to produce the same extract for retraining
- At prediction time, the same fields available for current records
That is a considerably smaller requirement than a warehouse. The fourth point is the one that separates a working system from a one-off analysis - if the extract cannot be reproduced on demand, you cannot retrain.
A proportionate middle ground
| Approach | Effort | Suits |
|---|---|---|
| Manual export | Lowest | Feasibility only - not sustainable |
| Scripted extract to files | Low | A first project, batch predictions |
| Small dedicated database | Moderate | Several related models, scheduled refresh |
| Full warehouse | High | Many consumers beyond machine learning |
The second and third rows cover most first and second projects. Moving to the fourth should be driven by having several consumers of the data, not by a single model.
When you do need the warehouse
There are real triggers, and recognising them prevents both premature building and indefinite deferral.
- Several models and reports need the same joined data, and each is rebuilding it separately.
- History must be preserved as it stood, and source systems overwrite.
- Extracts have started affecting the performance of operational systems.
- Nobody can say which version of a number is correct because several pipelines produce it differently.
- Compliance requires a consistent, auditable record of what the data was.
The fourth is the usual tipping point in practice. Once the business is arguing about which report is right, a single well-governed source pays for itself.
Do not paint yourself into a corner
Starting small should not mean starting badly. A scripted extract should be in version control, documented, scheduled, and structured so it could feed a warehouse later.
The failure to avoid is a one-off extract produced by hand that nobody can reproduce. That gives you a model you cannot retrain, which has a short and disappointing life.
Build the warehouse when several things need it. Not because one model might.