Volume is rarely the problem
Businesses commonly assume they do not have enough data. In practice the blocking issues are structural, and a business with a few years of ordinary trading records usually has enough volume for a first project.
The six questions below can be answered by someone who knows the business, without technical help, and they identify most reasons a project would fail.
The six questions
- Is the outcome recorded? To predict something, you need historical examples of it having happened or not. If nobody records whether the lead converted or the part failed, there is nothing to learn from.
- Can you reconstruct what was known at the time? If your systems overwrite rather than keep history, you may be unable to rebuild the state at the moment a decision was made.
- Can the systems be joined? If the customer in one system cannot be matched to the account in another, the combined view may not be available at any reasonable cost.
- Is the data consistent over time? A system change, a category restructure or a definition change mid-history splits your data into incomparable periods.
- Would two people record the same event the same way? Where they would not, that inconsistency caps what any model can achieve.
- Will the fields exist at prediction time? A field only populated after the event cannot be used to predict it.
Reading the answers
| Answer pattern | What it means |
|---|---|
| Yes to all six | Proceed to feasibility with confidence |
| No to outcome recording | Start recording; revisit in a year |
| No to reconstructing history | Limits features severely; possible but constrained |
| No to joining systems | Data engineering project first |
| No to consistency | Use the consistent period only, accepting less data |
| No to recording consistency | Fix definitions before modelling |
Notice that most of these are not 'no' but 'not yet, and here is the step'. That is usually the honest position, and it is far more useful than an unqualified assessment either way.
The question behind all six
Underneath is a single principle: a model can only learn from information that was recorded, at the time, in a consistent way, and that will be available again when a prediction is needed.
Most data readiness problems are a violation of one of those four. Checking each explicitly is faster than a general data quality exercise and points directly at what to fix.
If the answer is not yet
Discovering that data is not ready is a useful outcome, not a failure. The fix is usually specific and modest - record an outcome field, stop overwriting a status, agree a definition - rather than a large programme.
Businesses that make those changes now have usable data within a year, which is considerably faster than most expect. Our note on collecting data for future machine learning covers what to change.
You do not need more data. You need the data you already generate to be recorded properly.