Why Machine Learning Needs Less Data Than You Think, and When It Needs More

The myth that stops small firms starting

A lot of business owners assume machine learning is for companies with data centres. They have heard about models trained on the whole internet and conclude that their eleven years of order history is a rounding error.

It is not. The famous models needed enormous data because they learn general things, such as language or vision, from scratch. A model predicting whether one of your customers will reorder this month is learning something far narrower. For narrow problems with well-chosen inputs, a surprisingly modest dataset goes a long way.

Rough numbers, as a starting point

These are orders of magnitude from experience, not rules, and every project differs. They are useful for deciding whether a conversation is worth having.

Why less can be enough

Three things let modest data go far.

1. Good inputs. If you give the model information that genuinely drives the outcome, it does not need millions of examples to find the relationship. A delivery time model fed distance, vehicle type and time of day learns quickly. One fed only postcode strings struggles.
2. Simpler models. Classic methods such as decision tree ensembles work very well on tabular business data at modest sizes. Large neural networks are data-hungry and often unnecessary for spreadsheets.
3. Pre-trained starting points. For text and images, you can start from a model that already understands language or pictures and teach it only your specific task. That is called transfer learning, and it changed the economics for small firms.

Language models go a step further. For many text tasks, such as extracting invoice fields, they work with no training examples at all, just instructions. You still need a few hundred real examples, but for testing rather than training. There is more on that route in our post on extracting data from documents.

When you genuinely need more

• The outcome is rare. If 1 in 1,000 transactions is fraud, 10,000 rows contain ten frauds. That is nothing. You need enough rare cases, which may mean millions of rows or years of history.
• Many factors interact. Pricing across thousands of products, regions and channels needs data covering those combinations.
• Conditions vary a lot. A model for one warehouse may need little data. One that must generalise across forty sites with different layouts needs examples from many of them.
• Raw images, audio or sensor streams without a suitable pre-trained model to start from.
• High stakes and fine margins, where the last couple of points of accuracy carry most of the value and each point costs more data than the last.

The question is rarely how many rows you have. It is how many examples you have of the thing you care about, in conditions like the ones the model will face.

More data is not always better

Old data can hurt. If your customer base, prices or product range changed three years ago, the years before that may teach patterns that no longer apply. We regularly get better results by dropping the oldest data or giving it less weight.

Dirty data can hurt too. Ten thousand carefully recorded examples beat a hundred thousand where the outcome was typed in inconsistently. Doubling the volume of messy data mostly doubles the mess.

How to find out if you have enough

There is a straightforward experiment called a learning curve. Train the model on 10% of your data, then 25%, 50% and 100%, and plot the accuracy each time on held-back cases.

• If accuracy is still climbing steeply at 100%, more data will probably help, and collecting it is worth planning
• If it has flattened, more of the same data will not help, and better inputs or a different approach will
• If it is poor at every size, the inputs probably do not contain the signal, and volume will not fix that

This takes a day or two and replaces a lot of guessing. We run it early on almost every machine learning project at SpiderHunts, because it answers the question clients most want answered: is it worth collecting more before we build properly.

If you really do not have enough yet

Start with rules or a pre-trained tool, and design your processes to capture the outcome from today. Twelve months of well-recorded data is often worth more than ten years of patchy records. In the meantime, look at whether the problem can be narrowed: one product line, one region, one customer segment where the data is dense. Our guide to how long a model takes to build covers how data readiness affects timelines.

Frequently asked questions