Ten Mistakes Businesses Make on Their First ML Project
Last updated:
Why first projects are different
A company's first machine learning project carries a burden later ones do not. It is also a test of whether the organisation believes in the approach at all. If it stalls, the second project does not get funded, however good the idea.
That makes the common mistakes more expensive than they look. None of the ten below is technically exotic. Every one of them is cheap to prevent in week one and painful to discover in month five.
Mistakes one to three: the target
- Predicting something nobody will act on. 'Predict customer churn' sounds useful. Unless someone has a retention offer, a budget and a process for calling at-risk customers, the prediction goes into a dashboard and dies there. Design the action first.
- A fuzzy definition of the outcome. What counts as churn: no order in 60 days, a cancelled contract, a downgrade? Teams discover mid-project that sales and finance use different definitions. Write it down and get both to sign it.
- No baseline. If the current method is a manager's judgement or last year's figure plus 5%, measure how good that is before building anything. A model that beats nothing has proved nothing.
Mistakes four to six: the data
- Leakage. The model accidentally learns from information that would not exist at prediction time. A classic example is a 'refund issued' field in a model predicting which orders will be returned. The test scores look spectacular. Production scores do not.
- Random train and test splits on time-based data. If you shuffle three years of orders and test on a random 20%, the model has effectively seen the future. Test on the most recent period instead.
- Assuming the data is clean because the reports look fine. Monthly reports aggregate away the duplicates, blanks and test accounts. Row-level data is where they live. A two-day data audit up front is the best money in the project.
Our guide to data quality prerequisites covers the checks we run before any modelling starts.
Mistakes seven and eight: the build
- Starting with the most sophisticated method. Build the simplest reasonable model first. It gives you a floor, it often turns out to be good enough, and it tells you quickly whether there is any signal at all.
- Building in a notebook with no route to production. A model that lives on one analyst's laptop cannot be deployed, audited or retrained. Decide early how predictions will reach the system where people work, whether that is the CRM, the ERP or a nightly file.
Mistakes nine and ten: after launch
- No monitoring. Models degrade as the business changes. Without tracking live performance against actual outcomes, the first sign of trouble is a sales director asking why the forecast was so wrong last quarter.
- No owner. The contractor leaves, the analyst moves teams and nobody knows how to retrain. Every live model needs a named person responsible for it, even if that person only checks a report monthly.
A cheap checklist for week one
Most of these mistakes can be caught before a line of model code is written. We ask clients to answer these in a single page:
| Question | Prevents mistake |
|---|---|
| Who acts on the prediction, and how? | 1 |
| What exactly is the outcome, in one sentence? | 2 |
| How good is the current method, measured? | 3 |
| Which fields would not exist at prediction time? | 4 and 5 |
| Has anyone looked at 200 raw rows? | 6 |
| Where will the prediction appear for users? | 8 |
| Who owns the model in a year? | 9 and 10 |
If a question cannot be answered, that is not a reason to cancel. It is the first piece of work.
The people mistake that sits behind the other ten
Most of the list above has a common cause: the project was treated as a technical job handed to a technical person. The analyst builds the model, presents a chart, and the operations manager who would have to use it sees it for the first time at the end.
A first project needs three people in the room from week one, even if two of them only give it an hour a week. Someone who owns the decision being improved. Someone who knows where the data comes from and what its quirks are. And someone who can build and deploy the model. In a 30-person business those might be the sales director, the office manager who runs the CRM, and an outside team.
When the decision owner is involved early, mistakes one, two and eight almost fix themselves. They will tell you in the first meeting that the churn prediction is useless without a list their account managers can call, and that the list needs to be in the CRM on Monday mornings, not in a separate tool.
Picking a first project that can survive mistakes
The best first project has a clear outcome, plenty of historical examples, a human already making the decision and a modest cost of error. Lead scoring, invoice categorisation and demand forecasting for a stable product range tend to fit. Pricing, credit decisions and anything customer-facing without review tend not to.
When SpiderHunts takes on a first machine learning project, we deliberately choose scope that shows value within about ten weeks. For realistic timings by project type, see how long it takes to build a machine learning model.
Frequently asked questions
What is the most common machine learning mistake?
What is data leakage in machine learning?
How long should a first machine learning project take?
Do we need a data scientist for our first ML project?
How do we know if our first model is good enough?
Planning your first machine learning project?
Tell us the decision you want to improve and what data you hold. We will give you an honest read on whether it is a good first project and what to check before spending anything.
Related services
What we build for problems like this one