What We Need From You to Start a Machine Learning Project
Last updated:
Less than you fear, but different from normal software
Starting a machine learning project asks for a different kind of preparation from starting an ordinary build. Nobody needs wireframes or a feature list. What matters is data, and specifically whether the past has been recorded in a way a model can learn from. The general list for software projects is in what we need from you to start a project; this is the ML version.
Everything below is something only you can provide. Each item also has a real cost when it arrives late, and we have laid that out at the end so it is easy to plan around.
1. A decision and someone who owns it
We need one person who can say what the prediction is for and approve the scope, the success threshold and the go-live decision. Ideally they are close to the decision itself, or can bring in the person who is.
They also need to be able to answer the uncomfortable question early: what will people do differently when they have this prediction? If nobody can answer yet, that is fine, but it becomes the first thing we work out together in the discovery workshop rather than something to discover after the build.
2. History that records the outcome
A model learns from past cases where the answer is known. So the most important question about your data is not how much there is, but whether the outcome is in it. For a late-payment model, that means the date each invoice was actually paid. For churn, the date and reason each customer left. For demand, not only what sold, but ideally when you were out of stock.
How much you need depends on the task. As our machine learning service page puts it, typically somewhere between a thousand and a hundred thousand or more labelled examples, and enough examples of the rarer outcome to test against honestly. For anything seasonal, a couple of full years is a comfortable start.
Ten years of data without the outcome is worth less than two years with it.
3. The raw export, not a tidy sample
We ask for a genuine export from the source system, with its mess intact. Duplicate accounts, changed product codes, blank fields and odd dates are information. They tell us what the pipeline will have to handle every night in production. A cleaned sample prepared for us hides all of that, and the problems simply appear later at greater cost.
- Pseudonymise identities if you prefer; most models do not need them
- Include the awkward periods, such as system migrations and unusual years
- Include the fields you think are irrelevant; they are sometimes the useful ones
- Send a small sample first if the full export needs approval
Our scoping for AI work asks for around thirty real examples including the awkward ones, as described in what happens before we quote an AI build. The same principle applies to machine learning, just at a larger scale once the project starts.
4. Someone who knows how the data was recorded
Every dataset has a history nobody wrote down. The status code that meant something different before 2023. The branch that entered orders a day late. The field everyone fills with a full stop to get past validation. We need access to a person who knows those stories, for perhaps an hour or two in the first fortnight and again when predictions are first reviewed.
This is usually the most underestimated item on the list, and the one that most often separates a model that makes sense from one that learns an accident of data entry.
5. Access, requested on the day you sign
- Read access to the source systems or database, or a scheduled export
- A cloud account or environment where the work will run, ideally yours
- Access to the destination system where predictions will appear
- Your NDA and, where personal data is involved, a data processing agreement
- Any VPN or IT approvals your internal systems require
Access is the item most likely to delay a start, and almost never because anyone objects. It involves IT policies, third parties and people on leave. Start the requests the day the proposal is signed.
What happens when something is late
| Late item | Typical effect | Can we work around it? |
|---|---|---|
| Data access or export | Stops the data audit; often the biggest delay | Only briefly, using a sample |
| The person who knows the data | Slower audit, higher risk of learning errors | Partly, by listing questions in writing |
| Outcome data missing | May change the project entirely | Sometimes, by redefining the prediction |
| Destination system access | Delays integration, not modelling | Yes, until the integration phase |
| A decision on the success threshold | Blocks the proof of value verdict | Rarely |
You do not need a data scientist, a data warehouse, clean data or an opinion on algorithms. Those are our job. If you are unsure whether what you have is enough, a short conversation about your machine learning idea will usually settle it.
Frequently asked questions
How much data do we need to start a machine learning project?
Do we need to clean our data before contacting you?
Do we need a data warehouse first?
Can we start before all the access is approved?
Wondering if you have enough to start?
Tell us what data you hold and the decision you want to improve. We will tell you what is missing, if anything, and whether it is worth starting now.