Your history is not ground truth
Machine learning treats historical outcomes as the truth to learn from. In business data they are a record of what someone entered, which is a different thing.
- A return categorised as 'faulty' because that was the quickest option in the dropdown
- A lead marked 'lost' when it was actually never followed up
- A fault code chosen by an engineer in a hurry from a list of forty
- A customer type set at onboarding and never updated as the business changed
None of these is negligence. They are ordinary artefacts of systems used by busy people, and they place a ceiling on model quality that no algorithm removes.
Finding the wrong ones with the model itself
A useful trick: train a model, then look at the training examples it gets confidently wrong. A model that has learned the general pattern and strongly disagrees with a specific record is often right - the record is mislabelled.
- Train using cross-validation so every record gets a prediction from a model that did not see it.
- Rank records where the prediction disagrees with the recorded label, strongest disagreement first.
- Have a domain expert review the top of that list.
- Record how many were genuinely mislabelled - that gives you an estimate of noise across the dataset.
- Correct what you can, and feed what you learn back into the process that created the errors.
That review is often the most valuable hour of the project, because it reveals how the recording process actually behaves rather than how it is documented.
How much noise you can tolerate
Random noise scattered evenly across classes is survivable - models are reasonably robust to it, and more data helps. Systematic noise is the dangerous kind, because the model learns the bias faithfully.
| Noise type | Example | Effect |
|---|---|---|
| Random errors | Occasional mis-keying | Degrades accuracy, usually gracefully |
| Systematic by category | One team always picks 'other' | Model reproduces the habit |
| Systematic by period | Categories changed in 2024 | Model learns the wrong era |
| Systematic by outcome | Failures recorded, successes not | Model sees a distorted world |
Practical mitigations
Where a full clean is impractical, several things help. Use a recent, cleaner window rather than the full history if the process improved. Reweight towards records with corroborating evidence. Where several sources disagree, treat that as a signal rather than picking one arbitrarily.
And be honest in the reporting. A model measured against noisy labels has a noisy accuracy figure, and the true performance may be better or worse than reported. A small, carefully verified gold-standard test set is worth building for exactly this reason.
A model trained on what people clicked will learn what people clicked, not what happened.