The bottleneck nobody budgets for
Supervised learning needs examples with known answers. For most business problems those do not exist in usable form, and someone has to create them. This is routinely underestimated in both time and difficulty.
It is also delegated badly. Labelling gets handed to whoever is least busy, without guidelines, and the resulting inconsistency limits the model permanently. No amount of modelling recovers from labels that contradict each other.
Write the guidelines before labelling anything
Labelling guidelines are a short document that defines each category and, more importantly, rules on the awkward cases. The awkward cases are the whole point - the obvious ones need no guidance.
- Define each label in a sentence, in the business's own vocabulary.
- Give two or three real examples per label, taken from your actual data.
- Write explicit rules for the boundaries - when something could be two labels, which wins.
- Provide a way to mark 'unclear' rather than forcing a guess, and review those regularly.
- Version the document, because it will change and labels applied under different versions are not comparable.
That last point is often overlooked. If the guidance changes halfway through, earlier labels need revisiting or at minimum flagging.
Measure agreement, and take it seriously
Have two people label the same sample independently and measure how often they agree. This single number tells you more about project feasibility than any amount of discussion.
| Agreement | What it means |
|---|---|
| High | The task is well defined; a model can plausibly learn it |
| Moderate | Guidelines need work, or categories overlap |
| Low | The task as defined may not be learnable - rethink before spending |
Low agreement is not a failure of the labellers. It is information: the categories are ambiguous, or the task genuinely requires context the data does not contain. Finding that out in week one is worth a great deal.
Keeping quality up over time
Labelling is repetitive and quality decays. A few practical measures help more than exhortation.
- Keep an overlapping sample throughout, not just at the start, so drift is visible
- Insert occasional items with known answers as a quality check
- Cap session length - accuracy falls off after a couple of hours of continuous labelling
- Give labellers a route to raise cases the guidelines do not cover, and actually update the guidelines
- Review the 'unclear' pile periodically; it is where new categories reveal themselves
Who should do it
Domain experts produce better labels but are expensive and busy. A common and workable pattern is for experts to write the guidelines and label a gold-standard set, with a larger volume labelled by trained non-experts and measured against that gold standard.
Outsourcing is viable for general tasks and poor for ones needing your specific domain knowledge. If a labeller needs to know how your business categorises a product return, an external vendor will struggle regardless of their quality process.
Two people who cannot agree on a label have told you the model will not either.