Data Labelling: Who Does It, What It Costs and How to Keep It Accurate

Labels are the part of the model you actually own

Algorithms are free and interchangeable. The labelled examples that teach a model what 'urgent complaint', 'defective weld' or 'likely fraud' means in your business are not. They are the single most valuable asset a supervised machine learning project produces, and the one most often rushed.

A model trained on inconsistent labels learns the inconsistency. If two of your staff disagree about whether a ticket is a billing issue or a technical one, the model will split the difference and be confidently wrong on both. No amount of tuning fixes that later.

Who should do the labelling?

There are three realistic options for a smaller business, and most projects end up mixing them.

Our default is to have the most knowledgeable people label a small, carefully chosen set first, use that to write the guidelines, then decide whether the bulk can be handled by others.

How much does data labelling cost?

For most businesses the cost is time, so estimate it that way. Time a person labelling 50 real items, take the average, and multiply.

An illustrative example: a support team wants 6,000 tickets tagged into 12 categories. A careful agent averages 25 seconds per ticket once they have the hang of it. That is roughly 42 hours. Add 20% of items labelled a second time for quality checking, plus a few hours writing and revising guidelines, and you are near 55 hours. If an agent's time costs the business around £25 an hour, that is under £1,500 of effort, which is usually far less than people fear.

Specialist work is different. Radiology images, legal clauses or engineering drawings may take minutes each and need expensive people, and that is where careful sampling pays for itself.

• Seconds per item, measured on real data, not guessed
• Number of items actually needed, which is often fewer than planned
• Share of items double-labelled for checking, typically 10-20%
• Guideline writing and at least one revision round
• Tooling, which can be a free open-source labelling tool or a simple internal page

Write the guidelines before labelling at scale

Labelling guidelines are a short document that answers the questions labellers will otherwise answer inconsistently. What counts as a complaint rather than feedback? What if a ticket mentions two issues? What do you do when you cannot tell?

Good guidelines include real examples of each label, including the awkward borderline ones, and an explicit 'unsure' option. Forcing people to pick a category when they are uncertain produces noise that looks like data.

If two experienced people cannot agree on the label, a model will not agree with either of them. Fix the definition, not the model.

How to keep labels accurate

1. Double-label a sample. Have a slice of items labelled independently by two people and measure how often they agree.
2. Keep a gold set. A few hundred items with agreed, reviewed labels, used to check new labellers and later to evaluate the model.
3. Review disagreements weekly. Each disagreement is either a guideline gap or a genuinely ambiguous case, and both are worth knowing.
4. Track labeller consistency over time. Quality often dips after the first few hundred items as fatigue sets in.
5. Version the labels. When guidelines change, record which items were labelled under which version.

If agreement between people is low, pause. It is a strong sign the categories themselves need redesigning, and we would rather find that after 300 items than 6,000. The same thinking underpins how we approach evaluating AI output quality generally.

Using AI models to pre-label

One of the genuinely useful developments of the last couple of years is that general language and vision models can produce a reasonable first pass for many labelling tasks. A human then confirms or corrects, which is usually much faster than starting from nothing.

The risk is anchoring. People shown a suggested label tend to accept it. We handle this by hiding suggestions on a random sample and comparing, so we can measure whether reviewers are actually checking. For some tasks the pre-labelling model turns out to be good enough to use directly, which is worth knowing before commissioning a custom model at all.

Where real data is scarce or sensitive, some teams also consider generated examples. Our note on synthetic data for machine learning covers where that helps and where it quietly misleads.

When labelling is the wrong investment

Sometimes the labels already exist, hidden in your systems. A refund issued, a job re-opened, an invoice written off: these outcomes are labels that cost nothing. Before organising a labelling exercise, SpiderHunts always checks whether a system event already records what you want to predict.

Labelling is also a poor investment when the categories will change next quarter, or when a general AI model with a good prompt already performs well enough. In those cases the effort goes into evaluation rather than training data. If you are weighing up a custom model, our machine learning service starts with exactly that comparison.

Frequently asked questions