Most labels teach nothing
If you label a thousand random examples, a large share will be cases the model already classifies confidently and correctly. They cost the same to label as the difficult ones and contribute far less.
Active learning inverts the order: train on a small set, ask the model which unlabelled examples it is least sure about, label those, retrain, repeat. The labelling effort concentrates on the boundary where the decisions are actually hard.
How a round works in practice
- Label a modest random seed set - this has to be random, to establish an unbiased picture.
- Train an initial model, however mediocre.
- Score the unlabelled pool and select where confidence is lowest.
- Label that batch, add it, retrain.
- Repeat until accuracy on a held-out set stops improving materially.
Batch sizes matter practically. Labelling one example at a time is statistically neat and operationally absurd; a batch of fifty to a few hundred fits how people actually work.
The traps
- The labelled set stops being representative. It is deliberately concentrated on hard cases, so accuracy measured on it is misleading. Always keep a separate random test set.
- Rare categories get missed. If the model has never seen a category it cannot be uncertain about it. Mix in some random or targeted sampling.
- Uncertainty is misread. An example may be uncertain because it is genuinely ambiguous rather than informative - those add noise, not signal.
- Labelling gets harder. Every batch is drawn from the difficult end, which is tiring and slower per item. Set expectations with the people doing it.
When it is worth the machinery
Active learning adds a loop between labelling and training, which is operational complexity. It pays when labelling is expensive - expert time, medical or legal review - or when the unlabelled pool is enormous relative to what you can afford to label.
| Situation | Worth it? |
|---|---|
| Expert labelling at high cost | Usually yes |
| Millions of unlabelled records | Usually yes |
| A few thousand records, cheap to label | Probably not - just label them |
| Categories still being defined | No - settle the taxonomy first |
A simpler variant that often suffices
Full active learning is not always necessary. A single targeted round - train on what you have, find where the model is weakest, label more of that - captures much of the benefit at a fraction of the complexity.
This is particularly effective where one category is underperforming. Rather than labelling more of everything, label more of that, which is an obvious idea that projects routinely fail to act on.
The examples worth labelling are the ones the model gets wrong, not the ones it finds easy.