Most classification problems are category problems
Before any model, look at the categories. If your business uses sixty overlapping categories where a person cannot reliably choose between them, no model will either.
Fifteen clear, mutually exclusive categories produce far better classification than sixty overlapping ones, and they produce better reporting too.
Fix the categories first
- Take two hundred real items and have two people categorise them independently
- Measure how often they agree
- Where they disagree, the categories are ambiguous
- Merge, split or define more precisely until agreement is high
- Then, and only then, build the classifier
Confidence and routing
- High confidence: route automatically
- Medium: route with a flag for review
- Low: send to a person to categorise
- Corrections captured and added to the evaluation set
Thresholds set from a shadow run against real data, not from intuition. Running in shadow for two weeks gives you the numbers to set them properly.
Monitor the distribution
| Signal | Means |
|---|---|
| Distribution shifting | Input has changed |
| One category growing sharply | Either real, or a classifier problem |
| Low-confidence proportion rising | Input drifting from the examples |
| Correction rate rising | Quality degrading |
| A category never used | It probably should not exist |
Where it pays
Support ticket routing, document type identification, enquiry triage, expense categorisation, product classification. All high volume, all currently done by a person reading and deciding.
The value is not just the time. It is the consistency, which makes the resulting data usable for reporting in a way manual categorisation never is.