Sorting Things Automatically and Correctly

Most classification problems are category problems

Before any model, look at the categories. If your business uses sixty overlapping categories where a person cannot reliably choose between them, no model will either.

Fifteen clear, mutually exclusive categories produce far better classification than sixty overlapping ones, and they produce better reporting too.

Fix the categories first

1. Take two hundred real items and have two people categorise them independently
2. Measure how often they agree
3. Where they disagree, the categories are ambiguous
4. Merge, split or define more precisely until agreement is high
5. Then, and only then, build the classifier

Confidence and routing

• High confidence: route automatically
• Medium: route with a flag for review
• Low: send to a person to categorise
• Corrections captured and added to the evaluation set

Thresholds set from a shadow run against real data, not from intuition. Running in shadow for two weeks gives you the numbers to set them properly.

Monitor the distribution

Where it pays

Support ticket routing, document type identification, enquiry triage, expense categorisation, product classification. All high volume, all currently done by a person reading and deciding.

The value is not just the time. It is the consistency, which makes the resulting data usable for reporting in a way manual categorisation never is.

Frequently asked questions