Why this one usually works
Routing is a well-shaped machine learning problem: plenty of labelled history, a clear target, immediate feedback when wrong, and a modest cost to individual errors. Compared with most business predictions, it is forgiving.
It also has a clear baseline. Whatever share of tickets currently land in the right team first time is the number to beat, and most organisations can measure it from existing data.
Your accuracy ceiling is your label quality
The model learns from how tickets were categorised in the past. If two agents would classify the same ticket differently, the model cannot resolve that - it will reproduce the inconsistency.
Before any modelling, take a sample of a few hundred historical tickets and have two experienced people categorise them independently. Where they disagree is the ceiling on what any model can achieve, and it is often the most useful finding of the project.
- Overlapping categories that mean roughly the same thing to different teams
- A catch-all 'General' bucket absorbing anything ambiguous
- Categories that describe the resolution rather than the request, which is unknowable at arrival
- Historical categories that no longer match how the teams are organised
Let it say 'I am not sure'
A router forced to choose for every ticket will make confident mistakes on genuinely ambiguous ones. Allowing abstention below a confidence threshold turns those into the current process rather than into a misroute.
Set the threshold from the cost of each outcome. If a misroute costs a day of delay on an urgent issue and manual triage costs two minutes, the threshold should be conservative.
Publishing the abstention rate alongside accuracy keeps the picture honest. A router that handles 70% of tickets at high accuracy and passes 30% to a human is usually a better outcome than one that handles everything at moderate accuracy.
Rare but important categories
Some categories matter far more than their volume - a safety complaint, a data breach report, a regulator enquiry. These are exactly the classes a model trained on overall accuracy will handle worst, because ignoring them barely moves the headline number.
Handle them separately. A keyword or rule-based check running alongside the model, tuned to catch everything at the cost of some false alarms, is the appropriate design. Our note on predicting rare events covers the general problem.
The categories that matter most are usually the ones a model sees least.
Measuring the thing that matters
Classification accuracy is a proxy. The business outcome is time to first meaningful response, reassignment rate, and agent time spent triaging rather than resolving.
Track those from before launch so the comparison is real. A router with 90% accuracy that saves twenty minutes a day is worth less attention than one with 80% that removes a bottleneck at the start of every shift.