Classic ML vs LLM: Which to Use for Classification

A question that used to have one answer

Until recently, classifying text meant training a model. You collected labelled examples, built features, trained something like logistic regression or a gradient boosted model, and measured it. Now a language model can classify a support ticket from a paragraph of instructions and no training data at all.

That has pushed many teams to use an LLM for every classification task by default. Sometimes that is right. Often it means paying per call, waiting hundreds of milliseconds and accepting slight inconsistency for a job a small trained model would do in a millisecond for nearly nothing.

The right choice depends on four things: how much labelled data you have, how many items you classify, how stable the categories are, and how much reasoning each decision needs.

Side by side

When classic machine learning is the better choice

• High volume. A bank categorising a million transactions a month should not be making a million model calls.
• Plenty of history. Years of tickets already tagged by agents are a ready-made training set.
• Stable categories. Expense categories, product taxonomies and fraud flags do not change weekly.
• Structured or mixed inputs. Amounts, dates, merchant codes and a short description suit tabular models better than prose-reading models.
• Strict latency or offline needs. Real-time scoring in a checkout or on a device.

Our post on machine learning for spend classification is a good example of a problem where a trained model is the natural fit.

When an LLM is the better choice

• Little or no labelled data, and no appetite to label thousands of examples before seeing results
• Categories that change with new products, campaigns or regulations
• Decisions that follow written policy, such as 'a complaint mentioning injury always goes to the safety team'
• Long or messy inputs: email threads, contracts, multi-issue messages
• Modest volume, where per-call cost is trivial next to staff time
• Many languages without separate training data for each

LLMs are also excellent for getting started. Classify a few thousand items with an LLM, have people correct a sample, and you have the training data for a cheaper classifier later.

The hybrid pattern most systems end up with

In practice, the best production designs often use both.

1. A trained classifier scores every item and returns a confidence
2. Confident predictions, which are typically the bulk of traffic, are accepted directly
3. Low-confidence items go to an LLM with the written category rules
4. Items the LLM is unsure about go to a human queue
5. Human and LLM decisions feed back into the next round of classifier training

Illustratively, a logistics firm routing 5,000 customer emails a day might find a trained model handles the routine delivery queries confidently, while the LLM deals with the tangled multi-issue messages and a small residue goes to people. Cost tracks the hard cases, not total volume.

Embeddings offer a middle route too. Embed each item with an embedding model and train a light classifier on those vectors. It needs far fewer labels than training from scratch and runs far cheaper than a full LLM call.

What goes wrong with each

Classic classifiers fail quietly when the world shifts: a new product line, a change in how customers phrase things. Monitor prediction distributions and confidence over time, and retrain on a schedule.

LLM classifiers fail through inconsistency and over-reading. The same borderline ticket can land in different queues on different days, and the model may invent nuance the business did not intend. Fix temperature at zero, use a strict enum in structured output, include an 'unclear' category, and test against a labelled evaluation set before trusting it.

How SpiderHunts decides

At SpiderHunts, the first question we ask is how many labelled examples already exist and how many items arrive per day. With solid history and high volume we lean to a trained model, often with an LLM fallback. With little history or shifting categories we start with an LLM and plan the path to a cheaper model once corrections accumulate.

A quick way to decide for yourself: take 300 recent items, label them properly, and run a zero-shot LLM classifier over them in an afternoon. If accuracy is already acceptable and volume is modest, you may be done. If accuracy is acceptable but the monthly call bill at full volume looks uncomfortable, that labelled set plus the LLM's outputs is the start of a training set for a cheaper model. If accuracy is poor, the category definitions probably need work before either approach will succeed.

Either way, both approaches are scored on the same evaluation set before any decision, which keeps the debate about numbers. Our machine learning development team builds both, so there is no incentive to push one.

Frequently asked questions