Small Language Models for Agents

The agent works. The invoice is the problem.

A common pattern this year: an agent launched on a large general-purpose model, performed well in the pilot, and then ran into volume. Each task involves a dozen model calls. Ten thousand tasks a month turns a pleasing demo into a line item the finance director asks about, and customers notice that responses take several seconds.

Small language models are the practical answer for a lot of that work. Not for everything, and not by simply swapping the model name in the configuration, but for the steps inside an agent that are narrow and repetitive.

What counts as a small language model

There is no official line. In practice people mean models small enough to run cheaply on a single modest GPU, a standard server, or even a laptop or phone, as opposed to the largest frontier models served only through big providers' APIs. Many are open-weight, meaning you can download and host them yourself, and several are good enough at focused tasks to surprise people who last looked a couple of years ago.

Their limits are real. They hold less general knowledge, handle long multi-step reasoning less well, and are more easily confused by unusual inputs. That shapes where they belong.

Which agent steps suit small models

The pattern that follows is a model cascade. A small model handles the first pass and reports its confidence. Clear cases continue on the small model. Unclear ones go to a larger model. In many business workloads the majority of cases are routine, so the average cost per task falls sharply while quality on hard cases is kept.

Three reasons besides cost

• Latency. A small model answering in a fraction of a second changes how an agent feels, especially in voice and interactive use.
• Data control. Hosting a model yourself means sensitive text never leaves your infrastructure, which simplifies some compliance conversations.
• Predictability. A model version you host does not change underneath you when a provider updates it, so your evaluation results stay valid.
• On-device and offline use. Small models can run on laptops and phones, which we cover in on-device AI for business apps.

Self-hosting has costs of its own: servers, monitoring, updates and someone who understands them. For modest volumes, a small model through a hosted API is often more sensible than running your own.

Fine-tuning: when it helps and when it is a distraction

Small models improve a great deal when fine-tuned on examples of your specific task. A few thousand labelled examples of your support categories or your invoice fields can take a small model from mediocre to very good on that one job.

Fine-tuning is the wrong first step when the task is still changing, when you lack labelled examples, or when prompt improvements and better tool descriptions have not been tried. It also creates a model you now have to maintain. We usually start with a well-prompted small model, measure it, and fine-tune only if the gap to the large model is worth closing. That judgement sits squarely within our machine learning work.

Proving a small model is good enough

The only reliable way to decide is an evaluation set built from your real data. Opinions and public benchmarks tell you very little about your task.

1. Collect a few hundred real examples for the step in question, including awkward ones
2. Record the correct output for each, checked by someone who knows the work
3. Run the current large model and the candidate small models against the same set
4. Compare accuracy, cost per thousand calls and response time
5. Look at the failures by hand, since some errors matter much more than others
6. Keep the set and rerun it whenever a model, prompt or tool changes

This is evaluation-driven development, and it is what makes model choice a decision rather than a guess. Our agent evaluation framework goes into the method in more depth.

When to leave the large model alone

If your agent handles low volume, the savings may not justify the engineering. If most tasks are genuinely complex, a cascade sends nearly everything to the large model anyway and adds a failure point. And if the team has no capacity to maintain evaluation sets, a small model that quietly degrades is worse than a large one that costs more.

At SpiderHunts we look at the distribution of tasks before recommending anything. A workload dominated by routine classification is an easy win. A workload of unpredictable requests usually is not.

Frequently asked questions