The short answer
Production AI spend is rarely evenly distributed. A small number of code paths, or a small number of users, usually account for most of it.
Attribute the spend before optimising anything, because the obvious candidate is frequently not the expensive one.
Measure first
- Attribute cost per request to a feature and a customer.
- Rank features by total spend, not by request count.
- Look at the distribution across users, which is usually skewed.
- Identify repeated identical or near-identical work.
- Check what proportion of results are never actually used.
Point five surprises people regularly. Work generated speculatively, or for a screen the user leaves immediately, is pure waste and easy to remove.
The savings in order of value
| Change | Typical effect |
|---|---|
| Cache repeated work | Large, where inputs repeat |
| Stop speculative generation | Large, where it exists |
| Smaller model where sufficient | Large, needs evaluation |
| Shorter inputs and outputs | Moderate, steady |
| Batch where latency allows | Moderate |
Caching is first because it is usually the largest and the least risky. The result is identical, so there is nothing to evaluate.
Route by difficulty
Not every request needs the most capable model. Routing straightforward cases to a smaller one and escalating the rest is a substantial saving where it applies.
- Establish which cases the smaller model handles correctly
- Route those to it, with evaluation covering both paths
- Escalate on low confidence or on a detectable failure
- Monitor the split, because it drifts as inputs change
- Keep the ability to send everything to the larger model quickly
The last point is your safety valve. If quality drops, you want a single switch rather than a deployment.
Do not optimise quality away
Cost reductions that degrade output quietly cost more than they save, because the failures show up as support load and lost trust rather than on a bill.
Every cost change should be checked against your evaluation set, not just against the invoice.