Start from cost per task, never cost per token
Token pricing is published and reassuringly small, which is why it misleads. What matters is what one completed unit of business work costs: one enquiry triaged, one document summarised, one ticket resolved.
That figure includes the prompt, the output, the retrieval calls, the retries when the first answer fails validation, and any second model call used to check the first. It is routinely three to ten times a naive estimate.
The five lines a real model includes
- Input tokens, including system prompt and retrieved context — usually the largest slice for document-heavy tasks.
- Output tokens, priced higher than input almost everywhere.
- Retries and validation. Assume 5–15% of calls run twice.
- Retrieval infrastructure — embedding, vector storage, and re-embedding when documents change.
- Human review, which for the first months is frequently the biggest number on the page.
Context length is the cost lever nobody adjusts
The quickest way to halve an AI bill is usually to stop sending so much context. Teams retrieve twenty chunks because it feels safer, when five well-chosen ones give the same answer at a quarter of the cost.
Measure quality against chunk count deliberately. In most retrieval systems we have tuned, accuracy plateaus well before the point teams had settled on, and everything past the plateau is money spent on reassurance.
Cache aggressively
Two kinds of caching matter. Prompt caching, where a stable system prompt or document is charged at a reduced rate on repeat calls, is close to free money for anything with a long fixed preamble. Response caching for repeated identical questions — surprisingly common in support — removes the call entirely.
Both need deliberate design. Put the stable content at the front of the prompt and the variable content at the end, or the cache never hits.
Route by difficulty
Sending every request to the strongest available model is the most common cause of an AI feature that works beautifully and cannot be afforded. Most workloads are a mixture: 70% easy classification, 20% moderate, 10% genuinely hard.
A cheap model with a confidence threshold that escalates to a stronger one typically cuts spend by half or more with no measurable quality difference. Build the routing in from the start — retrofitting it means re-evaluating everything.
Put a ceiling on it before launch
Every AI feature should ship with a hard spend cap, per-user rate limits and an alert at a defined daily threshold. Not because anyone plans to abuse it, but because an unnoticed loop can spend a month's budget over a weekend.
Also decide in advance what happens when the cap is reached: degrade to a cheaper model, queue the work, or fail with a clear message. Choosing that under pressure at 11pm produces worse decisions.