Context, not volume
Most expensive AI services are expensive because they send far more context than the task needs, not because they handle many requests.
Sending an entire document when three paragraphs would do multiplies your bill and slows every response. Better retrieval is a cost lever, not only an accuracy one.
Five levers
- Retrieve fewer, better passages
- Summarise conversation history rather than resending everything
- Cache stable instructions and repeated context where the provider supports it
- Route by difficulty — cheap models for simple steps
- Cap output length, because generation costs more than input
Track cost per request, not total
Total cost rising with volume is expected. Cost per request rising means something has changed — longer prompts, more retries, or retrieval returning more than it should.
- Record tokens in and out on every request
- Aggregate by endpoint and by caller
- Alert when the per-request figure moves
- Review monthly against the previous month
Hard caps with decided behaviour
| Cap | Behaviour at the cap |
|---|---|
| Daily total | Queue non-urgent work |
| Per user or caller | Reject with a clear message |
| Per request | Truncate context, not silently |
| Monthly budget | Alert well before, then degrade |
A cap with no decided behaviour becomes a crash at the worst moment. Decide what happens before you need it.
Conversations get expensive quickly
Resending full history each turn means cost grows with the square of conversation length. A twenty-turn conversation can cost more than the first nineteen combined.
Summarise older turns, keep recent ones verbatim. Users notice nothing and the cost halves.