Where the money goes
Every request carries the instruction, the retrieved context, the conversation history and the question. Output is charged too, usually at a higher rate.
Most expensive integrations are expensive because they send far more context than the task needs, not because the model is costly.
Five levers
- Retrieve fewer, better passages — five relevant beats twenty adequate
- Summarise long history rather than resending every turn
- Cache repeated instructions and stable context where the provider supports it
- Route by difficulty so simple steps use cheaper models
- Cap output length, because generation is the expensive direction
Improving retrieval usually cuts cost and improves accuracy at the same time. It is the rare optimisation with no trade-off.
Long conversations get expensive quickly
Resending the full history every turn means cost grows with the square of the conversation length. A twenty-turn conversation can cost more than the first nineteen combined.
Summarise older turns and keep the recent ones verbatim. Users notice nothing; the bill halves.
Model the cost before you build
| Driver | Question to answer |
|---|---|
| Volume | How many requests per day, at peak? |
| Context size | How much text per request, realistically? |
| Output size | How long is a typical answer? |
| Retry rate | How often does a request need a second attempt? |
Those four give a monthly figure within about twenty per cent, which is enough to decide.
Caps are not optional
A hard daily cap, a per-user cap and an alert well below both. A loop or an abusive user should cost you an alert, not an invoice.
Decide the behaviour at the cap too — degrade or queue rather than crash.