The short answer
Repeated inputs are far more common than teams expect, particularly for classification and extraction. Caching those removes both cost and latency entirely for the repeats.
The risk is serving an answer produced by a prompt or model you have since changed, which is solved by putting both in the cache key.
What is safe to cache
| Case | Cacheable? | Note |
|---|---|---|
| Classifying the same text again | Yes | Input determines the answer |
| Extracting fields from the same document | Yes | Deterministic enough |
| Summarising a fixed article | Yes | Until the article changes |
| Answering about live data | No | The data moves |
| Anything personalised | Per user only | Never share across users |
| Creative generation | Usually not | Variety is the point |
The personalisation row is a correctness and privacy issue rather than a performance one. A cache keyed without the user can serve one person's answer to another.
Key it properly
- Normalise the input: trim, collapse whitespace, consistent case where appropriate.
- Include the model identifier.
- Include a prompt version that you increment when the prompt changes.
- Include anything else that changes the answer, such as language or user context.
- Hash the result for a compact key.
Point three is the one that prevents the worst failure. Without it, improving a prompt has no effect for every input already cached, and you will spend a day wondering why.
Expire sensibly
Answers about static content can live a long time. Anything touching data that changes needs a lifetime matched to how fast it changes, and anything about live state should not be cached at all.
Where you are unsure, a short lifetime still removes the burst of identical requests that arrive together, which is frequently where the cost is.
Measure the hit rate
If the hit rate is low, the cache is adding complexity without benefit and the key is probably too specific. If it is high, that is money saved and worth reporting.
Track it per use case rather than overall. One high-volume endpoint with repeated inputs frequently carries the whole benefit.