LLM Caching Strategies That Cut Costs

Where the money actually goes

Look at the logs of almost any LLM feature and a pattern appears. The same 3,000-token system prompt, the same product catalogue and the same policy documents are sent with every single request. The part that changes, the user's question, is often a few dozen tokens.

You are paying to send the model the same background reading thousands of times a day. On top of that, a support assistant might answer 'how do I reset my password' two hundred times a week, generating a fresh answer each time to an identical question.

Caching attacks both kinds of repetition. It is usually the first thing we look at in a cost review, before model routing or prompt trimming, because it changes nothing about answer quality when done properly. Our guide to reducing token costs covers the other levers.

The three kinds of LLM cache

They stack. A well-designed feature often uses prompt caching everywhere, exact caching for a few deterministic operations, and semantic caching only where questions are generic and answers are the same for everyone.

Prompt caching: the safe, boring win

Most major model providers now offer prompt caching. When the start of a prompt matches a recent request, the provider reuses its earlier processing and charges a fraction of the normal input price for those tokens. Responses also start faster, which users notice.

Getting the benefit is mostly about prompt layout.

1. Put stable content first: system instructions, tool definitions, reference documents, examples
2. Put variable content last: the user's message, retrieved chunks specific to this query, timestamps
3. Keep the stable prefix byte-for-byte identical; a date or request ID near the top breaks the cache for everything after it
4. Check the provider's cache hit metrics in responses, rather than assuming it is working

The most common mistake we see is a template that inserts 'Today is Monday 14 September, 10:42' in the first line. It seems harmless and it silently defeats the cache on every call.

Exact and semantic response caching

Exact caching stores the full response keyed on a hash of the model, prompt version, parameters and input. It suits deterministic tasks: classifying the same product title, summarising the same document, extracting from a file already processed. Include the prompt version in the key, or a prompt change will keep serving old outputs.

Semantic caching embeds each incoming question, looks for a previously answered question within a similarity threshold, and returns the stored answer. It can absorb a lot of repetitive traffic on public FAQ-style assistants.

It also has sharp edges. 'Can I return an item after 30 days' and 'Can I return an item within 30 days' are extremely similar as embeddings and need opposite answers. Thresholds loose enough to save money are often loose enough to be wrong occasionally.

When caching gives wrong or leaky answers

• Personalised answers. Never share cached responses across users when the answer depends on their account, orders or permissions. Scope the cache key by user or tenant.
• Changing facts. Stock levels, prices and opening hours go stale. Set expiry to match how fast the underlying data changes, and invalidate on updates.
• Negation and numbers. Semantic caches struggle with 'not', 'before', 'after' and different quantities. Exclude queries containing them, or require a higher threshold.
• Access-controlled documents. A cached answer built from a confidential document must not be served to someone without access to it.
• Creative tasks. Where varied output is the point, caching makes the product feel robotic.

A cache that saves 40% of calls and serves one customer another customer's order details has not saved anything.

A worked example: a B2B product assistant

Consider a parts supplier with an assistant answering around 20,000 questions a month over a technical catalogue. Illustratively: the system prompt and category guides make up most of each request, so reordering the prompt to enable prefix caching cuts input spend substantially with no change in behaviour. Exact caching on 'summarise this datasheet' requests removes repeated work on popular products.

Semantic caching is then applied only to general questions like delivery areas and account setup, with a high similarity threshold, a one-day expiry and no caching of anything mentioning a part number, price or customer account. Monitoring compares a sample of cached answers against fresh ones each week.

How SpiderHunts reviews caching opportunities

At SpiderHunts we start with a week of call logs and group requests by how much of each prompt is repeated and how often whole requests recur. That tells us, with real numbers rather than guesses, what each cache type could save. Then we list what must never be cached for that product, before writing any caching code.

It fits into the wider cost and architecture work in our AI integration projects. Often the result is an unexciting prompt reordering that pays for the review within the first month.

Frequently asked questions