Latency in AI Apps: Making AI Features Feel Fast

Where the seconds actually go

A user clicks 'summarise this account'. Eight seconds pass with a spinner. The summary is good, and the feature still feels broken, because eight seconds of nothing is a long time in a business application.

Before optimising, trace a real request end to end. The model call is often only part of the delay.

Output length deserves particular attention. Generation speed is roughly proportional to the number of tokens produced, so a 600-word answer where 150 words would do takes several times as long, and costs more.

Streaming: the biggest perceived improvement

Streaming shows output as it is generated. The total time may be identical, but users see the first words within a second or so, can start reading, and can stop generation if it is heading the wrong way.

For chat-style interfaces and drafting tools, streaming is close to mandatory. It is less straightforward for structured output, where half a JSON object is useless, but even there you can stream a progress indicator, render fields as they complete, or stream a short plain-language summary first.

Users tolerate a slow answer far better than a silent one.

Engineering fixes that cut real time

• Use the smallest model that meets the quality bar for each step. Classification, routing and extraction rarely need the largest model.
• Run independent work in parallel: retrieval, customer lookups and input guardrails do not need to wait for each other.
• Shorten prompts. Remove duplicated instructions and oversized context; reranking lets you send fewer, better passages.
• Enable prompt caching with stable content first, which reduces time to first token on long prompts. See LLM caching strategies.
• Ask for less output. Set length limits and use compact formats.
• Limit agent loops. Each extra step adds a full round trip; cap steps and handle simple cases with a direct path.
• Keep self-hosted models warm and place servers close to users.

Reasoning-heavy model modes deserve a mention. They can improve quality on hard tasks, but the thinking time is real latency. Reserve them for steps that need them, not every call.

Moving slow work out of the user's way

Not everything needs to happen while the user waits.

1. Pre-compute summaries and classifications when data arrives, not when someone opens the record
2. Generate overnight what can be generated overnight: account briefs, report commentary, product descriptions
3. Run long tasks as background jobs with a notification when done
4. Show partial results immediately and enrich them as further steps complete

A CRM that shows an account summary instantly because it was written when the last email arrived feels far faster than one that generates it on click, even if the underlying model is slower.

Interface design for waiting

• Show what is happening: 'Searching 3 policy documents', then 'Drafting reply'
• Keep the rest of the page usable while AI work runs
• Let users cancel and retry
• Show useful non-AI information first, such as the raw record, while the AI part loads
• Set expectations for genuinely long tasks rather than showing an endless spinner

These changes cost little and noticeably change how fast a feature feels, even before any backend work.

Measure the right numbers

Track latency by percentile, not average. A median of two seconds with a 95th percentile of fifteen means one user in twenty has a bad time, and those are the ones who complain. Record separately the time to first token, total time, and the time spent in each pipeline stage, per model and prompt version.

Set targets that match the interaction. Autocomplete needs well under a second. A chat reply should start within a second or two. A generated report can take a minute if the user is told so.

Watch for trade-offs between speed and the other two numbers that matter, quality and cost. Switching a drafting step to a smaller model might save a second and quietly increase how often users rewrite the draft, which costs them far more than a second. Run the evaluation set after every latency change, and keep an eye on user edit rates for a few weeks afterwards.

How SpiderHunts approaches latency

When SpiderHunts is asked to speed up an AI feature, we instrument before we change anything, because the slow part is frequently not where people assume. Typical wins come from parallelising context fetches, moving a step to a smaller model, trimming output and adding streaming, usually in that order of effort versus reward.

If you are building a new feature, it is cheaper to design for speed from the start. Our SaaS development and AI integration teams treat latency budgets as part of the specification, alongside accuracy and cost.

Frequently asked questions