The short answer
Users judge waiting by whether something is happening, not by elapsed time. A streamed response starting quickly feels faster than a complete one arriving sooner after silence.
Context matters too. Nobody tolerates a pause while typing; most people accept a wait for a substantial document.
Tolerance depends on the ask
| Interaction | Expectation |
|---|---|
| Inline suggestion while typing | Immediate or not at all |
| Short answer to a question | A moment, with visible progress |
| Analysis of a document | A wait is expected |
| Generating something substantial | Longer, if progress is visible |
| Background processing | Notify when done |
The first row is the strict one. A suggestion that arrives after the user has moved on is worse than no suggestion, because it interrupts.
Make the wait feel shorter
- Respond within a moment with something, even an acknowledgement.
- Stream output as it becomes available.
- Show what stage the work is at, in plain words.
- Let the user do something else while waiting.
- Allow cancellation.
Point five is underrated. Being able to stop removes the feeling of being trapped, which is most of what makes waiting unpleasant.
Reduce the real latency too
- Cache, which makes repeated work instant
- Start retrieval and other preparation in parallel
- Use a smaller model for cases that do not need more
- Keep inputs tight, because longer inputs cost time
- Move work out of the request path where it can be done in advance
Perceptual tricks have a limit. Beyond a point the work genuinely has to get faster, and caching is again the largest lever.
Measure the slow tail
Average response time hides the problem. The slowest few percent of requests are what generate complaints, and they are frequently far slower than the average suggests.
Set your target on that tail rather than on the mean, and alert on it.