The short answer
AI calls are slow relative to ordinary requests and cost money per call, so a burst of traffic is more damaging than it would be elsewhere. A queue absorbs the burst; backpressure stops you accepting more than you can ever process.
Without both, a spike turns into timeouts, retries and a bill for work nobody used.
Decide what is interactive
| Request type | Pattern |
|---|---|
| User waiting on screen | Synchronous, with a timeout and a fallback |
| User expects it shortly | Queue, notify when done |
| Background processing | Queue, no urgency |
| Bulk job | Batch, scheduled off peak |
Moving work down this table is the cheapest scaling decision available. A great deal of what is built as synchronous does not need to be.
Backpressure, not unlimited acceptance
A queue that accepts everything grows without limit, and work at the back becomes stale before it is processed. Telling a caller you are busy is better than accepting a job you will process in three hours.
- Cap the queue depth rather than letting it grow
- Reject or shed when the cap is reached, with a clear response
- Give callers a way to know how long the wait is
- Prioritise, so important work is not behind bulk jobs
- Expire work that is no longer useful rather than processing it
Retries need care
Retrying failed AI calls is normal and dangerous. A provider under load returns errors, everyone retries, and the load increases. That is how a small problem becomes an outage.
- Back off exponentially rather than retrying immediately.
- Add jitter so retries do not synchronise.
- Cap total attempts.
- Stop retrying when a circuit breaker opens.
- Count retries in your cost model, because they are billed.
Fail in a defined way
Decide what the user sees when the queue is full or the provider is down. A cached answer, a simpler non-AI path, or an honest message are all better than a spinner that never resolves.
That decision belongs in the design. Left undefined, the default is whatever the code happens to do under pressure, which is rarely what you would have chosen.