The short answer
Ordinary web traffic can be absorbed by adding servers quickly. AI capacity is frequently bounded by provider limits or hardware availability, so you cannot scale out of a spike in the same way.
That makes queueing, limiting and degradation design decisions rather than afterthoughts.
Design the degraded mode deliberately
| Under pressure | Better than failing |
|---|---|
| Queue with a stated wait | User can decide to wait |
| Serve a cached result | Slightly stale beats nothing |
| Fall back to a simpler method | Less good, still useful |
| Offer to notify when ready | Removes the wait entirely |
| Disable the feature clearly | Honest, not broken looking |
The fourth row suits anything that is not conversational. Telling someone you will notify them converts a failure into a normal asynchronous workflow.
Limit per customer, not just overall
- Set a per customer ceiling so one cannot consume the capacity.
- Set a global ceiling to protect the provider relationship.
- Queue rather than reject where the work is not interactive.
- Shed lowest value work first, deliberately chosen in advance.
- Make the limits visible so support can explain them.
Point four needs deciding when calm. Choosing what to drop during an incident produces choices you would not have made.
Know your actual ceiling
Find out what your provider limits are, whether they are per minute or per day, and what happens when you reach them. Many teams discover this during their first spike.
Ask about raising them before you need it, since increases frequently take time to arrange.
Test the spike
- Load test to the point of failure, not just to expected load
- Confirm the queue behaves and does not grow without bound
- Confirm the degraded path actually works
- Confirm alerts fire before users notice
- Confirm recovery is automatic once load drops
The last point catches a common defect. A system that survives a spike but needs a manual restart afterwards has not really handled it.