Rate Limits, Queues and Fallbacks in AI SaaS

Monday morning, 9:04

A customer uploads eight thousand archived documents the moment their onboarding call ends. Your worker pool dutifully sends them to the model provider as fast as it can. Within a minute you hit the provider's tokens-per-minute limit. Every other customer's chat responses start failing. Support tickets arrive before the customer who caused it has finished their coffee.

This is the most common scaling incident in AI SaaS, and it is almost entirely preventable. Protecting your own API from abuse is a separate topic, covered in our API rate limiting guide. Here the problem is the opposite direction: living within someone else's limits while keeping every tenant happy.

Know which limits you are actually hitting

Tokens per minute is the one teams underestimate. Capacity planning by request count looks fine until someone sends long documents. Estimate tokens before sending and budget against that.

Queues for everything that can wait

Separate AI work into interactive and background. Interactive work, such as a user waiting on a reply, needs low latency and a small, reserved share of capacity. Background work, such as bulk imports, nightly classification and re-indexing, goes on a queue and is processed at whatever rate your limits allow.

• Separate queues by priority, with interactive traffic never waiting behind bulk jobs
• A rate limiter in front of the provider that both queues respect, shared across all workers
• Idempotent jobs, so a retried task does not produce a duplicate result or a second charge
• Progress reporting in the UI, so a large import shows 'processed 1,200 of 8,000' rather than a spinner
• Dead letter handling for jobs that fail repeatedly, with alerts

Tools such as Celery, BullMQ, Sidekiq or a managed cloud queue all work. The design choices matter more than the library.

Fairness between tenants

Without per-tenant quotas, your largest or most enthusiastic customer determines everyone else's experience. Fair scheduling means each tenant gets a share of capacity, with bursts allowed when there is room.

1. Track token usage per tenant over a rolling window
2. Give each tenant a concurrency cap on background jobs, scaled by plan
3. Use round-robin or weighted scheduling across tenant queues rather than one global first-in-first-out queue
4. Let paid tiers buy higher throughput, which also aligns with usage-based billing
5. Alert your team when one tenant consumes an unusual share, before it becomes an incident

Retries that help rather than make things worse

Naive retries turn a brief provider slowdown into a self-inflicted outage, as every worker hammers the API again at once. Use exponential backoff with jitter, respect any retry-after information the provider sends, and cap total attempts.

Distinguish error types. Rate limit errors should wait and retry. Server errors can retry a couple of times, then fall back. Validation failures, where the model returned malformed output, may deserve one retry with a stricter instruction, then a route to human review. Authentication and bad request errors should never retry at all.

Fallbacks, from best to acceptable

1. Same model, different region or deployment, where your provider or cloud supports it
2. A comparable model from another provider, already evaluated on this feature
3. A smaller, faster model for a simplified version of the task
4. A cached or previous result, clearly labelled if it may be stale
5. Deferred processing, where the task is queued and the user is told honestly it will complete later
6. The manual path, letting the user do the step themselves without AI assistance

Every fallback needs evaluation before it goes live. A fallback model that produces noticeably worse output can do more damage than a short delay. For many features, deferred processing with a clear message is the most honest option. The provider abstraction layer is what makes options one to three practical.

Customers forgive 'this will be ready in ten minutes' far more readily than a wrong answer delivered instantly.

Monitoring and the costs of resilience

Watch queue depth, wait time per priority, error rates by type, fallback activation counts and latency percentiles. A rising fallback rate is an early warning, and a quiet cost change, since fallback models are priced differently.

Resilience is not free. Reserved capacity, second providers and extra evaluation all cost money and engineering time. A product doing a few thousand calls a day probably needs queues, quotas and backoff, and can live without a second provider. When SpiderHunts builds AI SaaS through our SaaS development practice, we size these measures to real traffic rather than to the worst case in a slide.

Frequently asked questions