Designing for the Day the Model Is Unavailable

Four failures, all certain

1. Provider outage — the API is down or degraded
2. Rate limit — you have exceeded your allowance, usually at your busiest moment
3. Timeout — the response is too slow for the user waiting
4. Unparseable output — the model returned something that does not fit the expected structure

Each needs a decided behaviour before launch. Deciding under pressure at 11pm produces worse decisions.

Queue, degrade or fail clearly

• Background work: queue and retry with backoff. Nobody is waiting; nothing is lost.
• Interactive, non-critical: degrade to a smaller or alternative model.
• Interactive, critical: fail with a clear message and a manual route. Never a spinner that never resolves.
• Unparseable output: retry once with stricter instruction, then quarantine for review.

Silent failure is the one that costs

A feature that quietly returns nothing, or returns empty results that look like “no matches”, can run wrong for weeks. Errors get noticed; silence does not.

So we alert on absence: zero output where output is normal, volume outside the expected band, a sudden change in refusal rate.

Cost failures count too

An unnoticed loop can spend a month's budget over a weekend. Hard caps per day and per user, an alert at a threshold, and a decided behaviour when the cap is reached.

That behaviour should be degrade or queue rather than a crash, and it should be chosen deliberately.

Give someone a switch

A person in the business should be able to disable the AI step in seconds without calling us, with the system falling back to the manual path.

When output is going wrong at volume, that matters more than any feature, and it is trivial to build if designed in from the start.

Frequently asked questions