The short answer
Two things change between pilot and production. Cost becomes real, because it scales with volume. And quality problems appear, because a wider audience sends inputs the pilot never produced.
Model the first before launch and test the second deliberately, because neither shows up while ten friendly users are trying it.
Cost at real volume
- Measure the actual cost of a single typical request, including retries.
- Multiply by realistic volume, not optimistic volume.
- Add the cases that cost more: long inputs, retries, fallbacks.
- Check what happens at peak, not just at average.
- Compare that against what the feature is worth.
Step three catches people out. An average cost per request hides the long tail, and the long tail is where a surprising share of the bill comes from.
What breaks first
| Constraint | Symptom at scale |
|---|---|
| Provider rate limits | Errors under load, not under test |
| Latency under concurrency | Fine alone, slow when busy |
| Retry storms | One failure cascades into many |
| Cost per request | Unit economics stop working |
| Input variety | Quality falls on inputs nobody anticipated |
Rate limits are the most common first failure and the least visible in testing, because a pilot never approaches them.
Reduce before you scale
- Cache responses for inputs that repeat, which is more than expected
- Batch where the work is not interactive
- Use a smaller model for the easy cases and escalate only when needed
- Shorten inputs, since much of what gets sent is not needed
- Precompute anything that does not depend on the moment
The third is usually the largest saving. Routing the straightforward majority to something cheaper and escalating the rest keeps quality where it matters and cuts cost substantially.
Test against real inputs
Collect the inputs the pilot produced, then deliberately gather the awkward ones: very long, very short, wrong language, nonsense, adversarial. Run those before launch rather than discovering them in production.
Keep that set and re-run it whenever anything changes. It becomes your regression suite, and it is the only thing that catches quality drift after a model or prompt change.