The human constraint arrives first
Doubling document volume doubles the model cost, which is usually manageable. It also doubles the review queue, which is not, because you cannot double a person.
The scaling conversation is really a conversation about raising the straight-through rate, because that is the only lever that decouples volume from headcount.
How to raise straight-through rate
- Analyse what reviewers actually correct, by category
- Fix the top two causes — usually specific document types or fields
- Add validation rules that resolve uncertainty without a person
- Raise thresholds only where the evidence supports it
- Sample rather than review, once a category is proven
Technical constraints, in order
| Constraint | Appears at | Remedy |
|---|---|---|
| Rate limits | Peaks, not averages | Queue and back off; request a higher tier |
| Cost | Gradually | Routing, retrieval quality, caching |
| Latency under load | Bursts | Batch, parallelise, prioritise |
| Index size | Large content sets | Better filtering, not more retrieval |
Peaks matter more than averages
Month-end, quarter-end and seasonal peaks are where systems break. Design for the peak, not the mean, and test at peak volume before you meet it in production.
A system comfortable at average and broken at peak fails precisely when the business least tolerates it.
Watch cost per item, not total
Total cost rising with volume is expected. Cost per item rising means something is wrong — prompts growing, retries increasing, or retrieval returning more than it should.
Track the per-item figure weekly. It is the early warning that a total-cost alert gives you far too late.