The short answer
Split the test. Your own application, queueing and database behaviour can be tested at full load with the provider replaced by a stub that returns after a realistic delay. The provider path needs a smaller, targeted test because it costs money.
Most of what breaks under load is yours, not theirs.
Stub realistically
- Return after a delay matching real observed latency, including the slow tail
- Fail occasionally, at roughly the real error rate
- Return rate limit errors sometimes, so your handling is exercised
- Vary response size, since that affects downstream processing
- Include the occasional very slow response, which is what causes pile-ups
The last point matters most. A stub that always returns in the same time will not reveal the queueing behaviour that a slow tail produces.
What you are looking for
| Question | Why |
|---|---|
| At what rate do queues stop draining? | That is your real capacity |
| What happens to latency as concurrency rises? | Where the experience degrades |
| Do retries amplify load? | The most common cascade |
| Does anything run out of connections? | Frequent, and easy to miss |
| Does cost per request hold at volume? | Unit economics |
Test the provider path separately
- Find the documented rate limits and confirm them in your account.
- Run a short controlled test approaching them.
- Confirm your handling of a rate limit response is correct.
- Confirm backoff behaves and does not synchronise.
- Stop before you spend meaningfully or trip a provider protection.
This is a small, cheap test with a specific question. It is not a general load test and should not be run as one.
Test the failure modes deliberately
Turn the provider off in a test environment and confirm your fallback works. Make it slow and confirm timeouts fire. Make it return errors and confirm the circuit breaker opens.
Those three tests find more real problems than any amount of throughput testing, because failure is what actually happens in production.