The cost of a fixed split
A standard A/B test sends half the traffic to each option for the whole test. If B is clearly worse, half your traffic gets the worse experience until the test ends - a known and accepted cost.
A bandit adjusts the split as evidence accumulates, sending more traffic to whatever is performing better while still exploring. Total performance during the test is higher.
The trade you are making
| A/B test | Bandit | |
|---|---|---|
| Traffic split | Fixed | Adapts to results |
| Performance during test | Lower | Higher |
| Clean effect estimate | Yes | Harder - traffic is confounded with time |
| Handles many options | Poorly | Well |
| Easy to explain | Yes | Less so |
The third row is the real cost. Because allocation changes over time, a bandit's results are entangled with when each option was shown. If something else changed mid-test - a campaign, a season, a news event - unpicking it is considerably harder.
Where bandits fit well
- Many options - a dozen subject lines or creatives, where a fixed split gives each too little traffic
- Short-lived content - a promotion running two weeks, where the decision does not outlive the test
- Continuous optimisation - ranking, layout, offers, where there is no final answer to reach
- Expensive exploration - where showing a bad option costs real money per impression
Where a plain A/B test is still right
Where the decision matters, is permanent, and will be questioned, you want a clean estimate of the effect and a defensible method.
- A pricing change you will live with for a year
- Anything with regulatory or contractual implications
- A result that needs to persuade sceptical stakeholders
- Where you need to understand why, not only which
- Where the effect is small and needs careful measurement
In those cases the traffic cost of a fixed split is a reasonable price for a result you can rely on and explain.
Practical cautions
Bandits converge on the option that looks best early, and early results are noisy. A variant that got lucky in the first hours can capture traffic before the truth emerges, so approaches that keep exploring matter.
They also assume the environment is stable. If the best option changes - a seasonal shift, a different audience arriving - a converged bandit keeps serving yesterday's winner. Continuing to explore a small share of traffic guards against that and is worth the modest cost.
A bandit optimises performance. An A/B test produces an answer. Decide which you need first.