A/B Testing With Small Numbers, Honestly

The uncomfortable arithmetic

To detect a modest improvement with any confidence you need a substantial number of conversions per variant, not visits. A site with thirty enquiries a month cannot run a meaningful test on enquiry rate — it would take many months, during which everything else changes.

Running the test anyway produces a result, and the result is noise. Acting on it is worse than not testing, because it feels evidence-based.

What to do instead

1. Make changes that are obviously better. Clearer headline, visible pricing, fewer form fields, faster pages. These do not need testing; they need doing.
2. Watch session recordings. Five recordings of real people attempting your conversion path teach more than a underpowered test.
3. Ask customers. Five conversations with recent buyers about what nearly stopped them is cheap and specific.
4. Measure the trend over quarters rather than testing variants over weeks.

Most small sites have several obvious improvements outstanding. Testing is what you do when you have run out of those, and very few businesses have.

Test higher up the funnel

You may not have enough enquiries to test on, but you may have enough clicks, scroll depth or form starts. Testing on a more frequent event gets you a usable signal sooner, provided the event genuinely predicts the outcome.

Be careful: optimising form starts can reduce completions if the change attracts less serious visitors. Check the downstream number even when testing an upstream one.

When testing becomes worthwhile

• Hundreds of conversions per month, not per year
• A stable site where other changes are not confounding the result
• A specific hypothesis rather than a general urge to improve
• Discipline to run the test to completion rather than stopping when it looks good

Stopping early is the classic error

Watching a test and stopping when one variant is ahead produces a false positive most of the time. Early differences are noise, and noise is at its largest when the sample is smallest.

Decide the duration and sample size before starting, and do not look at the result as a decision until then.

Frequently asked questions