Split testing needs volume you may not have
Reliable split testing requires a substantial number of conversions per variant. A store with forty orders a month cannot reach statistical confidence on a small change within any useful timeframe.
Running an underpowered test and acting on the result is worse than not testing, because it produces confident conclusions from noise.
Sequential testing instead
- Record four weeks of baseline before changing anything
- Make one change
- Measure four weeks after
- Compare, allowing for seasonality
- Then make the next change
It is slower and it is honest about what small-store data can support.
Prefer changes with obvious effects
- Showing delivery cost earlier
- Adding prices where there were none
- Halving the page load time
- Adding reviews to product pages
- Removing a required field from checkout
Those produce effects large enough to see without sophisticated measurement. Fine-grained optimisation needs volume you probably do not have.
Watch for confounds
| Confound | Guard |
|---|---|
| Seasonality | Compare against the same period last year |
| A campaign running | Note what else changed |
| Traffic source shifting | Segment by source |
| Stock availability | Note anything that went out of stock |
Write down what you changed
A simple log — date, what changed, why, what happened — is worth more than any testing tool for a small store. Six months later it is the only record of what actually worked.
Most stores make dozens of changes a year and can recall almost none of them.