Changing Things Without Losing Sales
Last updated:
Split testing needs volume you may not have
Reliable split testing requires a substantial number of conversions per variant. A store with forty orders a month cannot reach statistical confidence on a small change within any useful timeframe.
Running an underpowered test and acting on the result is worse than not testing, because it produces confident conclusions from noise.
Sequential testing instead
- Record four weeks of baseline before changing anything
- Make one change
- Measure four weeks after
- Compare, allowing for seasonality
- Then make the next change
It is slower and it is honest about what small-store data can support.
Prefer changes with obvious effects
- Showing delivery cost earlier
- Adding prices where there were none
- Halving the page load time
- Adding reviews to product pages
- Removing a required field from checkout
Those produce effects large enough to see without sophisticated measurement. Fine-grained optimisation needs volume you probably do not have.
Watch for confounds
| Confound | Guard |
|---|---|
| Seasonality | Compare against the same period last year |
| A campaign running | Note what else changed |
| Traffic source shifting | Segment by source |
| Stock availability | Note anything that went out of stock |
Write down what you changed
A simple log — date, what changed, why, what happened — is worth more than any testing tool for a small store. Six months later it is the only record of what actually worked.
Most stores make dozens of changes a year and can recall almost none of them.
Frequently asked questions
How much traffic before split testing is viable?
Can we test more than one thing?
What if the result is ambiguous?
Should we use a testing tool?
Changing things and hoping?
A four-week baseline and a change log costs nothing and tells you what actually worked.