The obvious design is the risky one
The cleanest experiment shows half of visitors one price and half another. It also means two customers can buy the same item at different prices on the same day, and one of them can find out.
Customers react strongly to this, more strongly than to a price rise applied to everyone. It reads as unfairness rather than as commerce, and the reaction is not proportionate to the size of the difference.
There may also be legal constraints depending on jurisdiction, sector and whether the variation correlates with protected characteristics. Take advice before running one.
Designs with less exposure
| Design | How it works | Trade-off |
|---|---|---|
| Time-based | Price A one period, price B the next | Confounded with seasonality and events |
| Region or store | Different prices in different locations | Regions differ; needs matched comparison |
| Product-based | Vary price across similar products | Products are not identical |
| New products only | Test at launch where no prior price exists | Limited to launches |
| Promotional depth | Vary discount rather than base price | Measures promotion response, not base elasticity |
Time-based is the most common and needs the most care, because anything else that changed between periods gets attributed to price. Alternating in short blocks rather than one long switch helps considerably.
What you are actually measuring
Units sold is not the objective. A price cut that increases units while reducing total profit has made things worse, and a test measuring volume alone will conclude the opposite.
- Gross profit, not revenue and not units
- Effects on other products - customers trading up or down within the range
- Whether demand was brought forward rather than created
- Returns rate, which sometimes shifts with price
- Any change in customer mix, which can affect future value
Cross-product effects are routinely missed. A successful price cut that simply moved customers from a higher-margin alternative is not a success.
Elasticity is not a constant
Businesses sometimes measure elasticity once and treat it as a property of the product. It moves with season, competitor pricing, customer mix and how the price is presented.
Treat any estimate as valid for the conditions it was measured in, and re-test periodically. A figure from a test run two years ago during a promotion is not a reliable basis for a decision today.
Be prepared to explain it
The practical test for any price experiment: could you explain this design to a customer who noticed, and would it sound reasonable?
Time-based and regional variation pass that test - prices change, and they differ by location, and customers understand both. Simultaneous personalised pricing generally does not, which is a reasonable signal about whether to run it.
If you would not want to explain the design to a customer, do not run it.