Tags: commerce experimentation concept
Price Testing
Date: 2026-08-16
Experimenting on price is harder than it looks — legally, technically and statistically. Which is why most price decisions are made on judgement, and why the alternatives to a live A/B test matter more here than anywhere else.
What it is
Price testing is measuring the effect of a price change, ideally causally.
The obvious approach — show different prices to different users simultaneously — is the one with the most problems.
Why a live price A/B test is awkward
Fairness and disclosure. Two customers seeing different prices for the same item at the same moment is a reputational risk, and it becomes a real problem the moment they compare notes — which they do, publicly.
Legal and platform constraints. UK consumer protection rules govern price presentation and comparison claims, and some marketplace and payment terms restrict differential pricing. [CHECK: current CMA position and your platform’s terms before running any live price test.]
Technical. Price must be consistent across the product page, basket, checkout, emails, feeds and the order system. A variant price that doesn’t propagate to a remarketing feed or an abandoned-basket email produces a mismatch the customer sees.
Statistical. Price affects revenue per visitor — a heavy-tailed metric needing roughly 2.4× the traffic of conversion rate — and the effects are often small. Most sites cannot power a price test on the metric that matters — Metric Sensitivity, Skewed and Heavy-Tailed Distributions.
The alternatives, in order of practicality
1. Sequential price changes with proper controls. Change the price, measure before and after, adjust for seasonality and mix. Confounded, but honest about being so — and it’s how most retail pricing is actually evaluated. Strengthen it with a comparison group of unchanged products.
2. Geographic testing. Different prices in different regions, where that’s defensible commercially. The unit of analysis is the region, so power is limited — Geo Holdout Tests.
3. Test the presentation, not the price. Formatting, prominence, framing, bundling display, unit-price display. No fairness problem, no legal exposure, and often where the effect is anyway — Psychological Pricing.
4. Test on new products. No reference price exists, so there’s no was/now issue and no existing customer expectation to violate.
5. Break-even analysis instead of a test. Compute the volume change that would make a price move worthwhile. Often the answer is so permissive — you can lose 25% of volume on a 10% rise — that a test isn’t needed to decide — Price Elasticity, Break-Even Analysis.
Option 5 is underused. For price rises in particular, the break-even volume loss is usually large enough that the decision is clear without measurement.
If you do run one
- Randomise by user and make it sticky. A customer who sees £50 then £55 on a return visit will notice, and that’s worse than either price
- Honour the lower price where a customer has seen it. Cheap insurance against a complaint
- Propagate the variant everywhere — feeds, emails, basket, checkout
- Judge on contribution per visitor, not conversion rate. A lower price will convert better and may contribute less, which is the entire point of the test — Contribution Margin
- Guardrail on margin and on mix. A price test that shifts customers to lower-margin lines can win on revenue and lose on contribution — Guardrail Metrics, Basket Composition
- Run it on a bounded set of products, not site-wide
- Have it reviewed by whoever owns legal risk, before launch rather than after
The realistic position
For most retailers, price is decided by margin requirements, competitive position and break-even arithmetic — not by experiment. That’s not a failure of rigour; it’s a recognition that the test is expensive, slow, risky and frequently underpowered.
Where experimentation earns its place in pricing is on the presentation and structure — thresholds, bundles, tiers, framing — all of which are testable without anyone paying a different price for the same thing.