Effect size
Effect size is how big the difference between control and variant actually is. Significance tells you whether to believe there’s a difference at all. Effect size tells you whether the difference is worth having, and the two are almost completely decoupled.
A vast sample makes a 0.05% lift significant. A small one leaves a 20% lift inconclusive. Neither tells you anything about the other, which is why a result reported as “significant” and nothing else is a result you can’t act on.
Relative and absolute
Section titled “Relative and absolute”This is where most of the practical damage happens, because “a 10% lift” is genuinely ambiguous and both readings are common.
- Absolute is the difference in the metric’s own units. A conversion rate moving from 2.0% to 2.2% is a rise of 0.2 percentage points.
- Relative is that difference as a proportion of the baseline. The same move is a 10% lift.
CRO conventionally reports relative, because it’s comparable across metrics and baselines - a 10% lift means the same kind of thing whether you’re measuring add-to-cart at 12% or purchase at 2%. Most testing platforms default to it.
The cost of that convention is that relative figures detach from the business at low baselines. A 10% relative lift on a 0.4% conversion rate is 0.04 percentage points, which sounds impressive in a readout and is four extra conversions per ten thousand visitors. Both numbers are true. Only one of them tells the finance director anything.
The rule I stick to is to state which one you mean every single time, in writing, including in the MDE you set before the test. A test brief that says “we want to detect 10%” without saying which has already lost the argument it’s going to have at the end.
The estimate is not the effect
Section titled “The estimate is not the effect”The number the test gives you is a point estimate, and it comes with a range around it that matters at least as much. A +4% observed effect with an interval spanning +0.4% to +7.6% is not a 4% result, it’s a result somewhere between trivial and substantial.
Worse, the point estimate is biased upward whenever the test is underpowered, because a weak test can only reach significance by overshooting. That’s the M-type error, and it’s why the lift you report and the lift you eventually realise so often diverge. Assume the true effect is smaller than the one you measured, and build the business case at the bottom of the interval rather than the middle.
Standardised effect sizes
Section titled “Standardised effect sizes”Academic work often reports effect sizes standardised by the variability of the data, most commonly Cohen’s d, so that results from different studies can be compared on one scale.
CRO almost never needs this, and it’s worth knowing why rather than just skipping it. Standardisation exists to make incommensurable things comparable, and our outcomes are already in units everyone understands - conversions, orders, pounds. Converting a conversion rate into standard deviations throws away the only property that made it useful.