Revenue per visitor

Revenue per visitor is total revenue divided by visitors. It’s the metric every stakeholder wants as the primary metric, because it’s the one that maps directly to money, and it’s the metric most likely to leave you with an inconclusive test.

The problem is the distribution. Most visitors contribute exactly zero. A small group contributes a normal order. A tiny group contributes an enormous one. That shape - zero-inflated with a long right tail - has huge variance relative to its mean, and variance is what your test has to fight through.

Compare it to conversion rate. A binary outcome per visitor has variance bounded by the maths of a coin flip. Revenue per visitor has no such ceiling: one £4,000 order lands in a variant and drags its mean around. Detecting the same relative effect on RPV typically needs several times the sample that conversion rate needs, sometimes much more depending on how heavy your tail is. On a mid-size store this is the difference between a three-week test and one you cannot run at all.

Decompose before you commit

RPV is the product of two things:

RPV = conversion rate × average order value

Most interventions plausibly move one of them. A clearer PDP layout, a trust badge, a simplified checkout - these act on conversion rate. If the mechanism in your hypothesis only touches conversion rate, test conversion rate. You get the answer faster, on a metric with tolerable variance, and the revenue implication follows arithmetically.

Reaching for RPV when your change can’t plausibly affect AOV buys you nothing except a wider confidence interval.

When RPV genuinely is the right primary

The exception matters: any change that trades one component against the other. Then conversion rate alone will actively mislead you.

• Free shipping thresholds. Raising the bar suppresses conversion rate and lifts AOV. Reading conversion rate alone calls this a loss.
• Bundles and upsells. Same trade in the other direction.
• Price changes. Obviously.
• Minimum order quantities, subscription-vs-one-off defaults, tier restructures.

For these, RPV or total revenue is the only honest primary. Accept the sample cost or don’t run the test. The alternative - running on conversion rate and eyeballing AOV separately - is how teams ship threshold changes that quietly reduce revenue.

A reasonable middle path is conversion rate as primary with revenue as a guardrail, which works when you’re confident the change is one-directional and you just want an alarm if it isn’t.

Taming the variance

Three things help, in rough order of how much:

• Winsorise. Cap order values at a high percentile, usually the 99th, applied identically to both variants and decided before you look at the data. This is the standard treatment and it’s covered properly in missing data and outliers. You lose a little accuracy in the tail and buy back a lot of precision.
• CUPED on pre-period spend. Variance reduction using each user’s historical revenue works unusually well here, because past spend genuinely predicts future spend. On a site with a strong returning-customer base this can be worth more than any other single adjustment. It needs user identity across the pre-period, so it’s off the table for anonymous-heavy traffic.
• Use the right variance formula. RPV is a ratio metric, so if you randomise on users and measure per session, the naive variance is wrong and your significance is overstated.

Log transformation gets suggested a lot and doesn’t work cleanly here, because you can’t take the log of the zeros that make up most of your data. Winsorising is the practical answer.

What I’d tell a stakeholder

The uncomfortable conversation is that “we should just measure revenue, that’s what matters” is correct in principle and often unaffordable in practice. The metric you can actually resolve in a sensible timeframe is usually one step upstream.

That’s not a reason to hide the revenue number. Report it, report its interval, and be explicit that the interval is wide. A test showing +4% conversion rate with a revenue estimate ranging from -2% to +9% has told you something real about behaviour and almost nothing precise about money, and saying so is more useful than picking whichever of the two numbers supports the decision you’d already made.