Variance reduction

In a normal test, the variance in your metric (how much it bounces around naturally) limits how small an effect you can detect at a given sample size. Variance reduction is the family of methods that strips out some of that noise without throwing away data, giving you more power per visitor. Run the same test with half the noise and you’ve effectively doubled your traffic.

CUPED

CUPED (Controlled Pre-Experiment Data) is the big one. The intuition: a chunk of why a user converts during your test has nothing to do with the variant. They were already a high-converting type of user. If you can measure their pre-test behaviour, you can subtract out the baseline difference and only attribute what’s left to the variant.

Concretely you regress your outcome metric on a pre-test covariate (often the same metric measured before the test started), compute the residual, and use that as your “adjusted outcome”. Microsoft reported 50% variance reduction on their experimentation platform, which roughly doubles your effective traffic - the original CUPED paper is worth reading if you’re implementing it rather than buying it.

CUPED needs pre-test data on the same users, which is fine for logged-in product analytics but harder for anonymous web traffic. If your visitors are mostly one-shot anonymous sessions (typical for Shopify) you can’t easily apply CUPED to most metrics.

The word doing the work in all of that is pre. Adjust on a covariate measured during or after the test and you’ve broken randomisation - the covariate is now downstream of the variant, so you’re partly subtracting out the effect you’re trying to measure. This is the one way to genuinely misuse CUPED and it’s an easy mistake to make when the pre-period data is awkward to get and the in-test data is right there.

Stratified sampling

Instead of randomising all traffic into one big pool, you stratify (split into bands by some characteristic - device, source, country, customer tier) and randomise within each stratum. This guarantees the variants are balanced on whatever you stratified by, removing one source of variance from the comparison.

Less commonly used in CRO platforms but baked into most academic and pharma trial design. The win is biggest when the stratification variable is strongly predictive of the outcome. Device type and traffic source usually qualify.

Why this matters in CRO

Most Shopify and small SaaS programmes are underpowered for the effect sizes they’re testing. Variance reduction is one of the few ways to claw back power without raising more traffic. If you’re running on a platform that supports CUPED (Statsig, Eppo, some custom internal stacks) it’s basically free statistical power. Turn it on.

It gets resisted as somehow cheating, which it isn’t - you’re not changing what you’re estimating, only measuring it more precisely. The honest limit is scale rather than legitimacy: it won’t turn a 5,000-session test into a usable result when you needed 50,000. It shifts the margin, it doesn’t rescue a test that was never viable.