Programme meta-analysis

After fifty tests you have something more valuable than fifty results: you have a sample of what happens when your team changes things. Treating that archive as a dataset in its own right is the step that separates a team that runs tests from an experimentation programme, and it’s the analysis I’ve seen skipped most consistently.

Win rate as a diagnostic

Count what proportion of your tests produced a real, shipped win. Across the industry the number tends to land somewhere in the 10-30% range, moving with programme maturity and with how bold the changes are.

The value isn’t in benchmarking. It’s in what an anomalous number tells you about your process:

• Win rate far above 30%. The likely explanation is not that you’re brilliant. It’s that you’re calling noise as wins - underpowered tests, peeking, or a decision rule that ships anything not obviously negative. Base rates explain why that happens so readily.
• Win rate near zero across many tests. Either the changes are too timid to clear your MDE, or the tests are chronically underpowered and everything comes back inconclusive. Those two have different fixes, and the effect size distribution tells you which you’re in.

A programme that knows its own win rate can forecast. If you win one test in six and the average win is worth £40k annualised, you can put a number on what twenty more tests is worth, which is how experimentation budget gets defended.

The winner’s curse

Add up every lift your programme has reported this year. Compare it to what actually happened to the business. The second number is always smaller, often dramatically, and this is the single most useful thing meta-analysis surfaces.

The mechanism is selection. You only ship the variants that looked good, and looking good is a combination of being genuinely good and getting lucky. Conditioning on “it crossed the threshold” selects for the lucky ones, so the observed effect of shipped winners systematically overstates their true effect. It’s the same machinery as regression to the mean, operating at the level of your portfolio rather than a single test.

The bias is worse when tests are underpowered, because a barely-powered test can only reach significance by overestimating. A programme running lots of small-sample tests will produce a lot of impressive, non-replicating wins.

Two practical responses. Discount reported lifts when you build the business case - assume the true effect is meaningfully below the point estimate rather than at it. And validate the portfolio directly with a long-running global holdout, which is the only clean measurement of whether the accumulated wins actually stack.

The effect size distribution

More useful than the win rate, and almost never produced: group your results by type of change and look at the distribution of effects in each group.

You find out which categories of intervention actually move your metrics on your site. Not what a conference talk says works, or what a competitor did. Copy and offer changes might dominate; layout might do nothing; friction removal in checkout might be reliable but small. Whatever the shape is, it’s evidence generated by your own traffic, and it should be re-weighting your prioritisation directly.

This is also the honest answer to “why should we keep funding this”. A category-level effect distribution is a map of where the remaining upside is.

The archive has to exist first

None of this is possible retrospectively if the records are a folder of screenshots. The minimum viable per-test record:

• Hypothesis, and the surface it touched
• Primary metric, MDE, and planned sample
• Result: effect size with interval, not just the verdict
• The decision taken, and whether that matched the result
• One line on what was learned

Record the losers and the flat ones with the same care as the wins. The archive’s value is mostly in them - they’re the majority of the data, they’re what stops the same idea being re-tested every eighteen months as people rotate, and a register where two thirds of entries say “no effect” is the most credible artefact an experimentation team can produce.

The habit worth keeping alongside it is scepticism toward your own standout numbers. Any result that looks remarkable is more likely to be an instrumentation fault than a discovery, and the archive is where that pattern becomes visible - because the third time a spectacular win fails to replicate, you stop believing the spectacular wins.