Base rates and false winners
Picture the next 100 ideas in your test backlog, and imagine building every one of them properly. Some genuinely move the metric. Most do nothing. The share that genuinely move it is your base rate.
If 20 of those 100 would really work, your base rate is 20%. It’s a property of the ideas going in, not of the tests you run on them, and it’s set by how good your research and prioritisation are.
It matters because it decides how much a significant result is worth believing. The same p-value means something very different coming out of a backlog where most ideas are sound than one where almost nothing is.
Why you can’t just look it up
Section titled “Why you can’t just look it up”Here’s the awkward part, and it’s the reason the concept feels slippery: you can never observe the base rate directly. To know which ideas really work you’d have to already know the answer, which is what testing is for.
It’s also not the same thing as your win rate, and conflating the two is the usual mistake. Your win rate is the share of tests that came back significant. That number is inflated by false positives and deflated by real effects your tests were too small to detect. It’s downstream of the base rate, not equal to it.
So you estimate it instead, and the estimate comes from three numbers you do have.
How to estimate yours
Section titled “How to estimate yours”Every test you run ends up significant either because it found something real, or because it got lucky. Written out:
observed win rate = (base rate × power) + (1 − base rate) × alpha
Rearrange for the thing you want:
base rate = (observed win rate − alpha) ÷ (power − alpha)
Three steps to use it:
- Count. How many tests did you run last year, and how many came back significant? That ratio is your observed win rate.
- Estimate your power. Not the 80% on the calculator, but what you actually ran at. If your tests routinely stop short of the planned sample, assume 50% or lower.
- Divide. That gives your implied base rate.
The same arithmetic run at other numbers, to give a feel for the shape:
| Tests | Significant | Observed | Power | Implied base rate | False share of wins |
|---|---|---|---|---|---|
| 20 | 2 | 10% | 50% | 11% | 44% |
| 40 | 6 | 15% | 80% | 13% | 29% |
| 18 | 4 | 22% | 50% | 38% | 14% |
| 20 | 6 | 30% | 50% | 56% | 7% |
| 20 | 8 | 40% | 50% | 78% | impossible |
That last row is the useful one. A 78% base rate would mean nearly four in five of your ideas are genuinely good, which nobody achieves. When the formula returns something implausible, your inputs are wrong - and the usual culprit is that alpha isn’t really 0.05 because someone has been peeking.
Where you use it
Section titled “Where you use it”Deciding how much to believe one result. A significant test from a well-researched backlog is worth more than the same p-value from a backlog of whims. Same test, same statistics, different amount of trust.
Diagnosing your win rate. Work the formula backwards. If your observed win rate implies a base rate above about 50%, something is wrong with the testing rather than brilliant about the ideas.
Deciding what to re-test. A result that contradicts your expectation had a low base rate going in, so it’s likelier to be false. That’s the one worth repeating before you build on it.
Setting alpha. For a change that’s expensive to build or hard to reverse, dropping to 0.01 cuts the false-positive count fivefold while leaving true positives nearly untouched.
Making the case for research. This is the real payoff. Everything above improves if the base rate rises, and the only way to raise it is better ideas going in - which is what voice of customer research and honest prioritisation actually buy you.
What it explains
Section titled “What it explains”Several things elsewhere on this site have one root here.
A win rate far above 30% isn’t a brilliant team, it’s a threshold being crossed by things that aren’t real. Shipped lifts that never appear in the revenue line are the same arithmetic seen from the finance side. And the winner’s curse is what you get when the false positives that survive are, by construction, the ones that looked best.
It also reframes power. Underpowering doesn’t only make you miss real effects, which is the standard complaint. It poisons the wins you do get, because low power cuts the true-positive count while leaving false positives untouched. The ratio between them is what degrades.
The tension with velocity
Section titled “The tension with velocity”This sits awkwardly against the argument for experiment velocity, and the tension is real rather than apparent.
Velocity says run more tests, because volume compounds. Base rates say a programme with a poor backlog produces mostly false winners, and running more of them produces more false winners rather than more knowledge.
Both are right. Velocity compounds only while the base rate holds up, and doubling throughput by working further down a ranked list lowers the base rate as it raises the count. Volume and prioritisation aren’t alternatives - velocity is worth having, and worth having on a queue that’s been filtered first.