The peeking problem

In a classical A/B test, you commit to a sample size up front, run to that sample, and check the result once. The maths assumes that single check. Every additional peek along the way gives randomness another chance to push the result across the alpha threshold, and the effective false-positive rate climbs with each look.

Stopping on day 5 or day 15 both give you a winner. Running to day 35 gives you the truth.

The intuition: at any given moment of a noisy test, the running result can swing above and below the threshold by chance. If you only check once at the end, you have a 5% chance of crossing at alpha = 0.05. If you check every day for 14 days, you have many independent chances to cross, and the effective alpha is much higher than 0.05 - often two or three times higher in practice. Evan Miller’s How Not To Run An A/B Test is the clearest demonstration of this and has been for years.

Why the maths punishes peeking

The classical statistical machinery assumes the sample size is fixed in advance. The “5% false-positive rate” is computed on the assumption that the analyst looks once. Peeking changes the procedure - now you stop whenever you cross threshold, which is mathematically equivalent to running many tests at the same alpha. Compounded across peeks, the family-wise false-positive rate rises.

The same effect shows up in multiple testing when you check many metrics or segments. Peeking is just multiple testing across time.

What actually happens in practice

Most CRO programmes peek constantly. The platform displays the running result, the team checks it daily, and “winners” get called the moment something crosses 95% confidence. The result: a high proportion of “significant” findings are false positives that wouldn’t replicate.

Worse, peeking interacts with regression to the mean. The points where you’d be tempted to stop early are the points where the running estimate is most inflated by noise. Stopping at those moments locks in the inflated estimate as the test result, even though the underlying effect is smaller.

The version nobody counts as peeking is killing a test early because it looks bad. It’s the identical problem pointing the other way - you’re still stopping on noise, you’re still inflating the error rate, and the cost is a real win abandoned in week one rather than a fake win shipped. Teams that would never call a winner at day three will kill a loser at day three without blinking.

How to handle it honestly

• Run to planned sample, look once. The simplest fix. Pick a sample size, run to it, then check.
• Use sequential testing methods that mathematically account for peeking. Works but requires platform support and the stopping rules are stricter than they look.
• Use Bayesian inference with proper priors. Bayesian methods have less theoretical peeking penalty but early stopping still biases the effect size upward. The decision rule needs setting before the test, not discovered during it.

The trap in all three is assuming your platform has handled it. Nearly all of them display a running result, and displaying a running result is not the same as applying a sequential correction. “Continuous monitoring” in a feature list usually means the chart updates.