Pre-registration

Pre-registration is committing to the test design before the data lands. Hypothesis, primary metric, sample size, alpha, stopping rule - all documented before assignment starts. It’s the single most reliable hygiene practice for keeping a test honest, and it’s almost universally skipped in commercial CRO.

The reason it matters: without pre-registration, every analysis choice can be made after looking at the data. The “primary metric” shifts to whichever metric won. The “exclusion criteria” are tightened until the segment with the lift shows up. The stopping point becomes the day the result looked best. None of this is intentional fraud, it’s just what humans do when nothing’s pinning them down.

What a pre-registration actually contains

It doesn’t have to be elaborate. A one-page doc with:

• The hypothesis - what you’re testing and what mechanism you think will move the metric
• The primary metric - the single measure you’ll use to call the test
• Secondary metrics and guardrails - the things you’ll also watch but won’t decide on
• Expected effect size (MDE)
• Sample size and how long the test runs
• Alpha and power
• Stopping rule - what triggers the analysis (planned sample reached, time elapsed, etc.)
• Pre-specified segments you’ll analyse, if any

The discipline isn’t in writing the document, it’s in committing to it before the data starts arriving and not editing it post-hoc.

How this prevents the common failure modes

Most of the ways tests produce fake winners get cut off by pre-registration:

• Metric shopping - you can’t switch primary metric after the fact if the doc says what it is.
• Segment fishing - segments you analyse have to be in the doc. Anything else is exploratory and labelled as such.
• Peeking and stopping early - the stopping rule is pre-committed, so the temptation evaporates.
• HARKing (hypothesising after results are known) - the hypothesis is locked, so post-hoc rationalisations can’t masquerade as predictions.

Where it usually breaks down

The doc gets written but stakeholders renegotiate it once results are in. “Actually let’s call it on conversion, not on revenue” after seeing revenue is flat. Pre-registration only works if there’s a culture of treating the doc as binding. In teams without that culture, the doc becomes a ceremony.

The fix is to make analysis automatic. If the primary metric and stopping rule are written into the test config, the analysis runs on schedule and the conclusion follows without human discretion. That removes the human renegotiation step.