Hypothesis formulation

A test without a hypothesis is just a guess with extra steps. The hypothesis is what you’re claiming and how you’d know if you’re wrong - everything else (sample size, primary metric, duration) flows out of it.

The classical setup has two hypotheses:

• Null (H0) - the boring one. There’s no difference between control and treatment. The change you’re testing did nothing.
• Alternative (H1) - the interesting one. There is a difference. The change did something.

You don’t “prove” H1. You collect enough evidence to reject H0, which is a different thing. This distinction matters because it’s why a “non-significant” test isn’t a negative result, it’s an inconclusive one. Failure to reject H0 doesn’t mean H0 is true, it means you haven’t seen enough to dismiss it. Easy to forget when stakeholders want a clean yes/no.

Directional vs non-directional

A non-directional (two-tailed) hypothesis just says “different” - the new design will change the conversion rate, either up or down. A directional (one-tailed) hypothesis specifies which way - the new design will increase conversion. Two-tailed is the safe default and what most tools assume. One-tailed gives you a bit more statistical power for the same sample size but at the cost of being blind to the opposite direction. If your one-tailed bet was wrong and the variant actually tanked conversion, the test won’t tell you.

In practice almost everyone should run two-tailed. Even if you think you know which way the change will go, “it tanked” is a result you want to detect.

What a good hypothesis actually looks like

This is the statistical framing - the CRO-flavoured version (testability, mechanism, three-part structure) is in strong hypotheses. A real hypothesis isn’t “let’s change the button colour”. It has three parts:

1. The change you’re making
2. The mechanism you think it’ll trigger (the why - usually a behavioural or UX claim)
3. The metric you expect to move, and roughly by how much

So: “Replacing the generic CTA copy with benefit-led copy on the PDP will increase add-to-cart rate by 5%+, because users currently bounce when the value isn’t clear above the fold.”

If your hypothesis can’t be falsified by the test, it’s not really a hypothesis, it’s a project. “Improve the checkout” isn’t testable. “Removing the discount code field on step 1 of checkout will increase completion rate by 3%+, because the field draws people off-site to hunt for codes” is.

Vagueness about the expected effect size is the part people skip, and it’s the part the sample size maths runs on. You need a number even if it’s a guess, because “lift conversion” gives the calculator nothing to work with.

HARKing

Hypothesising after the results are known. Run the test, look at the data, find the segment where something worked, then write the hypothesis around it and present it as though it came first.

It’s the most common way a programme manufactures wins that never replicate, and it rarely feels like cheating from the inside - it feels like being responsive to the data. The defence is pre-registration: write the claim down before the test runs, and anything you find afterwards gets labelled exploratory and re-tested rather than shipped.