Tags: experimentation concept
Hypothesis Design
Date: 2026-08-16
A prediction plus a mechanism. Without the mechanism a win teaches you nothing you can reuse and a loss teaches you nothing at all — which is how programmes accumulate results without accumulating knowledge.
What it is
A hypothesis is a statement of what you expect to change, for whom, and why — specific enough that both outcomes are informative.
Compare:
NOT A HYPOTHESIS
"The new product page design will perform better."
A HYPOTHESIS
"Moving delivery information above the buy button will increase
add-to-cart rate for first-time visitors, because unresolved
delivery cost is the most common reason cited for abandoning
at this step in our exit surveys."
The second names a change, an audience, a metric, a direction and a reason. The first names an aspiration.
The four parts
| Part | Question | Why it matters |
|---|---|---|
| Observation | What did you see that prompted this? | Grounds it in evidence rather than preference. Analytics, replay, research |
| Mechanism | Why would this change behaviour? | The reusable part. Names the barrier being removed |
| Prediction | What moves, in which direction, by roughly how much? | Forces an MDE conversation before the build |
| Audience | Who does this apply to? | Pre-registers the segment, so finding it later isn’t post-hoc slicing |
A workable template, and it’s worth being formulaic about:
Because we observed [evidence], we believe that [change] will cause [effect] for [audience], because [mechanism]. We’ll know when we see [metric] move by at least [MDE].
Why the mechanism is the point
A result without a mechanism is a fact about one page. A result with one is a belief you can apply elsewhere.
WITHOUT MECHANISM
"Green button beat blue on the product page."
→ applies to: that button, on that page, that month
WITH MECHANISM
"Increasing contrast between the primary action and the surrounding
page increased add-to-cart, because the action wasn't being found."
→ applies to: every page with a primary action
→ generates: the next five tests
→ is falsifiable: test it on the basket page and see
In plain terms: the mechanism is what turns one result into a rule. Without it, you have to retest everything from scratch on every page, forever.
It also makes losses valuable. A losing test with a mechanism tells you the mechanism is wrong, which removes a whole family of future ideas. A losing test without one tells you only that this particular thing didn’t work.
Testing one thing
If the variant changes layout, copy and imagery together, a win tells you the bundle beat the control and nothing about which part did the work. That’s sometimes the right trade — bundled changes have larger effects and are cheaper to power (Minimum Detectable Effect) — but it must be a deliberate choice, and the hypothesis should say so.
The honest framing for a bundle: the mechanism has to cover the bundle. “Reducing overall page complexity” is a legitimate mechanism for changing six things at once. “The new design is better” is not.
Failure modes
- Hypothesis written after the test is built. The build encodes decisions the hypothesis then rationalises, and the MDE conversation never happens
- No falsifiable prediction. “Will improve the experience” cannot lose
- Mechanism assumed rather than observed. “Users are confused by X” with no research behind it is a guess with a paragraph attached — Usability Testing and Voice of Customer Data are where mechanisms come from
- Audience left implicit, then discovered in the results — The Multiple Comparisons Problem
- Predicted effect chosen to be achievable rather than to be worth having. The MDE is a commercial threshold, not a hope
- Not recorded. A hypothesis that isn’t in the Experiment Archive alongside its result can’t be revisited when a later test contradicts it — see Institutional Learning
Locked in writing before launch, this becomes Pre-Registration, which is what makes the resulting p-value mean anything.