Building an experimentation programme

A programme is what you build when you want experimentation to outlast the people who set it up. A few one-off A/B tests aren’t a programme. A pattern of recurring tests, shared learnings, and infrastructure that supports both is.

The components, roughly in order of importance:

Infrastructure

Without the right tooling, every test costs an order of magnitude more than it should. The minimum:

• A testing platform that handles randomisation, assignment consistency, and basic statistical analysis correctly. Many “platforms” don’t.
• Analytics that can join experiment data with downstream metrics (LTV, retention, support tickets).
• A way to surface running tests to the rest of the org so nobody accidentally launches conflicting tests on the same surface.
• Documented pre-registration and analysis templates.

Process

The recurring rhythms that produce tests:

• Hypothesis backlog. A live document of testable ideas, ranked by expected value.
• Weekly or fortnightly test launches. Predictable cadence beats sporadic bursts.
• Standing analysis review. A regular meeting where finished tests get interpreted and ship / no-ship decisions get made together.
• Win and loss documentation. Both shipped winners and failed tests written up so the learning compounds across the team. Once the archive is big enough it becomes analysable in its own right - see programme meta-analysis.

Culture

The hardest part. The cultural shifts:

• Failure is data, not embarrassment. Most tests fail. A programme where failed tests get hidden or rationalised will run fewer tests and learn less.
• Stakeholders accept that the test decides. No post-hoc “I think we should ship it anyway”. If that conversation happens, the programme isn’t a programme yet.
• The HiPPO problem is checked. Senior stakeholders don’t override results because they have a hunch.
• Hypotheses come from everywhere. Customer support, paid media team, product, design - not just CRO specialists. The best hypotheses often come from people who see customer behaviour daily.

Common stages of programme maturity

A loose progression:

1. One-offs. Occasional A/B tests when someone has a strong opinion to settle. No real programme.
2. Habitual. Tests run regularly but each is bespoke. Inconsistent rigour, slow analysis.
3. Templated. Standard pre-registration, defined metric set, repeatable analysis. Velocity climbs.
4. Compounding. Multiple parallel tests, holdouts for long-run measurement, shared learning across teams. The programme is a source of ongoing competitive advantage.

Most programmes plateau at stage 2 or 3. Getting to stage 4 requires sustained investment in infrastructure and culture, often from leadership.

The two halves fail independently and neither is sufficient. Great tooling with no cultural muscle produces results nobody trusts and stakeholders overrule. Great culture with no tooling produces a willing team for whom every test takes a month. Buying a platform is the usual attempt to skip the harder half, and it doesn’t work - platforms enable programmes, they don’t generate them.

Once it’s running, resist measuring it by test count. Volume is the easiest number to report and the easiest to game: fifty button-colour tests learn less than ten well-chosen ones, and a programme optimised for its own throughput metric drifts toward small safe tests that clear quickly.