Experiment velocity

The case for velocity: the value of a CRO programme is dominated by how many tests you run, not by how perfect each one is. A programme that ships 50 tests a year at 80% rigour will compound past a programme that ships 12 tests at 95% rigour within a year or two.

The maths is straightforward. If 1 in 5 tests is a real winner (a reasonable hit rate for a mature programme), and each winning test lifts revenue 2-3%, then 50 tests gives you 10 winners and compounded gains of ~25% over the year. 12 tests gives you 2-3 winners and 5-7% compounded gains. The 50-test programme wins by miles.

Why velocity is hard to maintain

Each test has fixed overhead: hypothesis, instrumentation, design, build, QA, run, analysis, write-up. Most of that doesn’t scale down for “smaller” tests. Teams new to experimentation often spend 4-6 weeks per test, which caps annual velocity at maybe 10 tests.

The mature-programme overhead is closer to a week per test. The compression comes from:

• Hypothesis templates and pre-registration formats. Don’t rewrite the analysis plan structure every time.
• Test tooling that’s actually fast. Building a variant should take hours, not days. Visual editors for simple changes, code-based for complex ones, both well-supported. Shipping each variant as a vertical slice - flag, data, UI and tracking together - is what keeps that number in hours.
• Centralised analysis with pre-defined dashboards. The analyst’s job is interpretation, not data wrangling.
• Clear ship / no-ship rules pre-committed. No two-week post-test stakeholder negotiation.
• Concurrent tests on non-overlapping surfaces. Run 4-5 tests in parallel instead of in series. The fear of test collision costs more velocity than collision itself ever costs accuracy.

The trade-off with rigour

The velocity argument isn’t “be sloppy”. It’s “the marginal value of an extra week of rigour on each test is usually less than the marginal value of an extra test in the queue”. Within reasonable limits, faster wins.

The reasonable-limits part matters. Velocity that produces a steady stream of underpowered noise isn’t progress. The minimum viable rigour:

• Sample size calculated, not guessed.
• Pre-registration of primary metric and stopping rule.
• Don’t peek and stop early.
• Check guardrails before shipping.

Past those, additional rigour usually has lower ROI than another test in flight.

What high velocity breaks

• The win rate stops meaning anything. A programme judged on throughput starts shipping everything that crosses alpha, and a good share of those are false positives that quietly revert. The faster you ship, the more the reported wins need checking against a holdout - the accumulated lift and the actual revenue line diverge fastest in exactly the programmes that are proudest of their velocity.
• Headcount stops helping. Doubling the team rarely doubles the test count, because the binding constraint is almost always shared infrastructure rather than people. If every test queues behind the same instrumentation work or the same release train, more hands just lengthen the queue. That’s usually technical debt presenting as a staffing problem.