Multivariate testing

A multivariate test (MVT) tests multiple variables simultaneously, with one variant per combination. Two headlines × two CTAs × two images = 8 variants. The test measures both the effect of each variable individually and the interaction effects between them.

Sounds more powerful than A/B testing, which is why teams reach for it. In practice it usually shouldn’t be the first choice.

Why MVT is sample-hungry

Each variant needs its own sample size to produce a usable estimate. 8 variants means you need roughly 8x the traffic of a simple A/B test for the same per-variant power. Most CRO programmes don’t have the traffic to support that on realistic effect sizes.

The deeper issue is that MVT is testing multiple hypotheses at once. Each variable comparison is a separate test, and the multiple testing problem applies. Without correction, you’ll see “significant” individual-variable effects that wouldn’t replicate.

When MVT makes sense

Despite the sample cost, MVT is occasionally the right tool:

• Interaction effects matter. If you suspect headline A only works with image B, A/B tests won’t surface that - you need the full grid to see the interaction.
• You have abundant traffic. Large-scale product surfaces (Booking, Amazon, top-of-funnel ads) can run MVTs that smaller programmes can’t.
• The variables are cheap to vary. Ad creative testing, where you can spin up many combinations at low engineering cost, is a natural fit.
• You’re doing fractional factorial design. Testing 4 variables at 2 levels each (16 combinations) by sampling 8 strategically chosen combinations is a real technique. Almost no commercial CRO platforms support this properly.

When it doesn’t

The default situation in most CRO programmes:

• Traffic is limited. Splitting it 8 ways gives you nothing detectable per cell.
• The hypothesis is about a single change. Tying it into an MVT just dilutes the signal.
• Stakeholders want to know which specific change won. MVT outputs are harder to communicate than “B beat A by 7%”.

For most CRO work, sequential A/B tests beat MVT. Test the highest-leverage variable first, ship the winner, then test the next. You’ll learn faster and with less traffic than running everything at once.