Test collision and interaction effects

Run more than a handful of experiments at once and users start appearing in several simultaneously. Someone is in the PDP test, the shipping-messaging test, the email capture test and the search test, all at the same time. The natural worry is that the results contaminate each other.

The reassuring answer first, because it’s the correct default: if assignment to each test is independent, the other experiments are just noise, and that noise is distributed evenly across your control and treatment. Randomisation is doing exactly the job it exists to do. Your effect estimate stays unbiased. You lose a small amount of precision to the extra variance and that’s the whole cost.

Independent assignment is not automatic though. It requires each experiment to hash users with its own distinct salt or seed. Two experiments sharing a salt will bucket every user identically, so treatment in one is treatment in the other, and the two are now perfectly confounded. This is a real and quiet bug, worth checking once in whatever platform you use rather than assuming.

Where collision actually bites

Three situations where the default reassurance stops applying.

Same surface, same element. Two tests both modifying the add-to-cart area. Beyond statistics this is a build problem: the variants can render on top of each other, produce a layout nobody designed, or one silently overrides the other. The result isn’t biased so much as meaningless, because you no longer know what was on the page.

One test changes who enters the other. A landing page test that lifts click-through pushes a different mix of visitors into a checkout test downstream. The checkout test’s population composition now differs between its own variants, correlated with the upstream assignment. This is the genuinely dangerous case, and it’s the reason funnel-stage tests running concurrently on the same journey deserve more care than tests on unrelated surfaces.

True interaction effects. Where the combination does something neither change does alone - two separate urgency mechanics that are fine individually and read as desperate together. These exist. They’re also rarer and smaller than the fear of them suggests, and if you’re worried about a specific pair, that pair is a case for multivariate testing, which is designed to measure interaction directly.

Mitigations, cheapest first

• Orthogonal assignment. Independent salt per experiment. Free, do it always, verify it once.
• QA the overlaps that matter. For tests on the same page, actually view the variant combinations. This catches the render conflicts, which is the majority of the real damage.
• Mutual exclusion groups. Carve traffic so conflicting tests can never overlap. Effective and expensive: every excluded test is competing for a slice of the same traffic pool, so your concurrent capacity drops and every test takes longer.
• Post-hoc interaction checks. Test for an interaction between concurrent experiments after the fact. Available, and be careful - you’re now running a lot of comparisons, so expect false alarms unless you correct for multiple testing.

The opinion

Most teams over-correct here. Fear of collision leads to mutual exclusion as the default, mutual exclusion starves every test of traffic, and a programme that could have run twelve concurrent experiments runs three. The cost to experiment velocity is large, immediate and certain. The statistical risk being avoided is small, occasional and mostly theoretical.

Reserve exclusion for tests on the same surface or the same funnel step. Let everything else run orthogonally and accept the small precision cost. If you’re a programme that’s genuinely running enough tests for collision to be a live concern, you’re already in the top decile of experimentation maturity, and the answer to that problem is more traffic or better prioritisation rather than fewer tests.

The failure mode I’d actually watch for isn’t statistical at all. It’s a user landing on a page assembled from four concurrent variants that nobody has ever looked at together, having an incoherent experience, and none of the four tests being able to see it.