Interference and cluster randomisation

Every A/B test rests on an assumption nobody says out loud: my outcome depends on my assignment and nothing else. Your bucket doesn’t affect my behaviour. This has a formal name, SUTVA (stable unit treatment value assumption), and for most website testing it holds well enough to ignore.

When it doesn’t hold, the test doesn’t just get noisier. It gets biased in a specific and flattering direction: it reports a lift that’s actually a transfer from control to treatment. Nothing was created. The measured effect will not replicate when you ship to everyone, because at 100% there is no control group left to take from.

Where the assumption breaks

The tell is a shared, finite resource sitting between the variants.

• Shared inventory. Treatment users see a better product module, buy the last of a limited line, and control users hit an out-of-stock. Treatment’s conversion rate rises partly because control’s fell.
• Shared sales capacity. This is the one that bites leadgen hardest and gets spotted least. A variant that lifts form fills by 30% pushes 30% more leads into a sales team with fixed headcount. Response times stretch, and because the queue is shared, control leads get slower follow-up too. Your speed-to-lead SLA degrades for everyone while the form-fill metric looks great.
• Shared ad budget. Two landing page variants feeding the same campaign compete in the same auction for the same daily budget. Better-performing variant takes more impressions, starves the other.
• Social and referral paths. A variant that encourages sharing sends traffic to control users. Word of mouth, referral codes, and anything with a viral loop leak across the boundary by design.
• Two-sided systems. Marketplaces, delivery, booking. Treatment buyers consuming supply that control buyers wanted is the textbook case.

The common shape: your treatment effect propagates through something both groups touch. Ordinary web CRO on an infinite-inventory site is mostly safe. The moment scarcity is real, it isn’t.

Cluster randomisation

The fix is to randomise a unit big enough to contain the spillover. Instead of assigning individual users, assign whole groups: geographies, stores, accounts, households, cities, days of week for a location.

The cost is severe and worth stating plainly. Your effective sample size collapses from the number of users to roughly the number of clusters. Fifty thousand users split across twelve cities is a twelve-unit experiment, not a fifty-thousand-unit one. Users inside a cluster are correlated with each other, which is exactly why clustering works and exactly why it destroys your power. The correlation is measurable (intra-cluster correlation) and the more alike users inside a cluster are, the worse the trade.

Practically this means cluster-randomised tests can only detect large effects. Accept that going in, or don’t run it. Setting a realistic MDE for a geo test usually reveals that only fairly dramatic interventions are worth testing this way, which is itself useful information.

The related design is the geo holdout, which is a cluster test with one treatment: turn the intervention off in matched regions and compare. That’s the backbone of incrementality testing.

Switchback tests

When the interference is through a system rather than a place, randomise time instead. Turn the treatment on for everyone for an hour, off for an hour, alternate for weeks, and compare the periods.

This keeps the whole system in one state at a time, so there’s no cross-contamination between concurrent groups. It’s the standard approach in logistics and marketplace pricing. The costs:

• Time periods are not independent. Whatever happened in the previous window carries over into the next one, especially for anything with a decision lag.
• Day-of-week and time-of-day effects have to be balanced across conditions by design, not by luck.
• The unit of analysis is the time window, so again you have far fewer units than users.

Switchbacks are rare in ordinary CRO and common in operations-heavy businesses. Worth knowing exists so you recognise the situation when it appears.

The practical read

For most website work, don’t reach for any of this. Individual randomisation is right, the interference is negligible, and cluster designs will cost you the ability to detect anything.

What I’d actually do is keep a short mental checklist and run it at test design, alongside the usual validity threats: is there fixed stock, a fixed team, a fixed budget, or a sharing mechanic between these two groups? If yes, either pick a metric downstream of the constraint (qualified opportunities rather than form fills, for instance), or accept that the number is a ceiling rather than an estimate. Naming the transfer is most of the value even when you can’t design around it.