Control vs treatment

Every A/B test has at least two groups:

• Control - the existing version. The baseline against which the variant is measured.
• Treatment - the variant being tested. Embodies the hypothesis.

The comparison between them is the experiment. Random assignment makes the two groups equivalent in expectation, so any outcome difference is plausibly the effect of the change rather than the effect of which users happened to land where.

Why the choice of control matters

In most simple A/B tests the control is obvious - it’s whatever’s currently live. In bigger redesigns the question gets harder. Is the control the current site as-is, or the current site with minor cleanup that was going to ship anyway? Is the control the version users see now, or a tighter version that strips out known broken bits?

The principle is that control should represent the realistic counterfactual - what would have been live if the test hadn’t happened. Choosing an artificially weak control inflates the apparent lift. Choosing a “fairer” control that doesn’t actually represent the status quo distorts the strategic decision.

Multiple treatments

A/B/C/D tests use one control and multiple treatments. The maths is straightforward but the multiple testing problem kicks in. More variants compared against control means more chances for one to randomly cross threshold. Each additional treatment needs more sample to maintain the same false-positive rate.

Holdouts vs control

A holdout is a special kind of control. The variant ships to 90% of traffic; 10% stays on control permanently for long-run measurement. The holdout’s purpose is different. It measures the long-tail effect of the change after launch, not whether the change wins in a fixed-horizon test. See holdout groups.

Control drifts

The word “control” implies something fixed, and it isn’t. Three consequences that catch people out.

• Control performance isn’t stable. Seasonality, ad-mix changes and other tests on related surfaces all move it. This is why comparing a variant against historical control data instead of a concurrently-running control is almost always wrong, however tempting it is when you want to skip the split.
• Last month’s winner isn’t automatically this month’s baseline. Promoting a shipped variant to control is right in principle, but if it won a noisy underpowered test you’ve just built the next test on a baseline that was partly luck.
• “The current site” is not a record. Document what control actually was when the test ran. Three months later, reanalysing against a description that says “current site” tells you nothing about what was live at the time.