Missing data and outliers

Most test analyses ignore both. Sessions with missing fields get silently dropped. Extreme values (whales spending £5,000, sessions of three hours) get included as if they were typical. Either choice can flip a result without anyone noticing, and the defaults in most testing platforms are usually wrong.

Missing data

Three flavours of why data goes missing, with different implications:

• Missing completely at random. The missingness is unrelated to anything (random tracking failure, browser issues). Listwise deletion is fine - dropped users represent the underlying population.
• Missing at random. The missingness depends on observed variables but not on the outcome. e.g. iOS users have higher tracking opt-out, but within each platform the missingness is random. Listwise deletion works if you stratify, otherwise you bias the result.
• Missing not at random. The missingness depends on the outcome itself. Users who don’t convert close the tab before the conversion event fires. Listwise deletion biases toward over-counting converters. This is the dangerous case and it’s common.

The pragmatic CRO approach: check the sample ratio by variant. If it’s clean, missingness is probably balanced. If it’s not, missing-data bias is plausibly affecting the result and you should investigate before trusting it.

Outliers

Extreme values can dominate ratio metrics like AOV and revenue per visitor. One £10,000 order in a 10,000-session test moves the variant mean by £1 per session, which can flip a “win” into a “loss” or vice versa.

Three handling options:

• Include as-is. Honest but high-variance. The estimate is correct in expectation but the confidence interval is wide.
• Winsorise. Cap extreme values at a percentile (e.g. 99th). Reduces variance, keeps the user in the analysis. Standard in mature experimentation platforms.
• Trim. Drop extreme values entirely. Maximum variance reduction but discards information.

Whichever you pick, do it for both variants identically and decide before seeing the data. Otherwise you’re choosing the rule that confirms the result you wanted. If the conclusion flips depending on which rule you picked, that’s a sensitivity analysis finding and the result is fragile.

The interaction with ratio metrics

Both missing data and outliers are worse on ratio metrics than on means. A heavy-tailed AOV is more sensitive to extreme orders than a click-through rate is to extreme click patterns. Apply variance-stabilising treatments (winsorising, log-transforming) more aggressively for ratio metrics where the tail matters.

Find out what your platform already did

Both of these decisions get made whether or not you make them. Some platforms winsorise by default, some don’t. Some apply listwise deletion silently, some carry nulls through. The behaviour is frequently undocumented, and the number on your dashboard already has one of these rules baked into it.

So the first move on any unfamiliar platform is to find out which, because you can’t reason about a result whose preprocessing you don’t know. If the documentation doesn’t say, a quick check against raw backend data will tell you.

Worth reframing one thing while you’re here: sample ratio mismatch usually is missing-data bias. It gets filed as an assignment bug, and sometimes it is, but a variant that loses users before their data lands produces exactly the same signal. Reading SRM as “something is systematically dropping users, find out which ones” is more often right than reading it as “the splitter is broken”.