Randomisation

Random assignment is what makes A/B testing work. Without it, the comparison between control and variant is just two non-equivalent groups with whatever traffic happened to land in each. With it, the variants are statistically interchangeable in expectation, so any outcome difference is plausibly the effect of the change.

It carries one further assumption that’s easy to miss: that one user’s outcome doesn’t depend on another user’s assignment. That holds for most website testing and breaks wherever the two groups share something finite, which is the subject of interference and cluster randomisation.

The mechanism is usually a hash. Take the user ID (or session ID, or visitor ID from a cookie), hash it to a number, mod by 100, and assign to a variant based on the bucket. Same input always gives the same hash, so the same user keeps getting the same variant across visits.

What “consistent” assignment means in practice

A user assigned to variant B on Monday should still see variant B on Friday. Otherwise the visitor experiences variant inconsistency, which both confuses them and corrupts the test. The mechanisms that preserve consistency:

• Cookie-based assignment - works as long as the cookie sticks. Falls apart across devices, after cookie clearing, and in private windows.
• User-ID based assignment - works for logged-in users. Bulletproof in product analytics, useless for pre-login traffic.
• Hybrid - cookie for anonymous, user-ID after login, with a stitching mechanism to reconcile. Standard in most modern A/B testing platforms.

Sample ratio mismatch

If you assign 50/50 and 100,000 visitors arrive, you’d expect roughly 50,000 per variant. If the split is 47,000 / 53,000, something is broken. The usual causes:

• The variant takes longer to load and bots or impatient visitors bail before assignment is recorded
• A redirect-based test loses traffic in the redirect itself
• Assignment happens after rather than before a tracking event, so the variant gets undercounted
• Bots are routed differently from humans by an intermediate CDN rule

SRM is the most important diagnostic check. Every platform should run it automatically and flag any split that’s significantly off the planned ratio. A test with SRM is invalid regardless of the result.

Ways assignment gets built wrong

• Bucketing on IP. Everyone behind one corporate or household NAT gets the same variant. It looks random and it isn’t, and it’s worst exactly where B2B traffic concentrates.
• Bucketing on time. Splitting by hour or by day is the most common home-built shortcut and it maps day-of-week and time-of-day effects directly onto variant. Whatever it measures, it isn’t the change.
• Not locking assignment before first paint. If bucketing races the render, the user can see one variant then the other, and their data belongs to neither.
• Randomising on one unit and analysing another. Assign by user, measure by session, and the sessions inside a user aren’t independent, so the variance is wrong and significance is overstated. See ratio metrics for what to do instead.