Growth Notes

Randomisation Unit

Tags: experimentation analytics concept

Randomisation Unit

Date: 2026-08-16

What gets assigned to a variant — a user, a session, a device, an account. It decides what a consistent experience means, what your denominator can be, and whether your significance calculation is valid at all.

What it is

The randomisation unit is the entity the coin is flipped for. Everything that entity does afterwards belongs to the arm it was assigned to.

It is not the same as the analysis unit — the thing you count in the denominator. They should match, and most invalid tests are tests where they don’t.

The options

SESSION-LEVEL                          USER-LEVEL

user u1, session s1 → variant          user u1 → variant
user u1, session s2 → control          ├─ session s1 → variant
user u1, session s3 → variant          ├─ session s2 → variant
                                       └─ session s3 → variant
same person, three experiences         same person, one experience
UnitConsistent experienceCross-deviceNotes
User IDYes, everywhere they log inYesBest available, but only for identified users — see Identity Stitching
Cookie / devicePer browserNoThe ecommerce default. Breaks on cookie clearing
SessionWithin one visit onlyNoAlmost always wrong. See below
AccountYes, across all users on itYesFor B2B or shared accounts, where the buying decision is collective
Page viewNoNoValid only for genuinely one-shot decisions

Why session-level assignment is nearly always wrong

Three separate failures, and they compound.

1. The user sees both versions. They browse on Monday in variant, return on Wednesday in control, and their behaviour reflects a confusing mixture. The effect you measure is diluted towards zero.

2. Independence is violated, so the statistics are wrong. The significance maths assumes each observation is independent. Three sessions from one person are correlated — that person’s habits carry across all three. Treating them as three independent observations understates the true variance and overstates significance, producing false positives at a rate well above your α. See Type I and Type II Errors.

3. The denominator is a modelling artefact. Session counts move with the inactivity timeout, and worse, a variant that increases interaction generates fewer sessions — more events fill the timeout gaps. So session conversion rate can rise with no additional orders. Worked in Sessionisation:

        orders   sessions   session CR
control    400      2,500       16.0%
variant    400      2,100       19.0%   ← "wins" by 19% relative, sold nothing extra

In plain terms: randomising by session lets the change alter the thing you’re dividing by. The test can be won by making the site more engaging to click around in, which is not what anyone meant to measure.

The rule

Randomise on the unit you want to analyse, and analyse on the unit you randomised. For nearly all CRO work that’s the user — cookie-level where anonymous, user ID where identified.

Consequences to accept once you’ve chosen user-level:

  • Denominator is users, so conversion rate is users-who-converted ÷ users-exposed
  • Sample size is in users, not sessions or pageviews — Sample Size Calculation
  • Assignment must be sticky across visits, which requires durable storage and degrades as cookies expire — Browser Privacy Restrictions

Where a different unit is genuinely right

  • Session or pageview for a decision made entirely within one visit that has no memory — a search results ranking, say — and only where the outcome can’t be influenced by prior exposure
  • Account where several people share a buying decision. Randomising individuals inside one account means colleagues see different prices or flows, which is both confusing and, for pricing, a compliance problem
  • Geography or time where users can’t be independently assigned because they share a resource — delivery capacity, stock, a marketplace. That’s Switchback Tests and geo designs, and the effective sample size is the number of regions or time blocks, not users

Failure modes

  • Randomising by user, reporting by session — the most common invalid analysis. The split is fine; the denominator isn’t
  • Anonymous unit, logged-in outcome. Assignment on a cookie, conversion recorded against a user ID that spans three devices. Joins fail, and they fail more often for your best customers
  • Unit changes mid-test because someone logs in and the identifier switches — see Assignment and Bucketing on sticky assignment
  • Different units across tests in the same programme, so results can’t be compared or meta-analysed

Graph View

Backlinks