Novelty effect and regression to the mean

Two separate phenomena that both mess with early test reads. Different mechanisms, same practical consequence: stop a test too early and the lift estimate won’t hold up.

Novelty effect

When users encounter something new on a site they know well, they engage with it differently. They click the unfamiliar element just to see what it does. They notice the new layout because their eye isn’t trained yet. This bumps engagement metrics on the new variant for the first few days, then fades as users habituate.

Where it shows up:

• A new homepage banner gets unusually high CTR for the first week, then settles
• A redesigned PDP shows higher add-to-cart early, then drifts back toward control
• Repeat-visitor segments showing strong lifts that don’t appear in first-time visitors

The fix is patience. Run tests at least one or two full purchase cycles so the novelty has time to fade. For DTC that’s usually two to four weeks minimum, longer if your repeat purchase cycle is slow.

Regression to the mean

A statistical phenomenon, not a behavioural one. Extreme observations are more likely to be followed by less extreme ones, because the extreme reading included a chunk of randomness that won’t repeat. Check your test at day 3 and see the variant showing a 15% lift, the day-10 reading will almost certainly be smaller. Not because the effect “wore off” but because the day-3 number was inflated by noise.

This is why early peeking is dangerous. The cases where you’d be tempted to stop early (“the variant is crushing control!”) are precisely the cases where the early reading is the most noise-inflated. Sequential testing methods exist to allow valid peeking, but most testing platforms aren’t doing real sequential.

Why they get confused

Both make early effects look bigger than they really are. The mechanism is different (novelty is behavioural, regression is statistical) but the prescription is identical: don’t stop early, run to your planned sample size.

Where the reading goes wrong

• Attributing all the early lift to novelty. It’s usually partly real, partly novelty, partly regression, and you can’t separate them from inside the first week. That inability is itself the argument for waiting rather than for adjusting.
• Assuming a long test averages novelty out. It does, but only if you read the steady-state period. A cumulative average still has the novelty window baked into it, so a four-week test reported as one number is partly reporting week one.
• Confusing regression with confidence interval shrinkage. Regression is the point estimate moving toward the truth. CI shrinkage is the uncertainty around it narrowing. Both happen as data accumulates and they’re different things.