Interpreting results
Most teams read test results as a binary: did the primary metric cross alpha or not. That’s the worst possible interpretation. A test result is a multi-dimensional signal and reducing it to one bit throws away most of what the test told you.
A complete read of a test includes:
The point estimate and confidence interval
Section titled “The point estimate and confidence interval”The point estimate (the observed effect size, like a +4% lift) is the most likely value. The confidence interval is the range of values consistent with the data. The width tells you how much you actually learned.
A “significant” +4% lift with a CI of [+0.5%, +7.5%] is a different result from the same +4% with a CI of [+3.7%, +4.3%]. Both pass the threshold. The first is barely informative (the true effect could be trivial or large). The second is precise.
The primary metric in context
Section titled “The primary metric in context”Did it cross the threshold? By how much? Is the magnitude commercially relevant (above the MDE you set during sample size calculation)?
A statistically significant 0.3% lift on a metric where the MDE was 5% is technically a positive result and substantively a non-result. The test wasn’t powered for that effect size and the estimate is dominated by noise.
Guardrails
Section titled “Guardrails”Did any guardrail breach? A primary metric win that comes with a 15% bounce-rate increase isn’t a clean win. The full ship decision needs both.
Segments
Section titled “Segments”Did the effect look consistent across the main segments (mobile / desktop, new / returning, paid / organic)? Or is the headline result hiding a strong positive on one segment and a negative on another? Pre-specified segment analysis is part of interpretation. Exploratory segment fishing is not (see multiple testing).
Plausibility
Section titled “Plausibility”Does the magnitude of the effect make sense given the size of the change? A 30% lift from a button colour change is almost certainly noise or a measurement artefact, even if it’s “significant”. An effect that’s much bigger than the intervention should justify is a flag, not a celebration.
Non-significant is not “no effect”
Section titled “Non-significant is not “no effect””The read that costs the most value, and it’s the default read almost everywhere. A test that doesn’t cross the threshold has told you that you didn’t see enough evidence to call it. That is not the same claim as “the change does nothing”, and treating them as equivalent is how real wins get abandoned quietly.
The distinguishing question is whether the test was powered for the effect you’d have cared about. A non-significant result on a well-powered test is genuine evidence of a small-or-absent effect and you should update on it. A non-significant result on an underpowered test is close to no information at all, and the honest write-up says so rather than filing it as a loss.
Look at the interval to tell them apart. If it comfortably excludes anything worth having, you’ve learned something. If it spans everything from a meaningful loss to a meaningful win, you’ve learned that you ran a test you couldn’t read.