Sample size

The pre-test maths that tells you how many visitors you need before you have a reasonable shot at detecting an effect. Any decent A/B testing tool has a calculator built in, and Evan Miller’s is the one to check yours against. Feed it your baseline rate, your minimum detectable effect, your alpha, and your power, and it spits out the required sample per variant.

The four inputs:

Baseline conversion rate - what your control currently does. The lower the baseline, the bigger the sample you need.
MDE (Minimum Detectable Effect) - the smallest lift you actually care about detecting. The input people get wrong.
Alpha - usually 0.05.
Power - usually 0.8.

The MDE is the load-bearing assumption. It’s not “the effect you expect”, it’s the effect that’s worth detecting. Set MDE too low (say 1%) and you get huge required samples that smaller sites can’t reach. Set it too high (say 20%) and a real but smaller win will look “non-significant” and you’ll incorrectly conclude the change didn’t work.

In practice on Shopify

For a typical mid-size Shopify store doing 50k sessions a month, a test on add-to-cart rate with a 10% baseline, MDE of 10% relative, alpha 0.05, power 0.8, needs around 15,000 sessions per variant. That’s 30,000 in total, about three weeks of all your traffic going through the test. Two-variant tests fit. Multivariate designs don’t.

Same store, same 10% relative MDE, but measured on purchase conversion at a 2% baseline instead of add-to-cart at 10%.

Metric	Baseline	Per variant	Runtime at 50k/mo
Add to cart	10%	~15,000	3 weeks
Purchase	2%	~81,000	3 months

Nothing about the change being tested is different. The rarer the event, the more traffic it takes to see the same relative movement in it, which is why upper-funnel metrics are testable on stores where purchase conversion isn’t.

The honest output of a sample size calc is often “you can’t actually run this test in a reasonable time”. That’s useful information. It means either pick bigger swings or accept you’re running directional tests, not statistical ones.

My rule of thumb: if the calc returns more than six weeks, the test gets rescoped or shelved. Six weeks is where seasonality, traffic shifts, and stakeholder patience start to bite, and a test that runs longer than that is fighting more than the null.

Ignoring your own calculation

The calc is easy. Living with the answer is the hard part, and there are three ways teams get out of it.

Never running it. Most programmes just run tests until they feel ready, which in practice means until they look significant. That’s peeking with extra steps.
Stopping before you hit the number. The calculation assumes you run to the planned sample. Stop early because it looks good and your effective alpha is well above the nominal value. Sequential testing is the legitimate version of this and it requires deciding to use it up front.
Recalculating mid-test on the observed effect. Test looks underpowered, so you feed the current lift back in as your new MDE and discover you need fewer sessions than you’ve already collected. That isn’t a sample size calculation, it’s data dredging with a calculator.