Bayesian decision rules

Choosing Bayesian gets you a posterior distribution over the effect size. That’s a curve, not a verdict. Turning it into a ship-or-don’t decision needs a rule, and the rule you pick changes what you ship.

Three are in common use. Most tools show you the first, the second is the one you should be deciding on, and the third is the one that stops you shipping trivia.

Probability to beat control

The share of the posterior sitting above zero. “There’s a 94% chance the variant is better.”

It’s the friendliest output in experimentation and the easiest to over-read. Two things it doesn’t say:

• It says nothing about size. A 94% probability of an effect that’s almost certainly between +0.01% and +0.4% is a near-worthless win dressed up in a confident number.
• It’s not a probability about the observed lift. The point estimate has its own uncertainty. “94% chance to win” and “the lift is 3.2%” are two separate claims and the second is much softer than the first.

On its own it’s a poor decision rule, because it treats a rounding error and a transformational result identically as long as both clear the threshold.

Expected loss

The one worth building your process on. Expected loss asks: if I ship the variant and I’m wrong, how much do I lose on average, weighting every possible wrong-ness by how likely it is?

The output is in metric units, not probability. “Shipping B carries an expected loss of 0.008 percentage points of conversion rate.” You then set a threshold for how much downside you’re willing to accept - the caring threshold - and ship when expected loss falls below it.

Why this is better: it weights by magnitude. A high probability of a tiny effect produces a low expected loss, so it passes, and correctly so - if you’re wrong the damage is negligible. A moderate probability of a large effect also produces a decision you can reason about. You are no longer asking “is it real” but “what does being wrong cost me”, which is the actual business question.

The threshold is a business input in exactly the way MDE is. Set it by asking what size of loss you’d shrug at over the period the change is live. Most teams never set it and inherit whatever the platform defaults to, which is the Bayesian equivalent of accepting alpha = 0.05 without thinking.

Credible intervals and ROPE

The posterior also gives you a credible interval - the range the effect sits inside with a stated probability, and the one interval that genuinely means what people assume a confidence interval means.

On its own that’s a description rather than a decision. To get a rule from it, define a region of practical equivalence: a band around zero that counts as “no meaningful difference”, say anything between -0.5% and +0.5% relative. Then read the interval against it:

• Interval sits entirely above the band: ship.
• Interval sits entirely inside the band: call it flat, and mean it. This is the useful one, because it lets you conclude “no meaningful effect” rather than the frequentist non-answer of “we failed to reject the null”.
• Interval straddles the band: not enough data yet.

ROPE forces the conversation about what size of effect is worth having up front, which is the same discipline as setting an MDE, arriving from the other direction. It pairs well with expected loss - one bounds the downside, the other bounds the triviality.

Priors, briefly

Almost every CRO tool ships a weak or uninformative prior, which is defensible and slightly dishonest. The honest prior for website experimentation is pessimistic: most tests do nothing, and large effects are rare.

A prior centred on zero with modest variance encodes that, and its practical consequence is shrinkage - extreme observed effects get pulled toward zero, hardest when the sample is small. That’s a feature. It’s the mechanism that stops a 40% lift on 300 sessions from being reported as a 40% lift. If your platform lets you set it, a mildly sceptical prior will make you wrong less often.

The peeking caveat

Bayesian methods handle continuous monitoring better than naive frequentist tests, and they are not immune to it. Watching expected loss until it dips under your threshold and stopping there still biases the effect estimate upward, because you’re systematically stopping on favourable noise. The decision may be fine while the reported lift is inflated, which then feeds an inflated business case.

If you’re going to monitor continuously, fix the sample size in advance anyway and treat the early reads as operational monitoring rather than as the decision point. The rule you set before the test is the rule that protects you.

What I actually do

Expected loss against a threshold I’ve set deliberately, credible interval reported alongside it, and probability-to-beat mentioned to stakeholders because it communicates well - never as the thing the decision hangs on. When probability-to-beat is high and expected loss is uninteresting, that’s a flat test with good PR, and it should be read as one.