Credible intervals
A credible interval is the range the true effect falls inside with a stated probability, given the data and the prior. A 95% credible interval of [+1.2%, +6.8%] means there’s a 95% chance the real effect is somewhere in there.
That sentence is the entire point. It’s the reading everyone instinctively applies to a confidence interval, where it’s wrong, and it’s correct here. A frequentist interval is a property of the procedure - repeat the experiment forever and 95% of the intervals you’d construct would contain the truth, though you never learn which kind yours is. A Bayesian one is a statement about the effect itself, because in that frame the effect has a distribution rather than being a single fixed unknown.
Why it matters in practice
Section titled “Why it matters in practice”Mostly for communication, which isn’t a small thing. Reporting a confidence interval honestly requires either a caveat nobody listens to or a small lie of convenience, and most people quietly choose the lie. With a credible interval you can say the natural sentence and be right.
It also composes into a decision more cleanly. Once the output is a probability distribution over effects rather than a pass/fail verdict, you can ask what shipping the wrong thing would cost, which is what expected loss and ROPE are built on.
Where they agree
Section titled “Where they agree”Numerically, almost everywhere that matters. With a weak prior and a reasonable sample, a 95% credible interval and a 95% confidence interval land in nearly the same place - close enough that the difference is philosophical rather than operational. If a test has the traffic to be worth reading at all, the two will rarely disagree enough to change what you do.
Which is worth saying plainly, because the framework argument gets more attention than it deserves. Switching to Bayesian tooling changes how you talk about a result far more than it changes the result.
Where they come apart
Section titled “Where they come apart”Small samples and informative priors, which is exactly where most CRO tests live.
The prior does real work when the data is thin. That’s usually a feature: a mildly sceptical prior pulls a 40% lift on 300 sessions back toward zero, which is the correct response to a number that silly. But it means the interval you’re reading is partly the product of an assumption, and on most platforms that assumption was chosen by the vendor rather than by you. Worth knowing what yours is before you quote the number to anyone.
The other trap is that “95% credible interval” and “95% confidence interval” look identical in a slide, and get used interchangeably by people who’d give you the right definition if you asked. If a report doesn’t say which engine produced it, you don’t know which sentence you’re allowed to say out loud.