Bayesian A/B Testing Gives You Decision Probability, Not p-Values

Frequentist A/B testing answers whether the observed difference is unlikely under the null hypothesis. Bayesian testing answers what the probability is that variant B is better than variant A, and by how much. For business decisions, the second question is often more relevant. Decision probability is intuitive to stakeholders in a way that p-values are not.

In R, use bayesAB: the posterior distribution gives you P(B > A), the distribution of expected lift, and the probability that the lift exceeds the minimum effect you care about. You can also compute the expected loss of choosing each variant — the metric that maps directly to business risk.

Stop reporting p-values to product managers. Report: 'There is a 94% probability that the new checkout flow improves conversion by 2-8%, with an expected loss of 0.3% if the recommendation is incorrect.' That's a decision-ready statement.

The practical advantage of Bayesian testing: the posterior is a valid summary of the evidence whenever you look at it, so you can monitor a test continuously without the pre-specified stopping rules that frequentist sequential testing requires. That is not a licence to stop the moment P(B > A) crosses 95% — with weak priors, repeatedly peeking and stopping on a probability threshold still raises the rate of wrong calls. Stop early when the expected loss of the leading variant falls below a level you can live with, and the cost of being wrong stays bounded.

For implementation, the Beta-Binomial model handles conversion rate tests elegantly. For revenue tests, use a hierarchical model that accounts for the non-normal distribution of revenue data. The computation is fast enough for real-time dashboards showing posterior evolution over the test duration.