If you are a Bayesian and not a Frequentist, the main way you can apply Bayesian Statistics to an AB test result is through using an online tool or a programming language such as Python or R... well until this blog post. If you do not have any programming skills, here I share a Google Sheets that leverage Bayesian Statistics to calculate the probability of your variant being better than the control as well as the relative uplift associated. All you need to do is plug the result from your AB test into the tool, easy peasy. Let’s have a look.
Bayesian AB-test calculator
I am not going to take you through a step by step guide on how to build the calculator, but instead introduce the one I built and use. Feel free to make a copy and retro-engineer. The Bayesian AB-test calculator in Google-Sheets is made of three parts:
- Choice of Prior: this is where you define your prior function and here I am using the Beta-binomial. I usually choose a flat prior, which is basically stating that any value between 0 and 1 are equaly likely for the conversion rates. This is what is set by default on the Google Sheets but feel free to play with it and adapt it to your needs.
- Test Results: this is where you input the sample sizes and number of conversions for your AB test. The Google Sheets will plot the posterior distributions and calculate the probability of the variant beating the control.
- The Uplift: the tool will output the mean uplift, the median uplift and the 30th Percentile for the uplift. As advised by Cameron Davidson-Pillon I always report on a conservative figure here and prefer reporting on the 30th percentile as opposed to the mean or median.
Are you Frequentist or Bayesian?
I wanted to close this blog post by sharing a video from Cassie Kozyrkov where she explains very skillfully the differences between Frequentist and Bayesian statistics: very useful if you are still trying to wrap your head around these concepts.