Bayesian approximations in A/B tests
How close does an approximation get to numerical simulations?
A/B testing is common in businesses. This test puts two versions of the same experience in contest with other another. People (such as website users) see one experience through random assignment.
The research question is simple to state: which version is better? This article compares three different uncertainty intervals in R.
Uncertainty intervals
In an earlier post, I wrote, of an approximation:
For this example, that 95% (highest density) interval is from -0.1 to 8.1 points. This is alike to both numerical methods and classical approximations.
Suppose there are 1,000 users on each version of the web page. Each user converts or not, so the data follows a Binomial distribution. We set two independent prior distributions as uniform — with equal density between 0 and 1.
The first version (page A) had 300 conversions. The second version (B) had 340. The difference in conversion ratios is four percentage points.
set.seed(4744)
xA <- 300; nA <- 1000; alphaA <- 1; betaA <- 1
xB <- 340; nB <- 1000; alphaB <- 1; betaB <- 1
number_sims <- 50000The Beta distribution is a conjugate prior to our Binomial data. That means the prior and posterior distributions are in the same family.
