Bayesian Data Analysis – A/B Testing

We have seen how Bayesian analysis is done to get the most probable parameter that would have resulted in the observed data for a single set.

1. Data
2. A generative model: a mathematical formulation that can give simulated data from the input of parameters.
3. Priors: information for the model before seeing the data

This time, we analyse two sets of data and compare them. The method is similar to what we have done for the single set.

Problem statement

There were two campaigns: one received positive reviews in 6 out of 10 and the other in 9 out of 15. We must compare them and report the better method, including the uncertainty range.

Unlike before, we will run two models side by side this time. Draw a random parameter 1 value from the prior1.

prior1 <- runif(1, 0, 1)

Run the model using prior1 to estimate simulated value 1.

sim1 <- rbinom(1, size = 10, prob = prior1)

In the same way, run the second model using another uniform prior.

prior2 <- runif(1, 0, 1)
sim2 <- rbinom(1, size = 15, prob = prior2)

Accept the parameter values (prior1 and prior2) only if the posteriors (sim1 and sim2) match the data, 6 and 9, respectively.

Now, you can find the difference between the two posteriors, resulting in a new distribution.

Reference

Introduction to Bayesian data analysis – part 2: rasmusab