A/B Test Simulator

Katherine Bailey
July 25th 2015

A Shiny Dashboards app that lets you simulate the results of an A/B test.

A/B testing web content

In an A/B test on a web page, visitors are randomly shown one of a set number of variations and the performance of each is measured against some goal, e.g. whether the visitor clicks on a link.

A/B testing web content

In this simulator, the number of variations and the number of visits are taken as input, and “visits” are simulated where one of the variations was shown and the visitor either did or did not convert. The conversion probability is set higher for one of the variations (30%) than for the others (10%).

Try it out here!

Example Simulation

In this example we see that Variation 3 was the clear winner

       Choice      Mean  Std.Dev  Variance Times.Shown Goals
1 variation-1 0.1492540 0.359028 0.1289010          67    10
2 variation-2 0.0597015 0.238721 0.0569878          67     4
3 variation-3 0.3030300 0.463090 0.2144520          66    20

plot of chunk results

How confident are we in our winner?

The app also displays a confidence value measuring the degree of confidence, between 0 and 1, where 1 is total confidence, that our winning variation will indeed out-perform each of the other variations.

# Calculates a test statistic for the difference
# between two sets of observations. With this and 
# degrees of freedom we can calculate confidence.
calculateTScore <- function(nx, ny, meanx, meany, varx, vary) {
  top <- (meanx-meany)
  bottom <- sqrt((varx/nx) + (vary/ny))
  top / bottom
}

    Variation Confidence
1 variation-1  0.9827488
2 variation-2  0.9998738