Goals for this analysis script:

See whether Facebook A|B testing is a “true” randomized A|B test

Key Takeaways:

FB’s A|B testing is NOT a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets)

FB A|B testing algorithm analysis

Demographic variables distribution

We are interested in:

  • is FB A|B testing a “true” randomized A|B test (FB draws a random sample from the total population, and randomly assigns it to one of the 5 ad sets), OR
  • is it a different comparative test (FB divides up the total population into five equal samples, then runs an algorithm to identify attractive target populations to maximize cost efficiency on each sample)?

Metric:

  • if it is a true A|B test, we should see roughly the same demographic distributions in each of the five ad sets.
  • if it isn’t, we should see very different demographic distributions in each of the five ad sets.

Findings:

  • Using the Facebook ads data on demographics, if we combine the 18-24 and 25 - 34 population group, the age distribution is roughly balanced across five ad sets
  • However, gender is not balanced across five ad sets, with two ad sets having about 10% lower female ratio (~57.5%) than the other three ads sets (~67.5%), which might suggest we are hitting different populations.

Impression scope

Impression distribution with Ages

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Impression distribution with combined Ages

Impression distribution with Genders

Ad Set Version:

  • A: unnecessary airtime
  • B: unnecessary control
  • C: risky airtime
  • D: risky control
  • E: inaccessible airtime

Reach scope

Reach distribution with Ages

Reach distribution with combined Ages

Reach distribution with Genders

Result scope

Result distribution with Ages

Result distribution with combined Ages

Result distribution with Genders