Definition

A/B Testing is a general methodology used when the objective is to test a new product or feature. The goal is to design an experiment that is robust and gives repeatable results so as to make an informed decision to launch or not.

Assumptions

  1. Test Version A is the control group which depicts the existing products or features on a website.

  2. Test Version B is the experimental group to experiment the new version of product or feature to see if users like it or if it increases hotel bookings (conversions).

  3. Converted – Based on the given data set, there are two categories defined by logical values (True or false):

  1. Converted = True when customer successfully makes a booking
  2. Converted = False when customer visits the sites but does not make a booking

Out of Scope

I. Type 1 & Type II Errors

Objective

  1. To lay out the conclusions drawn from the results of sample data provided
  2. To make recommendations on possible paths and provide relevant justification.Pcont_A = PExp_B

A/B Test Hypothesis

Null Hypothesis:

Both version A and B have the same probability of driving customer booking or conversion. In other words, there is no effect or no difference between version A and B.

Alternative Hypothesis:

Both version A and B have a different probability of driving customer booking or conversion. There is a difference between version A and B. Version B is better than A in driving customer bookings. PExp_B != Pcont_A

Conclusions Drawn from Data

  1. There are 711 hits and 18 conversions for test version A and 720 hits and 33 conversions for test version B.

  2. Relative uplift of 81.04% based on a Conversion rate for A = 2.53%, Conversion rate for B = 4.58 %. Hence B is better than A by 81.04%.

  3. P-value computed for this analysis was 0.03633269 (p < 0.05 or p lower than 5% significance level). Hence, tests results show strong statistical significance. We can be 95% confident that this result is a consequence of the changes made and not a result of random chance.

  4. The computed pooled data confidence interval (lower = 0.001305794, upper = 0.03972796) does not include zero. The confidence interval of “B” is further away from zero than “A” which further strengthens the recommendation.

Analysis

  1. Take a subset of conversions for test_version_A:
  2. Number of Conversions for test_version_A:
## [1] 20

3.Number of Visitors for test_version_A:

## [1] 721
  1. Compute conversion rate for test_version_A:
## [1] 0.02773925
  1. Take a subset of conversions for test_version_B:
  2. Number of Conversions for test_version_B:
## [1] 37

7.Number of Visitors for test_version_B:

## [1] 730
  1. Compute conversion rate for test_version_B:
## [1] 0.05068493

Relative Uplift

  1. Calculate Relative uplift in Conversion Rate:
## [1] 82.71918

Deduction: B is better than A by 81%. This is high enough to decide a winner but we need more supporting evidence before we can arrive at a final decision.

Pooled Probability for Test Versions A & B

  1. Compute pooled probability:
## [1] 0.03928325
  1. Compute the standard error (SE_pool) for the pool:
## [1] 0.01020014
  1. Compute the margin of error for the pool:
## [1] 0.0199919
  1. Point Estimate or Difference in proportion:
## [1] 0.02294568
  1. Compute the Z-score so we can determine the p-value:
## [1] 2.249546

P-Value for Test Version A & B

  1. Compute p_value using the z_score value:
## [1] 0.02447777
  1. Compute Confidence interval for the pool
## [1] 0.002953777 0.042937584
## [1] 0.002953777
## [1] 0.04293758
  1. Compute Standard Error and Confidence Intervals for Test version A separately
## [1] 0.02773925
## [1] 0.006116051
## [1] 0.01575201 0.03972649
  1. Compute Standard Error and Confidence Intervals for Test version B separately
## [1] 0.05068493
## [1] 0.008118638
## [1] 0.03477269 0.06659717
  1. Extract the lower and upper confidence intervals into lower and upper respectively
## [1] 0.01575201
## [1] 0.03972649
## [1] 0.03477269
  1. Create a data frame of results for the pool
  2. Create a data frame of results per test version

Visualize the Results

  1. Create the data frame
  2. Create a sub-data frame we will call in our plot
  3. Ploting the distributions

Variant B’s observed conversion rate (5.07%) was 82.72% higher than Variant B’s conversion rate (2.77%). We can be 95% confident that this result is a consequence of the changes made and not a result of random chance.

  1. Visualize the pool using a normal distribution