Case IV: Measuring Online Ads Effectiveness at Star Digital

Please download and load data “Ad_Experiment.Rdata” from Canvas.

• Purchase: A dummy variable indicating whether the consumer eventually subscribed at Star Digital or not (= 0 if there was no purchase; = 1 if there was a purchase).
• Imp_1 - Imp_6: The number of impressions for each consumer at Website 1 to Website 6
• Test: a factor with 2 levels: “Control” and “Treatment”

load("Ad_Experiment.RData")
head(AdExp_Data)

## # A tibble: 6 x 9
##      ID Purchase Test      Imp_1 Imp_2 Imp_3 Imp_4 Imp_5 Imp_6
##   <dbl>    <dbl> <fct>     <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1    27        0 Treatment     0     0     0     0     0     1
## 2   126        0 Treatment     0     1     0     0     0     0
## 3   167        1 Treatment     0     0     0     1     0     0
## 4   195        1 Treatment     0     1     0     0     0     0
## 5   323        1 Control       0    10     0     0     0     0
## 6   365        0 Treatment     0     0     0     0     0     1

To get some variables for later use.

Get the no. of consumers in control vs. treatment group.

# N[1] - the no. of control cases; 
# N[2] - the no. of treatment cases.
N <- table(AdExp_Data$Test) 
N

## 
##   Control Treatment 
##      2656     22647

Get two index vectors of consumers in treatment vs. control group.

# Consumers in the treatment group
Treatment <- which(AdExp_Data$Test == "Treatment")

# Consumers in the control group
Control <- which(AdExp_Data$Test == "Control")

Bootstrapping ATE (i.e., Expected Lift) and its CI’s

• Start bootstrapping with an “Unpooled” procedure. That is, we obtain two random samples for Treatment and Control group, respectively.
• Respectively for the treatment and control group, we obtain a random sample with the same sample size as the original data (\(N_{control}=N[1]=2656\) and \(N_{Treatment}=N[2]=22647\)) WITH replacement (! very important).
• This procedure will deal with the unequal group sizes and the sampling variations are taken into consideration.
• We will use a function at R \(sample(x, size, replace = T)\), where \(x\) is the vector to sample from, \(size\) is the number of elements to choose from \(x\) (for bootstrapping, this is set at \(length(x)\)), and \(replace=T\) points to sampling with replacement for bootstrapping.

#Start boostrapping with an "Unplooed" procedure

#set the seeds for replication; you may delete this line. 
set.seed(123456789) 

#the no. of bootstrapping samples; can be changed to other values
S <- 2000 

# a vector to store boostrapped lift estimates
Lift <- rep(0,S) 

for (i in 1:S) {
  
  # To produce a bootstrappted data set:
    # A random sample with the same sample size, sampled with replacement.
    # Using the two indices variables "Treatment" and "Control".
    # Sampling two vectors that are of same length as "Treatment" and "Control" with replacement.
  Data.boot <- AdExp_Data[c(sample(Treatment,N[1],replace = T),
                      sample(Control,N[2],replace = T)),2:3]
  
  # To calcualte the ATE for each bootstrapped sample: Data.boot
  Lift[i] <- mean(Data.boot$Purchase[Data.boot$Test=="Treatment"]) - 
    mean(Data.boot$Purchase[Data.boot$Test=="Control"]) 
}

#Get the ATE and bootstrapped CIs
ATE <- mean(Lift)

# The 95% confidence interval
CI_95 <- quantile(Lift,c(.025,.975))

# The 90% confidence interval
CI_90 <- quantile(Lift,c(.05,.95))

#We can also plot "Lift" and the 90% and 95% CI's
hist(Lift,breaks = 100)
abline(v=c(CI_90,CI_95),
       col=c("blue","blue","red","red"),
       lty=rep(2,4),
       lwd=rep(2,4))

Calcuating ROI of Online Display Ads

• If we believe that the ATE of \(0.0190\) is credible, we can then use the ATE (average Lift) to calcualte ROI for further reporting.
• The ROI calculation requires two values: 1) the profits from the conversions; and 2) the costs of online ads. As the costs are the payment for impressions (CPM), we use consumers in the treatment group (as the control group were served no Digital Star ads, we cannot calcuate the costs).
• The overall expected profits should be: \(Profits=ATE \times N_{Treatment} \times 60\), with €60 the CLV of one subscription (see page 3 of the case).
• The overall costs are from the CPM model or costs per thousand impressions. We use: \(Costs=\dfrac{Impressions_{1-5}}{1000} \times €25 + \dfrac{Impressions_{6}}{1000} \times €20\), as the CPM for Website 1-5 is €25 and Website 6 is €20 (see page 3 of the case).

# Sum up the impressions for treatment group consumers
Impressions <- colSums(AdExp_Data[Treatment,-c(1:3)])

# Different CPMs as shown above: Website 1-5 - 25 and Website 6 - 20
Costs <- sum(Impressions/1000*c(rep(25,5),20))

#The return is the number of people in treatment * the ATE (Lift) * 60 euros
Profits <- ATE*N[2]*60

# Calculate ROI: Return/Investment = (Profits-Costs)/Costs
ROI <- (Profits-Costs)/Costs

# Show final results
Results <- c(Costs,Profits,ROI)
names(Results) <- c("Costs","Profits","ROI")
Results

##        Costs      Profits          ROI 
##  4254.390000 25801.946657     5.064782