Case V: Measuring Position Effects of Search Ads

Please download and load data “Keywords_Bid.Rdata” from Canvas.

• adrank: A variable based on the ad rank value which is defined as \(Bidding Price \times Quality Score\). This variable is the running variable which decides whether the adposition is first or second. The value of adrank variable = own adrank value - neighbor bidder ad rank value; If an advert is ranked first, then this value if positive. If an advert is ranked second, then this value is negative. These follow directly from the ladder auction setup of keywords bidding.
  
• adposition: A factor of two levels which indicates the positions of ads (first and second). The relationship between adposition and adrank variable is like this: \(adposition = 1(adrank>0)\).
• revenue: The revenue of the ad campaigns (in log scale)

load("Keywords_Bid.RData")
head(data)

## # A tibble: 6 x 3
##   adrank adposition      revenue
##    <dbl> <fct>             <dbl>
## 1 -0.220 Second Position   0.844
## 2  2.00  First Position    4.99 
## 3  0.278 First Position    5.85 
## 4  1.87  First Position    8.51 
## 5 -2.90  Second Position   0    
## 6  0.295 First Position    4.73

To visualize the relationship between the running variable (adrank) and the outcome (revenue)

Here we would like to check whether a “discontinuity” exists in the relationship between the running variable (adrank) and the outcome (revenue). To do so, we produce a scatter plot between adrank and revenue and separate into two groups by adposition.

library(ggplot2)
ggplot(data, aes(adrank, revenue, color = adposition)) + 
  geom_point() + stat_smooth(method = "loess",colour = "black",size = 1) + 
  geom_vline(xintercept=0, linetype="longdash")

## `geom_smooth()` using formula 'y ~ x'

Selecting the “optimal” bandwidth with cross-validation

• Here we try to select the “optimal” bandwidth with the cross-validation method. As our data size is relatively small, we use a 2-fold cross validation. The optimal bandwidth is the one that has the smallest mean squared error (MSE) on the testing sample.
• In general, a Jackknife or 10-fold CV is more appropriate. This practice is supposed to give you a taste of the RDD procedure.
• To select the optimal bandwidth, we adopt a regression-based estimator of position effects. To be more specific, we estimate a regression equation like this: \(Revenue_i=Adposition_i+Adrank_i+Adposition_i\times Adrank_i+e_i\). The interaction term is included in case the slope of adrank on revenue changes in first vs. second position.
• Note that other specs can be used, for example a local polynomial regression or the addition of higher order terms. These are usually added robustness checks, as the basic spec is easy to interpret.
• In the code below, we test 4 bandwidths just as a show-and-tell. In practice, you should compare more bandwidths to select the “optimal” one. We use here the half of standard deviation of the running variable “adrank” with \(0.25\), \(0.50\), \(1.00\) and \(2.00\) times the \(SD(adrank)\).

#===== to split data into test and train set --------------------

# Set seeds for replication 
set.seed(12345) 

# Permutate the index
N <- dim(data)[1]
idx <- sample(1:N,N,replace = F)

# To obtain a random subsample as train and test data
train <- data[idx[1:(N/2)],]
test <- data[idx[(N/2+1):N],]

#===== To select optimal bandwidths ------------------------------
  # Here we test 4 bandwidths just as a show-and-tell. 
  # In practice, you should try finer grids. 

# standard deviation of adrank
H <- sd(data$adrank)
H <- rep(H,4)*c(.25,.5,1,2)

# a vector to store MSE 
H.fit <- rep(0,4)

# Loop over 4 bandwidths
for (i in 1:4) {
  
  # This is the model specified above. More complex method may be used here. 
  mdl <- lm(revenue~adrank*adposition,train[abs(train$adrank)<H[i],])
  
  # To obtain the predicted outcome for test data. 
  revenue.hat <- predict(mdl,newdata = test[abs(test$adrank)<H[i],])
  
  # To calcualte and store MSE
  H.fit[i] <- mean((test$revenue[abs(test$adrank)<H[i]]-revenue.hat)^2)
}

# To get the bandwidth of the one with mininum MSE
H <- H[H.fit==min(H.fit)]
print(c("The optimal bandwidth is",H))

## [1] "The optimal bandwidth is" "0.56198147093334"

Estimating the position effects with the optimal bandwidth

• After having the optimal bandwidth \(H\), we rerun the model with the full data.
• This gives us a “local average treatment effects” (LATE). An extension is to put in characteristics of ads and then obtain the heterogeneous LATE.
• You can also extend to different positions such as “position 3 to 2” or “position 4 to 3”.
• By doing so, you may obtain a \(V_i=\beta_{adposition}/Impressions\) for ad campaigns which will guide your bidding strategies. So you know the max bid you should put there.

#Rerun a model on full data with bandwidth set to H
mdl <- lm(revenue~adrank*adposition,data[abs(data$adrank)<H,])

# The position effect is measured by the coefficient of adposition.
# This let you if you boost your ad position from 2nd place and 1st place, how much increase you expect in your revenue. 
summary(mdl)

## 
## Call:
## lm(formula = revenue ~ adrank * adposition, data = data[abs(data$adrank) < 
##     H, ])
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.0100 -0.6302  0.0005  0.6349  2.8303 
## 
## Coefficients:
##                                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                      2.02533    0.08785  23.054  < 2e-16 ***
## adrank                           1.99824    0.32495   6.149 1.02e-09 ***
## adpositionFirst Position         2.42697    0.13604  17.840  < 2e-16 ***
## adrank:adpositionFirst Position  1.08011    0.42983   2.513   0.0121 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.9282 on 1369 degrees of freedom
## Multiple R-squared:  0.826,  Adjusted R-squared:  0.8256 
## F-statistic:  2167 on 3 and 1369 DF,  p-value: < 2.2e-16