Design-led companies (Apple, Google, Airbnb, etc.) frequently apply design thinking to design new products (Naiman 2020). A/B testing (also known as split testing or bucket testing) is “a method of comparing two versions of a webpage or app against each other to determine which one performs better.” (Optimizly, 2019).
Some people get randomly assigned to group A, some others to group B. Each group is exposed to a different treatment of some underlying variable. This variable could be the discount amount, ad copy, etc. This underlying variable is what gets “manipulated.” Marketing researchers or data scientists then observe some outcome(s) that might be affected by the manipulated variables. We then create a dummy variable for the treatment group and for the control group.
The coefficient β1 can be interpreted as the additional effect that treatment A has on the outcome variable compared to treatment B. β0 can be interpreted as the average outcome, or the predicted value of Y, for treatment group B. Usually, if one of the treatments is considered a “control” group, the control group would be used as the baseline in the regression. i.e. the group that is left out and absorbed into the intercept β0.
Dummy coding is required for performing experimental research. Since we have dummy variables (i.e., a control/placebo group and a treatment group) in our model, the intercept has more meaning. Dummy coded variables have values of 0 for the reference/control/placebo group and 1 for the comparison/treatment group. Since the intercept is the expected mean value when X=0, it is the mean value only for the reference group (when all other X=0). Dummy coding is a way to make the categorical variable into a seri…
It is suggested that “the effect of advertising appears non-linear, with an optimum between two and three exposures per week (Tellis, 1987).” For our example on the relationship between advertising exposures and product purchase below, we will be testing the relationship between advertising and product purchase using regression analysis. Our null hypothesis (usually denoted as H0) is that there is no relationship between advertising exposures and product purchases using regression analysis. The alternati…
Null Hypothesis: H0: β1 = 0
Alternative Hypothesis: H1: β1 ≠ 0
First, we will be creating a new variable that has a value of one for each observation at that level and zeroes for all others. In our example using the variable (Ads), the first new variable (Ads1) will have a value of one for each observation in which the consumers are exposed to the 1st ads campaign and zero for all other observations. Likewise, we create Ads2 when the consumers are exposed to the 2nd ads campaign, and 0 otherwise. The level of the categorical variable that is coded as zero in the new…
# Please install the following package if the package 'readr' is not installed:
# install.packages("readr")
library(readr)
# Load data
data <- read_csv("ab_testing1.csv")## Rows: 29 Columns: 2
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (2): Ads, Purchase
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.ls(data) # list the variables in the dataset## [1] "Ads" "Purchase"head(data) #list the first 6 rows of the dataset## # A tibble: 6 × 2
## Ads Purchase
## <dbl> <dbl>
## 1 1 152
## 2 0 21
## 3 2 77
## 4 0 65
## 5 1 183
## 6 1 87# Creating the factor variable
data$Ads <- factor(data$Ads)
is.factor(data$Ads)## [1] TRUE# Showing the first 15 rows of the variable "Ads"
data$Ads[1:15]## [1] 1 0 2 0 1 1 2 2 2 0 2 2 0 2 2
## Levels: 0 1 2# Now we do the regression analysis and examine the results
summary(lm(Purchase ~ Ads, data = data))##
## Call:
## lm(formula = Purchase ~ Ads, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -59.75 -22.75 -3.75 30.25 64.29
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 49.00 10.21 4.800 5.69e-05 ***
## Ads1 69.71 15.91 4.383 0.000171 ***
## Ads2 24.75 13.82 1.791 0.084982 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 32.28 on 26 degrees of freedom
## Multiple R-squared: 0.4262, Adjusted R-squared: 0.3821
## F-statistic: 9.656 on 2 and 26 DF, p-value: 0.0007308# Alternatively, you can also use the factor function within the lm function, saving the step of creating the factor variable first.
summary(lm(Purchase ~ factor(Ads), data))##
## Call:
## lm(formula = Purchase ~ factor(Ads), data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -59.75 -22.75 -3.75 30.25 64.29
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 49.00 10.21 4.800 5.69e-05 ***
## factor(Ads)1 69.71 15.91 4.383 0.000171 ***
## factor(Ads)2 24.75 13.82 1.791 0.084982 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 32.28 on 26 degrees of freedom
## Multiple R-squared: 0.4262, Adjusted R-squared: 0.3821
## F-statistic: 9.656 on 2 and 26 DF, p-value: 0.0007308If the p-value for one of the ad groups is less than .05, we reject the null hypothesis in favor of the alternative hypothesis.
The coefficient for Ads1 in the regression output, if significant, indicates that the 1st Advertising campaign is more effective (relative to the group who did not receive any advertising exposure).
Example: If the estimates for β0 and β1 are 95.43 and 41.57,
respectively, then average sales would be:
- Control group: 95.43
- Ad 1 group: 95.43 + 41.57 × 1 = 137