Design-led companies (Apple, Google, Airbnb, etc.) frequently apply design thinking to design new products (Naiman 2020). A/B testing (also known as split testing or bucket testing) is “a method of comparing two versions of a webpage or app against each other to determine which one performs better.” (Optimizly, 2019).
Some people get randomly assigned to group A, some others to group B. Each group is exposed to a different treatment of some underlying variable. This variable could be the discount amount, ad copy, etc. This underlying variable is what gets “manipulated.” Marketing researchers or data scientists then observe some outcome(s) that might be affected by the manipulated variables. We then create a dummy variable for the treatment group and for the control group. ## To evaluate the effect of the 2 different treatments of (A and B), we run the following regression: Y = β0 + β1*XA + ϵ
The coefficient β1 can be interpreted as the additional effect that treatment A has on the outcome variable compared to treatment B. β0 can be interpreted as the average outcome, or the predicted value of Y, for treatment group B. Usually, if one of the treatments is considered a “control” group, the control group would be used as the baseline in the regression. i.e. the group that is left out and absorbed into the intercept β0. In the regression model above, this would be group since is an indicator for whether treatment was applied.
Dummy coding is required for performing experimental research. Since we have dummy variables (i.e., a control/placebo group and a treatment group) in our model, the intercept has more meaning. Dummy coded variables have values of 0 for the reference/control/placebo group and 1 for the comparison/treatment group. Since the intercept is the expected mean value when X=0, it is the mean value only for the reference group (when all other X=0). Dummy coding a way to make the categorical variable into a series of dichotomous variables (variables that can have a value of zero or one only). For more details, please see the UCLA statistics site for more details (available at https://stats.idre.ucla.edu/spss/faq/coding-systems-for-categorical-variables-in-regression-analysis/).
It is suggested that “the effect of advertising appears non-linear, with an optimum between two and three exposures per week (Tellis, 1987).” For our example on the relationship between advertising exposures and product purchase below, we will be testing the relationship between advertising and product purchase using regression analysis. Our null hypothesis (usually denoted as H0) is that there is no relationship between advertising exposures and product purchases using regression analysis. The alternative hypothesis (usually denoted as H1) is that there is a relationship between advertising exposures and product purchases. The hypothesis test can be represented by the following notation:
Null Hypothesis: H0: β1 = 0 Alternative Hypothesis: H1: β1 ≠ 0
First, we will be creating a new variable that has a value of one for each observation at that level and zeroes for all others. In our example using the variable (Ads), the first new variable (Ads1) will have a value of one for each observation in which the consumers are exposed to the 1st ads campaign and zero for all other observations. Likewise, we create Ads2 when the consumers are exposed to the 1st ads campaign, and 0 otherwise, and Ads3 is 1 when the consumers are exposed to the 3rd ads campaign, and 0 otherwise. The level of the categorical variable that is coded as zero in the new variables is the reference level or the level to which all of the other levels are compared. In our example, it is the reference level Ads0. Our objective is to see which ads campaign lead to more product sales.
#setwd("C:/Users/zxu3/Documents/R/abtesting")
#Please install the following package if the package "readr" is not installed.
#install.packages("readr")
library(readr)
data <- read_csv("abtesting.csv")## Rows: 38 Columns: 2
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (2): Ads, Purchase
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.ls(data) # list the variables in the dataset## [1] "Ads" "Purchase"head(data) #list the first 6 rows of the dataset## # A tibble: 6 × 2
## Ads Purchase
## <dbl> <dbl>
## 1 1 113
## 2 0 83
## 3 0 52
## 4 1 119
## 5 1 188
## 6 0 99# creating the factor variable
data$Ads <- factor(data$Ads)
is.factor(data$Ads)## [1] TRUE# showing the first 15 rows of the variable "Ads"
data$Ads[1:15]## [1] 1 0 0 1 1 0 0 1 1 1 0 0 0 1 0
## Levels: 0 1#now we do the regression analysis and examine the results
summary(lm(Purchase~Ads, data = data))##
## Call:
## lm(formula = Purchase ~ Ads, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -57.000 -23.250 3.071 22.643 51.000
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 95.429 6.441 14.816 < 2e-16 ***
## Ads1 41.571 9.630 4.317 0.000118 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 29.52 on 36 degrees of freedom
## Multiple R-squared: 0.3411, Adjusted R-squared: 0.3228
## F-statistic: 18.64 on 1 and 36 DF, p-value: 0.0001184Since the p-value for X is .00018, which is less than .05, we reject the null hypothesis in favor of the alternative hypothesis.
The coefficient for Ads1 in the regression output is 41.57, which indicates that the 1st Advertising campaign is more effective (relative to the group who did not receive any advertising exposure).
Now the estimates for β0 and β1 are 95.43 and 41.57, respectively, leading to a prediction of average sales of 95.43 for the control group (group A) and a prediction of average sales which is 95.43 + 41.57*1 = 137 for the treatment group or the group of consumers who were exposed to the advertising campaign.
You will also find exactly the same coefficients using the Regression Data Analysis Tool in Excel. However, Excel also can’t handle large datasets (hundreds of thousands of records (Gapintelligence, 2020). Additionally, if you would like to perform the analysis and document the whole process (e.g., objectives, methods, hypotheses, results, and discussions), then using Rstudio with RMarkdown is probably one of the best choices.
Reference: Understanding R programming over Excel for Data Analysis https://www.gapintelligence.com/blog/understanding-r-programming-over-excel-for-data-analysis/
note: You can also perform this analysis using Excel We would like to see which campaign lead to more product sales in 2 different ways below.
#now we do an analysis for a predictor with 4 different levels
display <- read_csv("ab_testing.csv")## Rows: 80 Columns: 2
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (2): Ads, Purchase
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.ls(display) # list the variables in the dataset## [1] "Ads" "Purchase"head(display)## # A tibble: 6 × 2
## Ads Purchase
## <dbl> <dbl>
## 1 1 152
## 2 0 21
## 3 3 77
## 4 0 65
## 5 1 183
## 6 1 87# creating the factor variable
display$Ads <- factor(display$Ads)
is.factor(display$Ads)## [1] TRUE# showing the first 15 rows of the variable "Ads"
display$Ads[1:15]## [1] 1 0 3 0 1 1 2 2 2 0 3 3 0 2 3
## Levels: 0 1 2 3#now we do a regression analysis for a predictor with 4 different levels
summary(lm(Purchase~Ads, data = display))##
## Call:
## lm(formula = Purchase ~ Ads, data = display)
##
## Residuals:
## Min 1Q Median 3Q Max
## -50.095 -27.891 -0.227 24.773 65.905
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 55.381 6.472 8.557 9.41e-13 ***
## Ads1 75.714 9.152 8.273 3.31e-12 ***
## Ads2 36.557 9.842 3.715 0.000386 ***
## Ads3 -2.654 9.048 -0.293 0.770096
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 29.66 on 76 degrees of freedom
## Multiple R-squared: 0.5624, Adjusted R-squared: 0.5452
## F-statistic: 32.56 on 3 and 76 DF, p-value: 1.216e-13#Alternatively, you can also use the factor function within the lm function, saving the step of creating the factor variable first.
summary(lm(Purchase~ factor(Ads), data = display))##
## Call:
## lm(formula = Purchase ~ factor(Ads), data = display)
##
## Residuals:
## Min 1Q Median 3Q Max
## -50.095 -27.891 -0.227 24.773 65.905
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 55.381 6.472 8.557 9.41e-13 ***
## factor(Ads)1 75.714 9.152 8.273 3.31e-12 ***
## factor(Ads)2 36.557 9.842 3.715 0.000386 ***
## factor(Ads)3 -2.654 9.048 -0.293 0.770096
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 29.66 on 76 degrees of freedom
## Multiple R-squared: 0.5624, Adjusted R-squared: 0.5452
## F-statistic: 32.56 on 3 and 76 DF, p-value: 1.216e-13Answer: Focusing on the results for the regression model, we can observe that R squared for the multiple regression model is equal to 0.5452; This means that 54.52% of the variability in product purchase (or dependent variable) can be explained by the number of advertisement exposure (the independent variables because in this example we had 4 ads not 1). According to Learn something (Simple linear regression: Basic concepts part I) https://www.youtube.com/watch?v=BLRjywb0mes, R squared can help us determine how well does the regression line fits the data. In this case, although slightly more than half of the variability in product purchase can be explained by exposure to advertising, the results are not good enough. The marketing implications of this could be that even though the company is spending money on advertising exposure, product purchase variability is not increased or related to it. Yet, I may be wrong because I believe we rejected the null hypothesis favoring the alternative hypothesis which means that there is a relationship between the ads and product purchases.
Answer:
Answer:Answer: According to Gebhart 2016, five important reasons why R programming might add value to our career are: 1.-First, R programming can handle very large data sets. This article compares the advantages of R over Excel, and without a doubt, R programming has more benefits when it comes to storing and working with data. As the article implies, “Excel is limited in that there are only so many rows and columns per spreadsheet.” 2.-Another reason why R programming might add value to one’s career is that with this tool, data can be automated and calculated much faster than with Excel. The problem with Excel is that while working with huge data sets, the program becomes slower and stops working efficiently. R on the other hand, can work with a huge amount of data and continue performing efficiently. 3.- R can provide more complex and advanced data visualizations. Even though one can produce several types of basic graphs on Excel, R allows users to take very complicated data and turn it into a visual representation that it’s easy to understand. 4.- R is free and Excel is not. One of the greatest advantages of R over Excel is that R can be downloaded by anyone anywhere on any platform. 5.- Community libraries: R provides its users with a great community of R users who collaborate to extend each other’s knowledge. R promotes sharing of functions to expand libraries with new and different reproducible statistical functions.
Reference: Understanding R programming over Excel for Data Analysis https://www.gapintelligence.com/blog/understanding-r-programming-over-excel-for-data-analysis/
Answer: The article Your Step-by-Step Guide to A/B Testing with Google Optimize by The Daily Egg posted on June 10, 2021, suggests that most people struggle with their first A/B tests because they don’t know which tools to use and/or how to set up a test. Therefore, the article emphasizes that there is a great, free A/B testing tool available to every website owner. In this article, people can learn in more detail what an A/B testing is, why it is useful, and why do people use it. The tool that can help people conducting an A/B testing is Google Optimize which is claimed to be extremely easy to use; people can straightforwardly create an account process and the tool can easily integrate with Google Analytics. This means that you can analyze visitors who see each variation in Analytics and you can use Google Analytics to target the right demographics. Among all the benefits of using this tool, you can also learn how to create an account for Google Optimize with this article.
Reference used: https://rpubs.com/utjimmyx/abtesting2020
A/B Testing: Test Your Own Hypotheses & Prepare to be Wrong - Stuart Frisby
https://www.youtube.com/watch?v=VQpQ0YHSfqM&t=189s
Naiman 2020. Design Thinking as a Strategy for Innovation. https://www.creativityatwork.com/design-thinking-strategy-for-innovation/
Tellis 1987. Marketing Science. https://www.msi.org/reports/advertising-exposure-loyalty-and-brand-purchase-a-two-stage-model-of-choice/
https://stats.idre.ucla.edu/r/modules/coding-for-categorical-variables-in-regression-models/
Create an A/B test, https://support.google.com/optimize/answer/6211930?hl=en Experiments at Airbnb,https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7
Your Step-by-Step Guide to A/B Testing with Google Optimize, https://www.crazyegg.com/blog/ab-testing-google-analytics/
https://firebase.google.com/products/ab-testing
https://www.sitepoint.com/perform-ab-testing-google-optimize/
https://marketingplatform.google.com/about/optimize/features/