Assessing the Impact of the New Recommendation Engine

A/B Test Analysis and Insights for Why Not Watch

Rafiu Olugbenga SALAMI - 4103596

Last updated: 19 June, 2025

Introduction

The business goal of this project is to assess the impact of a newly deployed recommendation engine on user engagement for the streaming platform “Why Not Watch?”(WNW). The key performance metric is “daily hours watched”. This metric directly determines ad revenue and earning potential. An A/B experiment was conducted: the Control group (A) continued with the existing recommendation system, while the Treatment group (B) represented the new engine after its commencement on 18 July 2023.

This report probes whether the new recommendation engine significantly increased engagement. Potential sampling biases were also assessed as well as the interconnections between user demographics and viewing behavior to provide useful recommendations for WNW’s leadership.

Context: Competition among digital streaming providers is intensifying; platforms are competing to capture viewer attention and engagement to drive ad revenue.
Objective: To evaluate whether the new recommendation engine (Group B) increases hours watched per user.
A/B test: Group A = control, Group B = treated users.
Approach: Explore and correct sampling bias, perform hypothesis testing and regression, visualize insights.

Problem Statement

Primary Question: Does the new recommendation engine trigger a statistically significant increase in daily hours watched?

-Statistical Approach:

To test for demographic balance for variables “age” and “gender.
To conduct two-sample t-test for difference in means.
To apply fit linear model - hours_watched ~ group + covariates.

The Sample Data,Variable Overview and Pre-processing Justification

Date: Date format represents the calendar date of each user’s session.
Group: Factor with two levels (Control and Treatment).
- While Control represents viewers on the existing recommendation engine, Treatment acts as users for the new engine. This is also presented as the main independent variable for hypothesis test and regression.
Gender: Factor with two levels (Female and Male). The variable is utilized to evaluate potential demographic bias.
Age: Integer that range between 18 and 55. Also used to assess age-related trends in engagement.
Social_metric : Integer (0–10).Represents composite engagement score based on past behavior.
Time_since_signup: Numeric (in months). Metric for user tenure with WNW.
Demographic: Categorical (1 to 4).
Hours_watched: Numeric. Represents the Primary dependent variable, measured in hours/day.

Justification:

All variables are retained due to their crucial impact on hours_watched and their relevant in testing the new recommendation engine.
Factors are converted appropriately to make visualizations and regression models successful.

# Loading and preprocessing data

wnw_dataset <- read_csv("streaming_data.csv") 

# Converting variable (date) from string to Date format

wnw_dataset$date <- as.Date(paste0(wnw_dataset$date, "-2023"), format="%d-%b-%Y")

 # Converting categorical variables (gender and group) to factors, with distinct labels of group variable for clarity.

wnw_dataset$group <- factor(wnw_dataset$group, levels = c("A", "B"), labels = c("Control", "Treatment"))

wnw_dataset$gender <- factor(wnw_dataset$gender)

# creating treatment period to present dates on or after the treatment rollout (18 July)

wnw_dataset$post_treatment <- wnw_dataset$date >= as.Date("2023-07-18")

str(wnw_dataset )

## spc_tbl_ [1,000 × 9] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
##  $ date             : Date[1:1000], format: "2023-07-01" "2023-07-01" ...
##  $ gender           : Factor w/ 2 levels "F","M": 1 1 1 2 2 2 1 2 2 2 ...
##  $ age              : num [1:1000] 28 32 39 52 25 51 53 42 41 20 ...
##  $ social_metric    : num [1:1000] 5 7 4 10 1 0 5 6 8 7 ...
##  $ time_since_signup: num [1:1000] 19.3 11.5 4.3 9.5 19.5 22.6 4.2 8.5 16.9 23 ...
##  $ demographic      : num [1:1000] 1 1 3 4 2 4 3 4 4 2 ...
##  $ group            : Factor w/ 2 levels "Control","Treatment": 1 1 1 1 1 1 1 1 1 1 ...
##  $ hours_watched    : num [1:1000] 4.08 2.99 5.74 4.13 4.68 3.4 3.07 2.77 2.24 5.39 ...
##  $ post_treatment   : logi [1:1000] FALSE FALSE FALSE FALSE FALSE FALSE ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   date = col_character(),
##   ..   gender = col_character(),
##   ..   age = col_double(),
##   ..   social_metric = col_double(),
##   ..   time_since_signup = col_double(),
##   ..   demographic = col_double(),
##   ..   group = col_character(),
##   ..   hours_watched = col_double()
##   .. )
##  - attr(*, "problems")=<externalptr>

summary(wnw_dataset )

##       date            gender       age        social_metric   
##  Min.   :2023-07-01   F:429   Min.   :18.00   Min.   : 0.000  
##  1st Qu.:2023-07-08   M:571   1st Qu.:28.00   1st Qu.: 2.000  
##  Median :2023-07-16           Median :36.00   Median : 5.000  
##  Mean   :2023-07-16           Mean   :36.49   Mean   : 4.911  
##  3rd Qu.:2023-07-24           3rd Qu.:46.00   3rd Qu.: 8.000  
##  Max.   :2023-07-31           Max.   :55.00   Max.   :10.000  
##  time_since_signup  demographic          group     hours_watched  
##  Min.   : 0.00     Min.   :1.000   Control  :880   Min.   :0.500  
##  1st Qu.: 5.70     1st Qu.:2.000   Treatment:120   1st Qu.:3.530  
##  Median :11.80     Median :3.000                   Median :4.415  
##  Mean   :11.97     Mean   :2.603                   Mean   :4.393  
##  3rd Qu.:18.70     3rd Qu.:4.000                   3rd Qu.:5.322  
##  Max.   :24.00     Max.   :4.000                   Max.   :8.300  
##  post_treatment 
##  Mode :logical  
##  FALSE:548      
##  TRUE :452      
##                 
##                 
##

# Checking for missing data

sum(is.na(wnw_dataset))

## [1] 0

Data Analysis and Bias Risks

Biases Identified:

Group imbalance: Control (A) has 880 users; Treatment (B) only 120 users. The test is unbalanced and can make the statistical power weak
Gender imbalance: Treatment group significantly skewed towards males (76%) compared to Control group (55%). This introduces confounding by gender.

Recommended Correction Strategies:

Stratified random sampling for future A/B tests should be used to guarantee demographic balance across groups.
Controlled Pre-experiment Data (CUPED) should be implemented in future assessments of CUPED to enhance experiment accuracy and minimise variance.

#  Sample Balance Checks
# Count number of users are in each group
count(wnw_dataset, group)

# Checking gender balance across groups
chisq_test <- chisq.test(table(wnw_dataset$group, wnw_dataset$gender))
tidy(chisq_test)

# Check if average age differs between groups
summary(t.test(age ~ group, data = wnw_dataset))

##             Length Class  Mode     
## statistic   1      -none- numeric  
## parameter   1      -none- numeric  
## p.value     1      -none- numeric  
## conf.int    2      -none- numeric  
## estimate    2      -none- numeric  
## null.value  1      -none- numeric  
## stderr      1      -none- numeric  
## alternative 1      -none- character
## method      1      -none- character
## data.name   1      -none- character

# Age t-test
t_age <- t.test(age ~ group, data = wnw_dataset)
tidy(t_age)

# Bias Correction (Post‑stratification)

# Define age groups-# Create age groups for stratification
breaks <- seq(18, 60, by=5)
wnw_dataset <- wnw_dataset %>% mutate(age_group = cut(age, breaks = breaks))

# Calculate population distribution by age group and gender
population_dist <- wnw_dataset %>% count(age_group, gender) %>% mutate(propulation = n/sum(n))

 # Hypothesis Testing
# Compare means with Welch t-test (does not assume equal variance)
t_test <- t.test(hours_watched ~ group, data = wnw_dataset, var.equal = FALSE)


tidy(t_test)

#  Multiple Linear Regression Analysis
model <- lm(hours_watched ~ group + gender + age + social_metric + time_since_signup + demographic, data = wnw_dataset)
summary(model)

## 
## Call:
## lm(formula = hours_watched ~ group + gender + age + social_metric + 
##     time_since_signup + demographic, data = wnw_dataset)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.5901 -0.6402 -0.0194  0.7024  2.8283 
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)        6.367203   0.157424  40.446  < 2e-16 ***
## groupTreatment     0.645976   0.102099   6.327 3.78e-10 ***
## genderM            0.096527   0.095112   1.015    0.310    
## age               -0.065295   0.006013 -10.859  < 2e-16 ***
## social_metric      0.095376   0.010958   8.704  < 2e-16 ***
## time_since_signup  0.003540   0.004541   0.780    0.436    
## demographic       -0.090338   0.064348  -1.404    0.161    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.034 on 993 degrees of freedom
## Multiple R-squared:  0.402,  Adjusted R-squared:  0.3984 
## F-statistic: 111.3 on 6 and 993 DF,  p-value: < 2.2e-16

tidy(model)

glance(model)

# Bar plot with error bars
coef_wnw_dataset <- tidy(model, conf.int = TRUE)

ggplot(coef_wnw_dataset, aes(x = reorder(term, estimate), y = estimate)) +
  geom_bar(stat = 'identity') +
  geom_errorbar(aes(ymin = estimate - std.error, ymax = estimate + std.error), width = 0.2) +
  coord_flip() +
  labs(title = "Regression Coefficients", x = "Term", y = "Estimate")

# 5. Visualization
# Group Distribution

ggplot(wnw_dataset, aes(x = group)) + geom_bar() + theme_minimal()

# Gender Distribution by Group
ggplot(wnw_dataset, aes(x = group, fill = gender)) + geom_bar(position = "stack") + theme_minimal()

# Hours Watched by Group

ggplot(wnw_dataset, aes(group, hours_watched, fill = group)) +
  geom_boxplot(alpha = 0.6) +
  labs(x = "Group", y = "Hours Watched")

# Age Distribution by Group

ggplot(wnw_dataset, aes(age, fill = group)) +
  geom_histogram(position = "identity", alpha = 0.5, bins = 20) +
  labs(x = "Age", y = "Count")  +
  theme_minimal()

# Social Metric vs. Hours Watched by Group

ggplot(wnw_dataset, aes(social_metric, hours_watched, color = group)) +
  geom_point(alpha = 0.4) +
  geom_smooth(method = "lm", se = FALSE, formula = y ~ x) +
  facet_wrap(~ age_group) +
  labs(x = "Social Metric", y = "Hours Watched")

# Coefficient Plot

tidy(model, conf.int = TRUE) %>%
  ggplot(aes(x = estimate, y = reorder(term, estimate))) +
  geom_point() +
  geom_errorbarh(aes(xmin = conf.low, xmax = conf.high), height = 0.2) +
  geom_vline(xintercept = 0, linetype = "dashed", color = "red") +
  theme_minimal()

# Residual Plot
plot(model, which = 1)

Discussion

Key findings:

Treatment group users watched significantly more hours per day than Control group users (p < 0.001). The observed difference in means is highly unlikely to be due to random chance.

Group B (treatment) shows a statistically significant uplift of ~0.65 h/day. Users in the treatment group watch **~0.65 more hours per day. Therefore, more hours watched per user lead to more ad reservations, and translate to higher total ad impressions.

To adjust for confounding variables and isolate the effect of the treatment,fit a multiple linear regression model was applied.

The regression model helps determine which variables significantly influence viewing time, and quantifies the treatment effect while holding other factors constant.

Residuals are approximately symmetric and randomly dispersed, supporting the model’s appropriateness.

References

Kutner, M., Nachtsheim, C., & Neter, J. (2004). Applied Linear Regression Models. 4th ed. McGraw-Hill Education.
Tidyverse packages. (2024). Data science packages for R. https://www.tidyverse.org/