Introduction

Why Not Watch? (WNW), a new streaming company has started to grow and be exposed to more viewers, having gained a decently large viewership.
In a race to compete with other competitive streaming companies, WNW understands that the most optimal way to get viewers to watch from their platform is to have a really good recommendation engine that engages with viewers enough to retain them and maintain a high average of hours watched per user per day.
WNW wants to test their new recommendation engine, as well as understand is their any bias in the data collection, If there is any, can it be corrected and what are some improvements that can be made on future A/B tests.

Problem Statement

How does WNW’s new recommendation engine impact user engagement, and what biases exist in the data collection that can be corrected to improve future A/B tests?
To answer this, I will utilise statistical tools such as linear regression analysis to analyse data accurately and identify key trends and findings.

Data

Date: Date at which point this data was collated.

Gender: Male and Female Age: 18 - 55

Social_Metric: Numeric variable

Months since sign up: How long has the customer been with WNW

Demographic number: Female young: 1, male young: 2, female old: 3, male old: 4. (Young: 0≥35, Old: 36+)

Group A/B: Control group: A, treated group (group with new recommendation engine): B

No. of hours watched: the time viewers spent watching content on WNW

Analysis

 A   B 
880 120 
group_counts <- table(data$group)
print(group_counts)

F   M 
429 571

 1   2   3   4 
216 268 213 303

Hours watched in group A - preliminary distribution.

Hours watched in group A - correlation coefficient plot.

correlation_coefficient <- cor(group_A_data$hours_watched, group_A_data$age)

Hours watched in group B - preliminary distribution.

Hours watched in group B - correlation coefficient plot.

correlation_coefficient_B <- cor(group_B_data$hours_watched, group_B_data$age)

Summary of correlation between hours watched and age.

We can see that group A’s findings suggest a somewhat strong negative linear relationship. What this means is, that as the older the viewer is, the number of hours watched decreases, (and as the age decreases, the number of hours watched increases). So, older viewers in group A tend to watch fewer hours compared to younger viewers.

for the findings of group B, the coefficient again suggests a somewhat strong negative linear relationship. Thus you will find that like in group A, older viewers are usually watching less hours.

Hours watched by gender.

F   M 
429 571

Mean hours watched per gender:
F      M 
4.35  4.42

Hours watched by demographic.

count per demographic:
 1   2   3   4 
216 268 213 303

# Calculating mean hours per demographic:
  demographic hours_watched
1           1      5.067778
2           2      5.097015
3           3      3.626714
4           4      3.828267

#Calculating variance per demographic: 
  demographic hours_watched
1           1      1.118386
2           2      1.353282
3           3      1.525003
4           4      1.316087

Chi-squared test

data:  contingency_table
X-squared = 27.402, df = 3, p-value = 4.849e-0

There is a significant association between demographic and group (p-value = 4.84915e-06 )

Given that there is a significant association between demographic, and group, it would suggest that the demographics and groups were not randomly sorted. Now we will check for any disproportionalities to confirm if there is a bias.

Checking for bias

The graph appears to show an uneven proportion, particularily in demographic 4, group B. This disporportionality can effect the general representation of each demographic, in this case over representing older males and as a result this can influence the outcome of results. To remedy this, I will implement a propensity score matching to see if the change had an effect on mean hours watched.

Mean hours watched in group A: 4.336125 
Mean hours watched in group B: 4.810875

Two Sample t-test

data:  hours_watched by group
t = -3.3839, df = 238, p-value = 0.0008357
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.9280095 -0.2450738
sample estimates:
mean in group A mean in group B 
       4.224333        4.810875

The change of mean hours watched after the propensity score matching test indicates the bias was affecting the results.
Now we can do further analysis, namely find out effect size, minimum sample size, as well as perform a t-test on each demographic.

Demographic group analysis - Group 1:

#Effect calculation
[1] "1A: 5.0243842364532"
[1] "1B: 5.74538461538462"

effect_1 <- mean_1B - mean_1A
print("Effect calculation:")
[1] "Effect calculation:"
print(effect_1)
[1] 0.7210004

#Calculating minimum sample size:
[1] "Min sample size D1 = 9"

Welch Two Sample t-test
data:  d1b and d1a
t = 3.5361, df = 15.979, p-value = 0.001375
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 0.3649911       Inf
sample estimates:
mean of x mean of y 
 5.745385  5.024384

Group 2:

[1] "2A: 5.01440677966102"
[1] "2B: 5.70625"

[1] 0.6918432

[1] "Min sample size D2 = 11"

Welch Two Sample t-test
data:  d2b and d2a
t = 3.312, df = 40.666, p-value = 0.0009754
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 0.3402398       Inf
sample estimates:
mean of x mean of y 
 5.706250  5.014407

Group 3:

[1] "3A: 3.57847715736041"
[1] "3B: 4.220625"

[1] 0.6421478

[1] "Min sample size D3 = 15"

Welch Two Sample t-test
data:  d3b and d3a
t = 1.8349, df = 17.015, p-value = 0.04204
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 0.03336681        Inf
sample estimates:
mean of x mean of y 
 4.220625  3.578477

Group 4:

[1] "4A: 3.71918032786885"
[1] "4B: 4.27940677966102"

[1] 0.5602265

[1] "Min sample size D4 = 17"

Welch Two Sample t-test
data:  d4b and d4a
t = 3.3189, df = 85.094, p-value = 0.0006658
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 0.27952     Inf
sample estimates:
mean of x mean of y 
 4.279407  3.719180

Conclusion:

We know that:

Older age equals to less watched hours.
Men tend to watch more than women.
The recommendation engine did have a positive effect on all demographics, least being the older females. However, most being the younger males.
There was a bias in over representing older males.

I would suggest WNW implement the new recommendation engine especially for younger viewers, more so than for older viewers.

MATH2406 Assignment 3