E-news A/B Test Analysis

E-news Express A/B Test Analysis

Problem Statement

E-news Express wants to test if a new landing page design increases user engagement and conversions compared to the existing page.

Objectives

1. Test if users spend more time on the new landing page
2. Test if the new page has higher conversion rates
3. Check if conversion depends on preferred language
4. Check if time spent varies by language on the new page

---

Exploratory Data Analysis

Data Overview

## First 6 rows:

user_id	group	landing_page	time_spent_on_the_page	converted	language_preferred
546592	control	old	3.48	no	Spanish
546468	treatment	new	7.13	yes	English
546462	treatment	new	4.40	no	Spanish
546567	control	old	3.02	no	French
546459	treatment	new	4.75	yes	Spanish
546558	control	old	5.28	yes	English

## 
## 
## Last 6 rows:

	user_id	group	landing_page	time_spent_on_the_page	converted	language_preferred
95	546550	control	old	3.05	no	English
96	546446	treatment	new	5.15	no	Spanish
97	546544	control	old	6.52	yes	English
98	546472	treatment	new	7.07	yes	Spanish
99	546481	treatment	new	6.20	yes	Spanish
100	546483	treatment	new	5.86	yes	English

## 
## 
## Dataset shape:  100 rows, 6 columns

## 
## Missing values:  0

## Duplicate rows:  0

Statistical Summary

## ## Numerical Variable Summary:

Variable	Min	Q1	Median	Mean	Q3	Max
user_id	546443.00	546467.75	546492.50	546517.00	546567.25	546592.00
time_spent_on_the_page	0.19	3.88	5.42	5.38	7.02	10.71

## 
## 
## ## Categorical Variable Summary:

Variable	Unique_Values	Most_Common
group	2	control
landing_page	2	new
converted	2	yes
language_preferred	3	French

## 
## 
## ## Group Distribution:

group	landing_page	Count
control	old	50
treatment	new	50

Univariate Analysis

Bivariate Analysis

---

Question 1: Time Spent Comparison

Visual Analysis

Hypothesis Testing

H₀: μ_new ≤ μ_old (New page time ≤ Old page time)
H₁: μ_new > μ_old (New page time > Old page time)

Test: Independent t-test (one-tailed)
Significance level: α = 0.05

## New page: Mean = 6.22 minutes (n=50)

## Old page: Mean = 4.53 minutes (n=50)

## 
## T-statistic: 3.787

## 
## P-value: 0.00014

## 
## 95% CI Lower: 0.95

## 
## 
## Inference: REJECT H₀. Users spend significantly more time on the new page.

---

Question 2: Conversion Rate Comparison

Visual Analysis

Hypothesis Testing

H₀: p_new ≤ p_old (New page conversion ≤ Old page)
H₁: p_new > p_old (New page conversion > Old page)

Test: Two-proportion z-test (one-tailed)
Significance level: α = 0.05

## Conversion Table:

##      
##       no yes
##   new 17  33
##   old 29  21

## 
## New page conversion: 33 /50 = 66 %

## 
## Old page conversion: 21 /50 = 42 %

## 
## 
## Test statistic: 4.871

## 
## P-value: 0.01365

## 
## 
## Inference: REJECT H₀. New page has significantly higher conversion rate.

---

Question 3: Conversion and Language Relationship

Visual Analysis

Hypothesis Testing

H₀: Conversion and language are independent
H₁: Conversion and language are dependent

Test: Chi-square test of independence
Significance level: α = 0.05

## Contingency Table:

##          
##           no yes
##   English 11  21
##   French  19  15
##   Spanish 16  18

## 
## Chi-square statistic: 3.093

## 
## P-value: 0.21299

## 
## 
## Inference: FAIL TO REJECT H₀. Conversion is independent of preferred language.

---

Question 4: Time Spent on New Page by Language

Visual Analysis

Hypothesis Testing

H₀: Time spent is equal across all languages
H₁: Time spent differs by language

Test: Kruskal-Wallis test (non-parametric)
Significance level: α = 0.05

## Summary by Language (New Page Users):

language_preferred	n	mean_time	sd_time	median_time
English	16	6.66	1.98	6.86
French	17	6.20	1.93	5.42
Spanish	17	5.84	1.53	6.20

## 
## Kruskal-Wallis statistic: 1.044

## 
## P-value: 0.59346

## 
## 
## Inference: FAIL TO REJECT H₀. Time spent is similar across languages.

---

Conclusion

Summary of Results

Question	Result	P_value	Decision
Time spent comparison	Significant - New page better	0.00014	Reject H₀
Conversion rate comparison	Significant - New page better	0.01365	Reject H₀
Language-conversion relationship	Not significant - Independent	0.21299	Fail to reject H₀
Time by language (new page)	Not significant - Similar	0.59346	Fail to reject H₀

Business Recommendations

Based on the statistical analysis at 5% significance level:

1. IMPLEMENT NEW LANDING PAGE IMMEDIATELY
   • Engagement: Users spend 1.7 minutes more on new page (6.22 vs 4.53 minutes, p < 0.001)
   • Conversion: 57% higher conversion rate (66% vs 42%, p = 0.014)
2. FULL-SCALE DEPLOYMENT RECOMMENDED
   • Both key metrics (engagement and conversion) show statistically significant improvement
   • Low risk with proven results from 100-user test
3. NO LANGUAGE-SPECIFIC OPTIMIZATIONS NEEDED
   • Conversion rates similar across English, Spanish, and French users
   • Time spent on new page consistent across all language groups
4. MONITORING AND NEXT STEPS
   • Deploy to 100% of users starting immediately
   • Track performance for 30 days post-implementation
   • Monitor subscriber retention rates
   • Consider A/B testing additional page elements

Expected Business Impact: - 37% increase in user engagement time - 57% increase in conversion rate - Higher subscriber acquisition - Increased revenue from subscriptions

Decision: PROCEED WITH NEW LANDING PAGE

---

Prepared by: Omondi Anjeline Akoth
Date: December 11, 2025