A/B testing is a statistical method used to compare two versions of a webpage, app, or product feature.
In this example: - Version A = old signup page - Version B = new signup page
The goal is to determine whether version B leads to a higher signup rate.
## group visitors signups conversion_rate ## 1 A 1000 120 0.120 ## 2 B 1000 155 0.155
The sample conversion rate is calculated by
\[ \hat{p} = \frac{x}{n} \]
where: - \(x\) = number of successful signups - \(n\) = total number of visitors
For this example,
\[ \hat{p}_A = \frac{120}{1000} = 0.12 \qquad \hat{p}_B = \frac{155}{1000} = 0.155 \]
To test whether version B performs better than version A, we use:
\[ H_0: p_A = p_B \]
\[ H_a: p_B > p_A \]
The bar chart shows that version B has a higher observed conversion rate than version A.
Confidence intervals show the uncertainty in the estimated conversion rates.
This interactive plot shows that as sample size increases, the margin of error decreases.
prop.test( x = c(120, 155), n = c(1000, 1000), alternative = "less" )
This code performs a hypothesis test for two proportions.
## ## 2-sample test for equality of proportions with continuity correction ## ## data: c(120, 155) out of c(1000, 1000) ## X-squared = 4.8738, df = 1, p-value = 0.01363 ## alternative hypothesis: less ## 95 percent confidence interval: ## -1.000000000 -0.008700515 ## sample estimates: ## prop 1 prop 2 ## 0.120 0.155
The p-value is less than 0.05, so we reject the null hypothesis.
This suggests that version B performs significantly better than version A.