Null hypothesis: Both version A and B have the same probability of driving customer conversion. In other words, there is no effect or no difference between version A and B.
Alternative hypothesis: Hiding grade A products from display will contribute a positive impact to the company business performance by driving more conversions.
Before we continue with the alternative hypothesis, we need to confirm if the results are statistically significant by checking if a p-value that is lower than significance level (5%), or p-value < 0.05.
library(dplyr, warn.conflicts = FALSE)
options(dplyr.summarise.inform = FALSE)
library(pwr)## Warning: package 'pwr' was built under R version 4.0.3case_data <- read.csv("/Users/quan.ngo/Downloads/experiment_case_data.csv")
conversion_rate <- case_data %>% group_by(Variant) %>% summarise(conversion_rate= sum(Transactions)/ sum(Sessions)*100) %>% ungroup()
transaction <- case_data %>% group_by(Variant) %>% summarise(total_transaction = sum(Transactions)) %>% ungroup()
print(conversion_rate)## # A tibble: 2 x 2
## Variant conversion_rate
## <chr> <dbl>
## 1 A 0.564
## 2 B 0.568print(transaction)## # A tibble: 2 x 2
## Variant total_transaction
## <chr> <int>
## 1 A 6981
## 2 B 6983transaction_A <- case_data %>% filter(Variant== "A") %>% summarise(total_transaction=sum(Transactions))
transaction_B <- case_data %>% filter(Variant== "B") %>% summarise(total_transaction=sum(Transactions))
session_A <- case_data %>% filter(Variant== "A") %>% summarise(total_sessions=sum(Sessions))
session_B <- case_data %>% filter(Variant== "B") %>% summarise(total_sessions=sum(Sessions))
print(c(transaction_A,transaction_B,session_A,session_B))## $total_transaction
## [1] 6981
##
## $total_transaction
## [1] 6983
##
## $total_sessions
## [1] 1238010
##
## $total_sessions
## [1] 1229306prop.test(c(6981,6983),c(1238010,1229306))##
## 2-sample test for equality of proportions with continuity correction
##
## data: c(6981, 6983) out of c(1238010, 1229306)
## X-squared = 0.18194, df = 1, p-value = 0.6697
## alternative hypothesis: two.sided
## 95 percent confidence interval:
## -0.0002295754 0.0001464701
## sample estimates:
## prop 1 prop 2
## 0.005638888 0.005680441Here we can see p-value = 0.6697, which is much greater than 0.05.
Base on the high p-value of the experiment, the test results doesnt show strong significant difference, we have to accept the null hypothesis and reject the alternative one.
On the other words, selling grade A products does not have any significant impact in driving more ecommerce conversions.
There are 1238010 hits and 6981 conversions for variant A, 1229306 hits and 6983 conversions for variant B.
If the results shows strong statistical significance (p < 0.05), we can calculate uplift and probability to be the best of variant B to confirm if the alternative hypothesis can be accepted or not.
By now we know that selling grade A products does not impact the business performance at all, we can run another AB test but with variant B as hiding grade E products and analyze if it would contribute positively to the overall conversion.