User should provide:

• Number of visitors to the control and Test web pages
• Number of conversion of the Control and Test web pages
• Significance level (%)

The Application will provide:

• A plot of the conversion rates for the Control and Test web pages
• Results from the statistical analysis.
  • p-value
  • Significance level
  • Answer to the question: is the difference statistical significant?. Here we also provide a recommendation to the user.

Imagine you have a web page and you want to increase the number of new users. So, you offer the users randomly two versions of your web page, control and test pages. You collect the number of visitors and the number of new users from each web page. Let say you have for the control page 200 visitors and only 55 become new users and for the test page you have 195 visitors and 69 become new users, is this increase in new users significant enough to change from the control to the test web page ? To answer this question you need to do a t-test. Below is the solution we offer in our Shiny app.

  n1 <- 200; c1 <- 55; n2 <- 195; c2 <- 69  
# conversion rates
  p1 <- c1/n1; p2 <- c2/n2
# Pooled sample proportion 
  pool <-  (p1 * n1 + p2 * n2) / (n1 + n2)  
# Standard error
  Se <-  sqrt( pool * ( 1. - pool ) * (( (1./n1) + (1./n2) )))

Now, we can calculate the p-value and compare with the confidence level chosen by the users. We usually take the confidence level equal to 5 %.

Z_score <- (p2 - p1) / Se
p_value <- pnorm(Z_score,lower.tail = F)*100

• If p-value < 5 %, we reject the null hypothesis that both web pages give similar conversion rate. It is time to move to the Test web page :-))
• If p-value > 5 %, we fail to reject the null hypothesis, so the difference in conversion rate is not statistical significant, so we should keep working with the old web page :-(

Here, the value of p-value is 4.5689413%, so we recommend that it is time to change to the new web page.