Introduction

This experiment recruits participants to a chatbot survey using Facebook ads. After clicking on the Facebook ad, a participant is randomized into one of 5 treatment arms (the appendix considers 7 treatment arms).

The power analysis varies along two dimensions:

  • hypothesized outcomes for each treatment arm (and hence, treatment effects)
  • number of observations (clicks)

There are three outcomes, which are correlated: Pr(unvax complete|click), Pr(unvax website click|complete), and Pr(unvax useful free text|complete). We have an estimate of 14% for Pr(unvax complete|click) from the pilots and a rough estimate of 50% for Pr(unvax useful free text|complete), but we do not have an estimate for Pr(unvax website click|complete).

In the following power calculations, the hypothesized level of Pr(unvax complete|click) ranges from 0.04 to 0.20, with treatment effects ranging from 2 to 8 percentage points when comparing adjacent treatments (i.e. treatment 1 to 2, or 2 to 3). The hypothesized level of Pr(unvax useful free text|complete) ranges from 0.35 to 0.55, with treatment effects ranging from 2 to 10 percentage points when comparing adjacent treatments. The hypothesized level of Pr(unvax website click|complete) ranges from 0.05 to 0.35, with treatment effects ranging from 3 to 10 percentage points when comparing adjacent treatments. I do not anticipate that the Pr(unvax website click|complete) outcome will necessarily achieve a high of 0.35, but the outcome we end up using in its place (based on pilot results) should.

For the number of observations, we consider 50k to 150k clicks in 25k intervals.

Hypotheses

The pre-analysis plan specifies two research questions, with five hypotheses (one in two parts) to answer them. I list the research questions and accompanying hypotheses below.

  1. Is a chatbot a more effective tool for eliciting information than a traditional survey?
    • T1 = T4
  2. What quality of a chatbot makes it a more effective tool?
    • T1 = T2
    • T2 = T3
    • T3 = T4
    • T4 = T5
    • (T2 - T1) = (T4 - T5)

Note: The current code only includes the first four hypotheses for research question 2. It will be updated soon.

Key Takeaways

  • We are well-powered to answer research question 1 with 50k clicks for all outcomes, assuming a 2-3pp difference between adjacent treatment arms (so an 8 to 11pp difference between treatments 1 and 4).
  • For Pr(unvax complete|click), we are well-powered to detect 2pp differences between adjacent treatment arms at 50k clicks.
  • For Pr(unvax useful free text|complete), we are well-powered to detect 5pp differences between adjacent treatment arms at 75k clicks.
  • For Pr(unvax website click|complete), we are well-powered to detect 5pp differences between adjacent treatment arms at 50k clicks and 3pp differences at 150k clicks.

I recommend that the main experiment aim for 75k clicks at an approximate cost of $15k at $0.146 per click (plus mobile airtime costs which would amount to no more than $1500 at the most optimistic completion rate of 20%).

Power Calculations

To assess the power for testing multiple correlated hypotheses, I simulate data, test the hypotheses, correct the p-values, and repeat for r repetitions, then calculate the proportion of repetitions in which I reject each hypothesis. The variables needed in the simulated data are the treatment assignment and the three outcomes.

The data generation process is as follows:

  • Assign each observation to a treatment group. (treatment variable)
  • Take a draw from a binomial distribution with probability equal to the hypothesized unvax completion probability for the observation’s treatment group. (completion variable)
  • For observations who completed the survey, take a draw from a binomial distribution with probability equal to the hypothesized unvax website click probability for the observation’s treatment group. (webclick variable)
    • For observations who completed the survey, take a draw from a binomial distribution with probability equal to the hypothesized free text proportion for the observation’s treatment group. (free text variable)

Power is calculated using the following parameters:

  • \(\alpha=0.05\)
  • replications \(r=1000\)
  • number of observations (clicks) \(n=\{50k, 75k,100k,125k,150k\}\) for a cost of \(\{11k,15k,22k,29k\}\) at $0.146 per click.

Results are reported for uncorrected p-values and for p-values corrected using Bonferonni, Holm, and Hochberg.

Hypothesis

Hypothesis 1. Participants in Treatment 4 provide more information, higher-quality information, and are more receptive to new information about the vaccine compared to participants in Treatment 1.

Hypothesis 2. Participants in Treatment 3 provide more information, higher-quality information, and are more receptive to new information about the vaccine compared to participants in Treatment 2.

Hypothesis 3. Participants in Treatment 4 provide more information, higher-quality information, and are more receptive to new information about the vaccine compared to participants in Treatment 3.

Hypothesis 4. Participants in Treatment 4 provide more information, higher-quality information, and are more receptive to new information about the vaccine compared to participants in Treatment 5.

Hypothesis 5. (A) Participants in Treatment 2 provide more information, higher-quality information, and are more receptive to new information about the vaccine compared to participants in Treatment 1. (B) Moreover, the difference between participants in Treatment 2 and participants in Treatment 1 is greater than the difference between participants in Treatment 4 and participants in Treatment 5.

Treatment Weights

set.seed(94305)
### DEFINE PARAMETERS ###

n <- seq(10000, 40000, 10000)

# outcomes
outcomes <- matrix(c(   0.08,   0.12,   0.14,   0.16,   0.14, # Pr(unvax complete|click)
                 0.14,  0.19,   0.22,   0.25,   0.22, # Pr(unvax web click|complete)
                 0.44,  0.48,   0.5,    0.52,   0.5), # Pr(useful free text| complete)
                  nrow = 3, byrow = T)


outcomes2 <-  matrix(c( 0.04,   0.12,   0.16,   0.2,    0.16, # Pr(unvax complete|click)
                 0.05,  0.15,   0.2,    0.25,   0.2, # Pr(unvax web click|complete)
                 0.35,  0.45,   0.5,    0.55,   0.5), # Pr(useful free text| complete)
                  nrow = 3, byrow = T)

outcomes3 <-  matrix(c( 0.04,   0.12,   0.16,   0.2,    0.14, # Pr(unvax complete|click)
                 0.22,  0.29,   0.32,   0.35,   0.32,
                    0.3,    0.4,    0.45,   0.55,   0.5), # Pr(useful free text| complete)
                  nrow = 3, byrow = T)

outcomes_list <- list(outcomes, outcomes2, outcomes3)

# treatments
t_list <- lapply(outcomes_list, FUN=function(x){c(1:ncol(x))})

# alpha
alpha <- 0.05

# number of replications
r <- 1000

### DEFINE FUNCTION ###

trial_subset <- function(N, outcomes, t){
  
  ### Simulate Data
  
  # create dataframe
  d <- data.frame(matrix(nrow = N, ncol = 0))
  
  # treatment assignment
  d$treat <- sample(t, N, replace=T, prob = c(3/14, 3/14, 2/14, 4/14, 2/14)) 
  
  # generate data with completion outcome
  d <- d %>% 
    mutate(complete = rbinom(n(),1,outcomes[1,treat])) 
  
  ## Outcome 1: completion
  
  # make df with columns for treatment and completion
  # this is for diff in diff pvalues
  d_complete <- d 
  
  # aggregate completion counts and click counts by treatment
  # this is for the pairwise prop tests
  d_complete_agg <- d_complete %>% 
    group_by(treat) %>% 
    summarise(
      complete = sum(complete),
      clicks = n()
    )

  ## Outcome 2: webclicks
  
  # make df with columns for treatment and webclick
  # conditional on completes
  # this is for diff in diff pvalues
  d_website <- d %>% 
    filter(complete == 1) %>% 
    mutate(webclick = rbinom(n(),1,outcomes[2,treat]))
  
  # aggregate webclick counts and completes counts by treatment
  # this is for the pairwise prop tests  
  d_website_agg <- d_website %>% 
    group_by(treat) %>% 
    summarise(
      webclick = sum(webclick),
      completes = n()
    )

  ## Outcome 3: Useful free text
  
  # make df with columns for treatment and useful
  # conditional on completes
  # this is for diff in diff pvalues
  d_useful <- d %>% 
    filter(complete == 1) %>% 
    mutate(useful = rbinom(n(),1,outcomes[3,treat])) 
  
  # aggregate webclick counts and completes counts by treatment
  # this is for the pairwise prop tests  
  d_useful_agg <- d_useful %>% 
    group_by(treat) %>% 
    summarise(
      useful = sum(useful), 
      completes = n()
    )
  
  ### Calculate p-values
  
  # create matrix for p-values
  # number of outcomes (3) by number of hypothesis (6)
  p_values <- matrix(nrow = nrow(outcomes),ncol = 6)
  
  
  # calculate for completions outcome
  
  # first get pvalues for usual comparisons (tA vs tB)
  d_subset <- cbind(d_complete_agg[,2], d_complete_agg[,3])
  p <- pairwise.prop.test(d_subset[,1], d_subset[,2], 
                                      p.adj = "none")$p.value
  p_values[1, 1:5] <- c(p["4", "1"], 
                    p["3", "2"], 
                    p["4", "3"], 
                    p["5", "4"], 
                    p["2", "1"])
  
  # now get pvalue for diff in diff
  model = lm(complete~0+treat, d_complete %>% 
               # remove observations with treatments not relevant to diff in diff
               filter(treat != 3) %>% 
               # make numeric categories a character so lm doesn't treat them as continuous
               mutate(treat = paste("W", treat, sep = ""))) 
  
  # run hypothesis test for diff in diff
  hyp = glht(model, "(treatW2 - treatW1) - (treatW5 - treatW4) = 0")
  
  # extract pvalue
  pval <- summary(hyp)$test$pvalues
  
  p_values[1, 6] <- c(pval)
  
  # calculate for webclicks outcome
  d_subset <- cbind(d_website_agg[,2], d_website_agg[,3])
  p <- pairwise.prop.test(d_subset[,1], d_subset[,2], 
                                      p.adj = "none")$p.value
  p_values[2, 1:5] <- c(p["4", "1"], 
                    p["3", "2"], 
                    p["4", "3"], 
                    p["5", "4"], 
                    p["2", "1"])
  
  ## diff in diff
  model = lm(webclick~0+treat, d_website %>% filter(treat != 3) %>% mutate(treat = paste("W", treat, sep = "")))

  hyp = glht(model, "(treatW2 - treatW1) - (treatW5 - treatW4) = 0")
  pval <- summary(hyp)$test$pvalues
  
  p_values[2, 6] <- c(pval)
  
  
  # calculate for useful information outcome
  d_subset <- cbind(d_useful_agg[,2], d_useful_agg[,3])
  p <- pairwise.prop.test(d_subset[,1], d_subset[,2], 
                                      p.adj = "none")$p.value
  
  p_values[3, 1:5] <- c(p["4", "1"], 
                    p["3", "2"], 
                    p["4", "3"], 
                    p["5", "4"], 
                    p["2", "1"])
  
    ## diff in diff
  model = lm(useful~0+treat, d_useful %>% filter(treat != 3) %>% mutate(treat = paste("W", treat, sep = "")))

  hyp = glht(model, "(treatW2 - treatW1) - (treatW5 - treatW4) = 0")
  pval <- summary(hyp)$test$pvalues
  
  p_values[3, 6] <- c(pval)
  ### MHT corrections
  
  return(rbind(p.adjust(p_values[1,], method = "none"),
               p.adjust(p_values[1,], method = "bonferroni"),
               p.adjust(p_values[1,], method = "holm"),
               p.adjust(p_values[1,], method = "hochberg"),
               p.adjust(p_values[2,], method = "none"),
               p.adjust(p_values[2,], method = "bonferroni"),
               p.adjust(p_values[2,], method = "holm"),
               p.adjust(p_values[2,], method = "hochberg"),
               p.adjust(p_values[3,], method = "none"),
               p.adjust(p_values[3,], method = "bonferroni"),
               p.adjust(p_values[3,], method = "holm"),
               p.adjust(p_values[3,], method = "hochberg")))
}

### POWER CALCULATIONS ###

for (i in n){
  
    cat('###',i,' \n')

  
  for(o in 1:length(outcomes_list)){
    treatments <- t_list[[o]]
    outcome_matrix  <- outcomes_list[[o]]
    
    # complete r replications of n trials
    results <- replicate(r,  trial_subset(i, outcomes = outcome_matrix, t = treatments))
    
    # create matrix to store power calculations
    power_calc <- matrix(NA, 
                         nrow = nrow(outcome_matrix)*4, 
                         ncol = 6) 
    
    # for each pairwise treatment comparison
    for(j in 1:6){
      
      # for each mht correction
      for(k in 1:4){
        
        # calculate power for completions outcome
        power_calc[k, j] <- sum(results[k, j, ] < alpha)/r
        
        # calculate power for website clicks outcome
        power_calc[4+k, j] <- sum(results[k+4, j, ] < alpha)/r
        
        # calculate power for useful informaton  outcome
        power_calc[8+k, j] <- sum(results[k+8, j, ] < alpha)/r
        
      }
    }
    
    # Generate Graph
  
    
        power_calc <- data.frame(outcome = c(rep('Unvax completes|Click',4),
                           rep('Unvax web clicks|Complete', 4),
                           rep('Unvax useful freetext|Complete', 4)), 
                         test = rep(c("None","Bonferonni", "Holm", "Hochberg"), 3),
                         power_calc)
   
    colnames(power_calc) <- c("Outcome","Correction", 
                              'T1 v. T4', 'T2 v. T3','T3 v. T4','T4 v. T5', 'T1 v. T2', 'T2 - T1 v. T5 - T4')
 
      
    kable(power_calc,  format = "html", caption = paste("Power Calculations: ",i/1000," Thousand Clicks", "Matrix", o), digits = 3)%>%
  kable_styling()%>%
  collapse_rows(columns = 1)%>%
      scroll_box(height = "100%")%>%
      print()
  }  
}

10000

Power Calculations: 10 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.389 0.405 0.385 0.990 0.988
Bonferonni 1.000 0.193 0.178 0.175 0.953 0.934
Holm 1.000 0.277 0.275 0.282 0.973 0.947
Hochberg 1.000 0.289 0.287 0.294 0.973 0.948
Unvax web clicks|Complete None 0.850 0.090 0.101 0.104 0.196 0.314
Bonferonni 0.610 0.026 0.025 0.026 0.063 0.119
Holm 0.613 0.028 0.034 0.032 0.079 0.135
Hochberg 0.613 0.028 0.034 0.032 0.080 0.136
Unvax useful freetext|Complete None 0.389 0.062 0.061 0.067 0.109 0.143
Bonferonni 0.170 0.021 0.013 0.011 0.028 0.036
Holm 0.171 0.021 0.015 0.013 0.033 0.042
Hochberg 0.171 0.021 0.015 0.013 0.034 0.043
Power Calculations: 10 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.907 0.885 0.909 1.000 1.000
Bonferonni 1.000 0.752 0.679 0.724 1.000 1.000
Holm 1.000 0.900 0.869 0.881 1.000 1.000
Hochberg 1.000 0.905 0.878 0.893 1.000 1.000
Unvax web clicks|Complete None 0.999 0.278 0.273 0.284 0.667 0.803
Bonferonni 0.982 0.092 0.094 0.106 0.330 0.466
Holm 0.983 0.127 0.133 0.151 0.401 0.520
Hochberg 0.983 0.130 0.142 0.159 0.412 0.528
Unvax useful freetext|Complete None 0.907 0.154 0.219 0.235 0.309 0.548
Bonferonni 0.753 0.035 0.077 0.081 0.120 0.267
Holm 0.755 0.045 0.099 0.108 0.149 0.299
Hochberg 0.755 0.046 0.103 0.112 0.151 0.301
Power Calculations: 10 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.908 0.877 0.997 1.000 1.000
Bonferonni 1.000 0.758 0.694 0.986 1.000 1.000
Holm 1.000 0.906 0.874 0.994 1.000 1.000
Hochberg 1.000 0.908 0.876 0.994 1.000 1.000
Unvax web clicks|Complete None 0.640 0.087 0.122 0.101 0.199 0.290
Bonferonni 0.359 0.027 0.031 0.025 0.056 0.105
Holm 0.366 0.028 0.038 0.029 0.071 0.118
Hochberg 0.367 0.028 0.040 0.030 0.073 0.119
Unvax useful freetext|Complete None 0.987 0.184 0.685 0.200 0.323 0.525
Bonferonni 0.941 0.058 0.417 0.060 0.126 0.259
Holm 0.942 0.072 0.458 0.090 0.162 0.310
Hochberg 0.942 0.075 0.460 0.092 0.168 0.315

20000

Power Calculations: 20 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.688 0.685 0.640 1.000 1.000
Bonferonni 1.000 0.421 0.407 0.370 1.000 0.999
Holm 1.000 0.611 0.588 0.553 1.000 1.000
Hochberg 1.000 0.629 0.604 0.570 1.000 1.000
Unvax web clicks|Complete None 0.992 0.187 0.181 0.188 0.436 0.585
Bonferonni 0.953 0.065 0.044 0.061 0.214 0.311
Holm 0.954 0.077 0.064 0.091 0.243 0.344
Hochberg 0.954 0.077 0.067 0.093 0.249 0.349
Unvax useful freetext|Complete None 0.682 0.078 0.086 0.099 0.183 0.272
Bonferonni 0.419 0.020 0.018 0.023 0.061 0.095
Holm 0.420 0.020 0.022 0.029 0.074 0.110
Hochberg 0.420 0.020 0.022 0.032 0.075 0.112
Power Calculations: 20 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.997 0.992 0.994 1.000 1.000
Bonferonni 1.000 0.981 0.958 0.962 1.000 1.000
Holm 1.000 0.997 0.991 0.993 1.000 1.000
Hochberg 1.000 0.997 0.992 0.994 1.000 1.000
Unvax web clicks|Complete None 1.000 0.512 0.541 0.518 0.950 0.979
Bonferonni 1.000 0.265 0.288 0.250 0.803 0.869
Holm 1.000 0.387 0.409 0.386 0.881 0.909
Hochberg 1.000 0.408 0.430 0.409 0.890 0.914
Unvax useful freetext|Complete None 0.998 0.328 0.408 0.417 0.614 0.837
Bonferonni 0.985 0.120 0.189 0.188 0.326 0.605
Holm 0.988 0.160 0.239 0.256 0.410 0.645
Hochberg 0.988 0.176 0.250 0.267 0.422 0.650
Power Calculations: 20 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.996 0.991 1.000 1.000 1.000
Bonferonni 1.000 0.980 0.967 1.000 1.000 1.000
Holm 1.000 0.996 0.991 1.000 1.000 1.000
Hochberg 1.000 0.996 0.991 1.000 1.000 1.000
Unvax web clicks|Complete None 0.921 0.158 0.210 0.175 0.390 0.531
Bonferonni 0.773 0.052 0.056 0.045 0.171 0.259
Holm 0.777 0.058 0.076 0.072 0.208 0.294
Hochberg 0.777 0.060 0.076 0.073 0.209 0.296
Unvax useful freetext|Complete None 1.000 0.312 0.955 0.386 0.629 0.837
Bonferonni 1.000 0.143 0.837 0.184 0.335 0.619
Holm 1.000 0.202 0.879 0.274 0.448 0.685
Hochberg 1.000 0.218 0.893 0.291 0.472 0.691

30000

Power Calculations: 30 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.851 0.845 0.823 1.000 1.000
Bonferonni 1.000 0.636 0.625 0.602 1.000 1.000
Holm 1.000 0.832 0.812 0.786 1.000 1.000
Hochberg 1.000 0.841 0.824 0.799 1.000 1.000
Unvax web clicks|Complete None 1.000 0.241 0.273 0.257 0.628 0.741
Bonferonni 1.000 0.083 0.082 0.084 0.350 0.486
Holm 1.000 0.101 0.121 0.134 0.408 0.537
Hochberg 1.000 0.109 0.126 0.143 0.412 0.541
Unvax useful freetext|Complete None 0.849 0.108 0.113 0.116 0.262 0.358
Bonferonni 0.659 0.030 0.031 0.031 0.096 0.142
Holm 0.663 0.033 0.039 0.042 0.109 0.160
Hochberg 0.663 0.033 0.039 0.042 0.109 0.160
Power Calculations: 30 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1 1.000 1.000 1.000 1.000 1.000
Bonferonni 1 1.000 0.999 0.999 1.000 1.000
Holm 1 1.000 1.000 1.000 1.000 1.000
Hochberg 1 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 1 0.700 0.741 0.727 0.990 0.999
Bonferonni 1 0.440 0.470 0.482 0.972 0.988
Holm 1 0.634 0.674 0.646 0.987 0.993
Hochberg 1 0.651 0.693 0.664 0.987 0.993
Unvax useful freetext|Complete None 1 0.450 0.567 0.578 0.818 0.950
Bonferonni 1 0.218 0.322 0.306 0.583 0.850
Holm 1 0.310 0.428 0.424 0.694 0.880
Hochberg 1 0.330 0.449 0.442 0.706 0.884
Power Calculations: 30 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 1.000 1.000 1.000 1.000 1.000
Bonferonni 1.000 0.999 1.000 1.000 1.000 1.000
Holm 1.000 1.000 1.000 1.000 1.000 1.000
Hochberg 1.000 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 0.988 0.202 0.271 0.263 0.564 0.720
Bonferonni 0.941 0.059 0.095 0.087 0.302 0.458
Holm 0.942 0.082 0.131 0.138 0.357 0.506
Hochberg 0.942 0.087 0.132 0.140 0.362 0.509
Unvax useful freetext|Complete None 1.000 0.460 0.996 0.533 0.793 0.935
Bonferonni 1.000 0.194 0.980 0.281 0.554 0.810
Holm 1.000 0.351 0.984 0.436 0.688 0.866
Hochberg 1.000 0.371 0.984 0.454 0.704 0.868

40000

Power Calculations: 40 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.937 0.934 0.928 1.000 1.000
Bonferonni 1.000 0.807 0.804 0.766 1.000 1.000
Holm 1.000 0.933 0.924 0.915 1.000 1.000
Hochberg 1.000 0.934 0.929 0.921 1.000 1.000
Unvax web clicks|Complete None 1.000 0.331 0.344 0.392 0.771 0.868
Bonferonni 1.000 0.135 0.140 0.139 0.520 0.678
Holm 1.000 0.178 0.191 0.221 0.602 0.722
Hochberg 1.000 0.189 0.204 0.232 0.609 0.726
Unvax useful freetext|Complete None 0.950 0.113 0.144 0.151 0.358 0.442
Bonferonni 0.822 0.034 0.031 0.040 0.139 0.217
Holm 0.822 0.039 0.045 0.056 0.168 0.239
Hochberg 0.822 0.039 0.048 0.057 0.172 0.242
Power Calculations: 40 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1 1.000 1.000 1.000 1.000 1.000
Bonferonni 1 1.000 1.000 1.000 1.000 1.000
Holm 1 1.000 1.000 1.000 1.000 1.000
Hochberg 1 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 1 0.818 0.838 0.840 1.000 1.000
Bonferonni 1 0.596 0.629 0.628 0.997 0.999
Holm 1 0.782 0.793 0.796 1.000 1.000
Hochberg 1 0.796 0.804 0.812 1.000 1.000
Unvax useful freetext|Complete None 1 0.558 0.697 0.711 0.903 0.985
Bonferonni 1 0.296 0.474 0.466 0.708 0.932
Holm 1 0.467 0.601 0.606 0.833 0.950
Hochberg 1 0.488 0.623 0.621 0.844 0.952
Power Calculations: 40 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 1.000 1.000 1.000 1.000 1.000
Bonferonni 1.000 1.000 1.000 1.000 1.000 1.000
Holm 1.000 1.000 1.000 1.000 1.000 1.000
Hochberg 1.000 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 0.999 0.255 0.354 0.307 0.705 0.819
Bonferonni 0.986 0.108 0.154 0.128 0.433 0.597
Holm 0.986 0.138 0.197 0.189 0.501 0.642
Hochberg 0.986 0.145 0.204 0.197 0.506 0.643
Unvax useful freetext|Complete None 1.000 0.606 0.999 0.662 0.907 0.983
Bonferonni 1.000 0.329 0.991 0.405 0.748 0.925
Holm 1.000 0.533 0.998 0.584 0.863 0.956
Hochberg 1.000 0.554 0.998 0.605 0.876 0.958

Without Treatment Weights

set.seed(94305)
### DEFINE PARAMETERS ###

n <- seq(10000, 40000, 10000)

# outcomes
outcomes <- matrix(c(   0.08,   0.12,   0.14,   0.16,   0.14, # Pr(unvax complete|click)
                 0.14,  0.19,   0.22,   0.25,   0.22, # Pr(unvax web click|complete)
                 0.44,  0.48,   0.5,    0.52,   0.5), # Pr(useful free text| complete)
                  nrow = 3, byrow = T)


outcomes2 <-  matrix(c( 0.04,   0.12,   0.16,   0.2,    0.16, # Pr(unvax complete|click)
                 0.05,  0.15,   0.2,    0.25,   0.2, # Pr(unvax web click|complete)
                 0.35,  0.45,   0.5,    0.55,   0.5), # Pr(useful free text| complete)
                  nrow = 3, byrow = T)

outcomes3 <-  matrix(c( 0.04,   0.12,   0.16,   0.2,    0.14, # Pr(unvax complete|click)
                 0.22,  0.29,   0.32,   0.35,   0.32,
                    0.3,    0.4,    0.45,   0.55,   0.5), # Pr(useful free text| complete)
                  nrow = 3, byrow = T)

outcomes_list <- list(outcomes, outcomes2, outcomes3)

# treatments
t_list <- lapply(outcomes_list, FUN=function(x){c(1:ncol(x))})

# alpha
alpha <- 0.05

# number of replications
r <- 1000

### DEFINE FUNCTION ###

trial_subset <- function(N, outcomes, t){
  
  ### Simulate Data
  
  # create dataframe
  d <- data.frame(matrix(nrow = N, ncol = 0))
  
  # treatment assignment
  d$treat <- sample(t, N, replace=T) 
  
  # generate data with completion outcome
  d <- d %>% 
    mutate(complete = rbinom(n(),1,outcomes[1,treat])) 
  
  ## Outcome 1: completion
  
  # make df with columns for treatment and completion
  # this is for diff in diff pvalues
  d_complete <- d 
  
  # aggregate completion counts and click counts by treatment
  # this is for the pairwise prop tests
  d_complete_agg <- d_complete %>% 
    group_by(treat) %>% 
    summarise(
      complete = sum(complete),
      clicks = n()
    )

  ## Outcome 2: webclicks
  
  # make df with columns for treatment and webclick
  # conditional on completes
  # this is for diff in diff pvalues
  d_website <- d %>% 
    filter(complete == 1) %>% 
    mutate(webclick = rbinom(n(),1,outcomes[2,treat]))
  
  # aggregate webclick counts and completes counts by treatment
  # this is for the pairwise prop tests  
  d_website_agg <- d_website %>% 
    group_by(treat) %>% 
    summarise(
      webclick = sum(webclick),
      completes = n()
    )

  ## Outcome 3: Useful free text
  
  # make df with columns for treatment and useful
  # conditional on completes
  # this is for diff in diff pvalues
  d_useful <- d %>% 
    filter(complete == 1) %>% 
    mutate(useful = rbinom(n(),1,outcomes[3,treat])) 
  
  # aggregate webclick counts and completes counts by treatment
  # this is for the pairwise prop tests  
  d_useful_agg <- d_useful %>% 
    group_by(treat) %>% 
    summarise(
      useful = sum(useful), 
      completes = n()
    )
  
  ### Calculate p-values
  
  # create matrix for p-values
  # number of outcomes (3) by number of hypothesis (6)
  p_values <- matrix(nrow = nrow(outcomes),ncol = 6)
  
  
  # calculate for completions outcome
  
  # first get pvalues for usual comparisons (tA vs tB)
  d_subset <- cbind(d_complete_agg[,2], d_complete_agg[,3])
  p <- pairwise.prop.test(d_subset[,1], d_subset[,2], 
                                      p.adj = "none")$p.value
  p_values[1, 1:5] <- c(p["4", "1"], 
                    p["3", "2"], 
                    p["4", "3"], 
                    p["5", "4"], 
                    p["2", "1"])
  
  # now get pvalue for diff in diff
  model = lm(complete~0+treat, d_complete %>% 
               # remove observations with treatments not relevant to diff in diff
               filter(treat != 3) %>% 
               # make numeric categories a character so lm doesn't treat them as continuous
               mutate(treat = paste("W", treat, sep = ""))) 
  
  # run hypothesis test for diff in diff
  hyp = glht(model, "(treatW2 - treatW1) - (treatW5 - treatW4) = 0")
  
  # extract pvalue
  pval <- summary(hyp)$test$pvalues
  
  p_values[1, 6] <- c(pval)
  
  # calculate for webclicks outcome
  d_subset <- cbind(d_website_agg[,2], d_website_agg[,3])
  p <- pairwise.prop.test(d_subset[,1], d_subset[,2], 
                                      p.adj = "none")$p.value
  p_values[2, 1:5] <- c(p["4", "1"], 
                    p["3", "2"], 
                    p["4", "3"], 
                    p["5", "4"], 
                    p["2", "1"])
  
  ## diff in diff
  model = lm(webclick~0+treat, d_website %>% filter(treat != 3) %>% mutate(treat = paste("W", treat, sep = "")))

  hyp = glht(model, "(treatW2 - treatW1) - (treatW5 - treatW4) = 0")
  pval <- summary(hyp)$test$pvalues
  
  p_values[2, 6] <- c(pval)
  
  
  # calculate for useful information outcome
  d_subset <- cbind(d_useful_agg[,2], d_useful_agg[,3])
  p <- pairwise.prop.test(d_subset[,1], d_subset[,2], 
                                      p.adj = "none")$p.value
  
  p_values[3, 1:5] <- c(p["4", "1"], 
                    p["3", "2"], 
                    p["4", "3"], 
                    p["5", "4"], 
                    p["2", "1"])
  
    ## diff in diff
  model = lm(useful~0+treat, d_useful %>% filter(treat != 3) %>% mutate(treat = paste("W", treat, sep = "")))

  hyp = glht(model, "(treatW2 - treatW1) - (treatW5 - treatW4) = 0")
  pval <- summary(hyp)$test$pvalues
  
  p_values[3, 6] <- c(pval)
  ### MHT corrections
  
  return(rbind(p.adjust(p_values[1,], method = "none"),
               p.adjust(p_values[1,], method = "bonferroni"),
               p.adjust(p_values[1,], method = "holm"),
               p.adjust(p_values[1,], method = "hochberg"),
               p.adjust(p_values[2,], method = "none"),
               p.adjust(p_values[2,], method = "bonferroni"),
               p.adjust(p_values[2,], method = "holm"),
               p.adjust(p_values[2,], method = "hochberg"),
               p.adjust(p_values[3,], method = "none"),
               p.adjust(p_values[3,], method = "bonferroni"),
               p.adjust(p_values[3,], method = "holm"),
               p.adjust(p_values[3,], method = "hochberg")))
}

### POWER CALCULATIONS ###

for (i in n){
  
    cat('###',i,' \n')

  
  for(o in 1:length(outcomes_list)){
    treatments <- t_list[[o]]
    outcome_matrix  <- outcomes_list[[o]]
    
    # complete r replications of n trials
    results <- replicate(r,  trial_subset(i, outcomes = outcome_matrix, t = treatments))
    
    # create matrix to store power calculations
    power_calc <- matrix(NA, 
                         nrow = nrow(outcome_matrix)*4, 
                         ncol = 6) 
    
    # for each pairwise treatment comparison
    for(j in 1:6){
      
      # for each mht correction
      for(k in 1:4){
        
        # calculate power for completions outcome
        power_calc[k, j] <- sum(results[k, j, ] < alpha)/r
        
        # calculate power for website clicks outcome
        power_calc[4+k, j] <- sum(results[k+4, j, ] < alpha)/r
        
        # calculate power for useful informaton  outcome
        power_calc[8+k, j] <- sum(results[k+8, j, ] < alpha)/r
        
      }
    }
    
    # Generate Graph
  
    
        power_calc <- data.frame(outcome = c(rep('Unvax completes|Click',4),
                           rep('Unvax web clicks|Complete', 4),
                           rep('Unvax useful freetext|Complete', 4)), 
                         test = rep(c("None","Bonferonni", "Holm", "Hochberg"), 3),
                         power_calc)
   
    colnames(power_calc) <- c("Outcome","Correction", 
                              'T1 v. T4', 'T2 v. T3','T3 v. T4','T4 v. T5', 'T1 v. T2', 'T2 - T1 v. T5 - T4')
 
      
    kable(power_calc,  format = "html", caption = paste("Power Calculations: ",i/1000," Thousand Clicks", "Matrix", o), digits = 3)%>%
  kable_styling()%>%
  collapse_rows(columns = 1)%>%
      scroll_box(height = "100%")%>%
      print()
  }  
}

10000

Power Calculations: 10 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.436 0.413 0.423 0.994 0.986
Bonferonni 1.000 0.217 0.182 0.184 0.954 0.935
Holm 1.000 0.318 0.279 0.286 0.972 0.953
Hochberg 1.000 0.328 0.292 0.301 0.977 0.954
Unvax web clicks|Complete None 0.782 0.119 0.109 0.097 0.187 0.317
Bonferonni 0.507 0.037 0.029 0.027 0.059 0.106
Holm 0.513 0.039 0.035 0.032 0.075 0.123
Hochberg 0.514 0.040 0.035 0.034 0.076 0.125
Unvax useful freetext|Complete None 0.341 0.065 0.070 0.065 0.113 0.154
Bonferonni 0.137 0.020 0.014 0.016 0.034 0.045
Holm 0.139 0.021 0.016 0.018 0.038 0.048
Hochberg 0.139 0.021 0.016 0.018 0.038 0.048
Power Calculations: 10 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.949 0.890 0.901 1.000 1.000
Bonferonni 1.000 0.834 0.719 0.740 1.000 1.000
Holm 1.000 0.944 0.871 0.877 1.000 1.000
Hochberg 1.000 0.946 0.878 0.886 1.000 1.000
Unvax web clicks|Complete None 0.996 0.286 0.308 0.324 0.640 0.779
Bonferonni 0.969 0.106 0.128 0.142 0.281 0.476
Holm 0.969 0.139 0.177 0.192 0.367 0.523
Hochberg 0.969 0.145 0.187 0.205 0.379 0.529
Unvax useful freetext|Complete None 0.889 0.187 0.230 0.218 0.293 0.497
Bonferonni 0.692 0.052 0.081 0.081 0.107 0.243
Holm 0.695 0.063 0.097 0.104 0.133 0.271
Hochberg 0.695 0.063 0.100 0.108 0.137 0.276
Power Calculations: 10 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.954 0.911 1.000 1.000 1.000
Bonferonni 1.000 0.837 0.756 0.991 1.000 1.000
Holm 1.000 0.953 0.910 0.999 1.000 1.000
Hochberg 1.000 0.954 0.911 0.999 1.000 1.000
Unvax web clicks|Complete None 0.562 0.106 0.116 0.107 0.181 0.286
Bonferonni 0.283 0.026 0.026 0.028 0.045 0.096
Holm 0.286 0.028 0.028 0.030 0.065 0.109
Hochberg 0.290 0.028 0.029 0.031 0.066 0.112
Unvax useful freetext|Complete None 0.975 0.189 0.741 0.200 0.296 0.485
Bonferonni 0.914 0.051 0.505 0.066 0.092 0.238
Holm 0.919 0.070 0.540 0.101 0.138 0.282
Hochberg 0.920 0.077 0.547 0.108 0.146 0.285

20000

Power Calculations: 20 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.759 0.706 0.711 1.000 1.000
Bonferonni 1.000 0.528 0.436 0.453 0.999 0.999
Holm 1.000 0.697 0.646 0.635 1.000 0.999
Hochberg 1.000 0.708 0.661 0.651 1.000 0.999
Unvax web clicks|Complete None 0.972 0.197 0.212 0.210 0.420 0.581
Bonferonni 0.893 0.065 0.067 0.076 0.154 0.294
Holm 0.894 0.079 0.086 0.104 0.205 0.335
Hochberg 0.895 0.080 0.090 0.106 0.208 0.337
Unvax useful freetext|Complete None 0.644 0.097 0.105 0.094 0.190 0.243
Bonferonni 0.362 0.021 0.022 0.024 0.055 0.079
Holm 0.367 0.023 0.030 0.031 0.072 0.096
Hochberg 0.368 0.024 0.031 0.032 0.073 0.096
Power Calculations: 20 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.999 0.994 0.992 1.000 1.000
Bonferonni 1.000 0.993 0.969 0.971 1.000 1.000
Holm 1.000 0.999 0.993 0.992 1.000 1.000
Hochberg 1.000 0.999 0.993 0.992 1.000 1.000
Unvax web clicks|Complete None 1.000 0.533 0.599 0.565 0.932 0.972
Bonferonni 1.000 0.291 0.325 0.302 0.766 0.874
Holm 1.000 0.414 0.470 0.443 0.851 0.904
Hochberg 1.000 0.443 0.496 0.471 0.867 0.908
Unvax useful freetext|Complete None 0.998 0.362 0.462 0.442 0.572 0.833
Bonferonni 0.979 0.141 0.191 0.197 0.301 0.606
Holm 0.980 0.193 0.258 0.273 0.391 0.648
Hochberg 0.980 0.206 0.272 0.286 0.403 0.650
Power Calculations: 20 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 1.000 0.995 1.000 1.000 1.000
Bonferonni 1.000 0.995 0.979 1.000 1.000 1.000
Holm 1.000 1.000 0.995 1.000 1.000 1.000
Hochberg 1.000 1.000 0.995 1.000 1.000 1.000
Unvax web clicks|Complete None 0.900 0.157 0.221 0.222 0.366 0.529
Bonferonni 0.709 0.052 0.063 0.068 0.160 0.282
Holm 0.715 0.062 0.082 0.087 0.196 0.308
Hochberg 0.716 0.062 0.089 0.092 0.201 0.310
Unvax useful freetext|Complete None 1.000 0.378 0.954 0.426 0.595 0.840
Bonferonni 0.999 0.153 0.874 0.206 0.314 0.612
Holm 0.999 0.234 0.908 0.307 0.445 0.677
Hochberg 0.999 0.259 0.912 0.327 0.465 0.686

30000

Power Calculations: 30 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.891 0.854 0.858 1.000 1.000
Bonferonni 1.000 0.726 0.644 0.669 1.000 1.000
Holm 1.000 0.883 0.830 0.826 1.000 1.000
Hochberg 1.000 0.887 0.837 0.835 1.000 1.000
Unvax web clicks|Complete None 0.999 0.255 0.299 0.329 0.624 0.791
Bonferonni 0.991 0.101 0.119 0.130 0.342 0.533
Holm 0.991 0.125 0.162 0.178 0.398 0.572
Hochberg 0.991 0.132 0.166 0.181 0.407 0.576
Unvax useful freetext|Complete None 0.817 0.133 0.121 0.133 0.236 0.358
Bonferonni 0.584 0.035 0.031 0.040 0.079 0.146
Holm 0.588 0.039 0.037 0.051 0.102 0.164
Hochberg 0.591 0.040 0.039 0.051 0.106 0.167
Power Calculations: 30 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1 1.000 0.999 1.000 1.000 1.000
Bonferonni 1 1.000 0.998 1.000 1.000 1.000
Holm 1 1.000 0.999 1.000 1.000 1.000
Hochberg 1 1.000 0.999 1.000 1.000 1.000
Unvax web clicks|Complete None 1 0.737 0.769 0.787 0.997 0.999
Bonferonni 1 0.491 0.526 0.557 0.946 0.989
Holm 1 0.702 0.709 0.717 0.986 0.994
Hochberg 1 0.712 0.734 0.735 0.988 0.994
Unvax useful freetext|Complete None 1 0.501 0.621 0.601 0.750 0.934
Bonferonni 1 0.245 0.347 0.336 0.508 0.801
Holm 1 0.370 0.473 0.454 0.631 0.847
Hochberg 1 0.383 0.488 0.468 0.643 0.848
Power Calculations: 30 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 1.000 1.000 1.000 1.000 1.000
Bonferonni 1.000 1.000 1.000 1.000 1.000 1.000
Holm 1.000 1.000 1.000 1.000 1.000 1.000
Hochberg 1.000 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 0.982 0.234 0.266 0.252 0.531 0.719
Bonferonni 0.906 0.086 0.100 0.096 0.247 0.423
Holm 0.909 0.112 0.137 0.123 0.303 0.472
Hochberg 0.909 0.113 0.140 0.128 0.307 0.475
Unvax useful freetext|Complete None 1.000 0.500 0.997 0.596 0.778 0.948
Bonferonni 1.000 0.272 0.973 0.319 0.516 0.817
Holm 1.000 0.432 0.988 0.472 0.687 0.862
Hochberg 1.000 0.450 0.989 0.497 0.702 0.866

40000

Power Calculations: 40 Thousand Clicks Matrix 1
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 0.961 0.949 0.938 1.000 1.000
Bonferonni 1.000 0.854 0.807 0.805 1.000 1.000
Holm 1.000 0.953 0.937 0.924 1.000 1.000
Hochberg 1.000 0.956 0.943 0.931 1.000 1.000
Unvax web clicks|Complete None 1.000 0.372 0.416 0.378 0.737 0.862
Bonferonni 0.998 0.149 0.177 0.167 0.464 0.660
Holm 0.999 0.193 0.261 0.245 0.562 0.698
Hochberg 0.999 0.207 0.279 0.257 0.579 0.701
Unvax useful freetext|Complete None 0.912 0.138 0.162 0.168 0.340 0.455
Bonferonni 0.737 0.037 0.044 0.056 0.142 0.218
Holm 0.741 0.040 0.053 0.069 0.175 0.237
Hochberg 0.743 0.042 0.055 0.071 0.179 0.238
Power Calculations: 40 Thousand Clicks Matrix 2
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1 1.000 1.000 1.000 1.000 1.000
Bonferonni 1 1.000 1.000 1.000 1.000 1.000
Holm 1 1.000 1.000 1.000 1.000 1.000
Hochberg 1 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 1 0.860 0.892 0.885 1.000 1.000
Bonferonni 1 0.651 0.695 0.704 0.992 1.000
Holm 1 0.843 0.871 0.849 1.000 1.000
Hochberg 1 0.852 0.880 0.861 1.000 1.000
Unvax useful freetext|Complete None 1 0.642 0.748 0.744 0.866 0.976
Bonferonni 1 0.356 0.472 0.489 0.671 0.916
Holm 1 0.549 0.642 0.640 0.800 0.938
Hochberg 1 0.585 0.672 0.665 0.810 0.938
Power Calculations: 40 Thousand Clicks Matrix 3
Outcome Correction T1 v. T4 T2 v. T3 T3 v. T4 T4 v. T5 T1 v. T2 T2 - T1 v. T5 - T4
Unvax completes|Click None 1.000 1.000 1.000 1.000 1.000 1.000
Bonferonni 1.000 1.000 1.000 1.000 1.000 1.000
Holm 1.000 1.000 1.000 1.000 1.000 1.000
Hochberg 1.000 1.000 1.000 1.000 1.000 1.000
Unvax web clicks|Complete None 0.997 0.294 0.410 0.307 0.678 0.828
Bonferonni 0.973 0.125 0.162 0.131 0.392 0.607
Holm 0.973 0.163 0.219 0.185 0.471 0.649
Hochberg 0.973 0.170 0.234 0.196 0.482 0.652
Unvax useful freetext|Complete None 1.000 0.655 0.998 0.729 0.883 0.983
Bonferonni 1.000 0.378 0.995 0.491 0.696 0.928
Holm 1.000 0.587 0.998 0.660 0.846 0.952
Hochberg 1.000 0.610 0.998 0.683 0.859 0.953