Graphing power vs effect size

In the frequentist section, we propose several alternative rules for making particular decisions but we don’t provide too much evidence that these proposed rules are reasonable or good. I think that it would be really helpful to show, graphically, the probability of making a certain decision for different rules, effect sizes, and study designs. This would, hopefully, show that, for example, the proposed double barrel rule is a reasonable one.

I have shown how we could do this graphically for the first scenario (the non-inferiority one) below.

library(tidyverse)

Registered S3 method overwritten by 'dplyr':
  method           from
  print.rowwise_df     
Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
-- Attaching packages --------------------------------------- tidyverse 1.3.0 --
v ggplot2 3.2.1     v purrr   0.3.3
v tibble  2.1.3     v dplyr   0.8.3
v tidyr   1.0.0     v stringr 1.4.0
v readr   1.3.1     v forcats 0.4.0
-- Conflicts ------------------------------------------ tidyverse_conflicts() --
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

library(ggplot2)
output_dir <- "C:/Users/dougj/Documents/Bayes stuff/Output"

Testing for non-inferiority

Our working paper proposes two candidate rules for deciding whether to scale up the new intervention

• Rule 1: Null of no difference between the two treatments against one-sided alternative that effect of treatment two is greater than treatment one. (Note that the current version of the text uses a two sided test but I have used a one sided test here since it makes the math slightly more straightforward)
• Rule 2: Null that the effect of the new intervention is at least as large as the effect of the first intervention minus .1 against the alternative that the new intervention is larger than this.

With these two rules, the probability of rejecting the null with the first rule is…

\[ P(reject null, rule 1)=1-\Phi \left({\frac{\Phi^{-1}(1-\alpha_1 )se-\delta}{se}}\right) \]

Where se is the standard error of the estimate, \(\alpha_1\) is our significance level, and \(\delta\) is the difference in the effect sizes. Likewise, the probability of rejecting the null using rule 2 is…

\[ P(reject null, rule 1)=1-\Phi \left({\frac{\Phi^{-1}(1-\alpha_2)se-\delta-\tau}{se}}\right) \] Where se and \(\delta\) are the same as before, \(\alpha_2\) is our new significance level, and \(\tau\) is our non-inferiority threshold. (.1 in the text)

For an RCT with no baseline and equal size treatment and control groups, the standard error of the difference between treatment and control is…

\[ \hat{\sigma_{\Delta}}=\sigma_y\sqrt{\frac{2}{N}} \]

To simplify things further, we may define \(\delta_{sd}=\delta/\sigma_y\) and \(\tau_{sd}=\delta/\sigma_y\). Then we have…

\[ P(reject null, rule 1)=1-\Phi \left({\frac{\Phi^{-1}(1-\alpha_1 )(2/N)^.5-\delta_{sd}}{(2/N)^.5}}\right) \]

Define the functions

p1_reject <- function(deltasd, N, alpha = .05) {
  se <- (2/N)^.5
  prob <- 1-pnorm((qnorm(1-alpha)*se-deltasd)/se)
  return(prob)
}

p2_reject <- function(deltasd, N, tausd, alpha = .05) {
  se <- (2/N)^.5
  prob <- 1-pnorm((qnorm(1-alpha)*se-deltasd-tausd)/se)
  return(prob)
}


# test the probabilty is alpha if delta = 0
p1_reject(0, 100)

[1] 0.05

Graph the functions for various values of delta and N

p <- ggplot(data = data.frame(x = 0), mapping = aes(x = x))
fun.1 <- function(x) p1_reject(x, 100)
fun.2 <- function(x) p2_reject(x, 100, .1)
fun.3 <- function(x) p1_reject(x, 300)
fun.4 <- function(x) p2_reject(x, 300, .1)
p + stat_function(fun = fun.1, aes(color = "Rule 1, N = 100")) + 
  stat_function(fun = fun.2, aes(color = "Rule 2, N=100, tausd = .1")) +
  stat_function(fun = fun.3, aes(color = "Rule 1, N = 300")) + 
  stat_function(fun = fun.4, aes(color = "Rule 2, N=300, tausd = .1")) +
  scale_color_brewer(palette = "Spectral") +
  labs(x = "Effect size difference in SDs" , y = "Prob. scale new intervention", colour = "Decision rule + study design") +
  xlim(-.2,.3)

NA
NA