Here's a quick and simple (and maybe a bit simplistic) example of what low site-level treatment compliance [in terms of summary sheet generation, printing, delivery (where needed), and filing (for all but on-demand printing) ] might have on our ability to detect a treatment effect.
In a step not shown to keep things tidy, we import actual data from the RAs collected between Aug and Dec 2013. Data from 2014 are not much different.
Here's the plot from the baseline report showing mean summary sheet “uptime”. See the report for more details.
ggplot(mydata) +
geom_point(aes(y=mean, x=site.name)) +
coord_flip() +
theme_bw() +
theme(
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
panel.border = element_blank(),
axis.line = element_line(color = 'black'),
panel.background = element_blank()) +
xlab("Facilities") +
ylab("Mean Uptime")
Let's sort these sites into “high” and “low” summary sheet uptime (strata) and then randomly assign sites to treatment or control within each strata.
mydata <- mydata[order(mydata$mean),]
mydata$hiperf <- 1
mydata$hiperf[1:9] <- 0
set.seed(2)
mydata$rand <- c(runif(9, min=0, max=1), runif(9, min=0, max=1))
mydata <- mydata[order(mydata$hiperf, mydata$rand),]
mydata$trt <- c(rep(0,4), rep(1,5), rep(0,5), rep(1,4))
One key ingredient for changing provider INH prescribing behavior is the summary sheet with our shiny new reminders. A necessary but not sufficient condition for change, right?
If so, we should see a bigger bump in INH prescriptions in study sites that have 80 percent summary sheet uptime compared to sites with 14 percent summary sheet uptime. The former are doing a much better job delivering our “treatment”.
So for the sake of argument, let's assume that summary sheet uptime is correlated with an increase in INH prescripitions. Let's also assume that INH prescriptions are at 5 percent currently, and we think we can bump this up to between 10 and 40 percent with the reminders when delivered through a 100 percent operational CDSS system.
If all sites were 100 percent functional, then every treatment site would jump from about 5 percent to between 10 and 40 percent (we assume there are some site-level factors that produce variability in the treatment effect).
mydata <- mydata[order(mydata$trt),]
mydata$pre <- runif(18, min=.04, max=.06)
mydata$post <- c(mydata$pre[1:9],
runif(9, min=.10, max=.40))
m1 <- lm(post ~ trt, data=mydata)
summary(m1)
##
## Call:
## lm(formula = post ~ trt, data = mydata)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.11374 -0.04597 -0.00052 0.00799 0.14625
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.0479 0.0256 1.87 0.08 .
## trt 0.2004 0.0362 5.53 4.5e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.0768 on 16 degrees of freedom
## Multiple R-squared: 0.657, Adjusted R-squared: 0.635
## F-statistic: 30.6 on 1 and 16 DF, p-value: 4.53e-05
ate.1 <- round(as.numeric(coefficients(m1)[2]*100), 1)
Check out the coefficient on trt; a treatment effect of 20 percent. The intercept is the rate among the control group, so add the treatment effect to the intercept to get the rate among the treatment group.
Now let's use our data on actual performance to scale down the max possible increase in INH prescription rates. For example, if a site has a mean summary sheet uptime of 60 percent, we'd assume that the intervention would be 40 percent less effective. So rather than increasing INH prescription rates from 5 to 25 percent (20 pp), let's say, the site would only increase by 12 pp from 5 to 17 percent.
mydata$post2 <- c(mydata$pre[1:9],
mydata$pre[10:18] + (mydata$mean[10:18]/100)*runif(9,
min=.10,
max=.40))
Here's what happens to our treatment effect:
m2 <- lm(post2 ~ trt, data=mydata)
summary(m2)
##
## Call:
## lm(formula = post2 ~ trt, data = mydata)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.06559 -0.04558 -0.00487 0.00168 0.21693
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.0479 0.0240 1.99 0.0634 .
## trt 0.1211 0.0339 3.57 0.0026 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.072 on 16 degrees of freedom
## Multiple R-squared: 0.443, Adjusted R-squared: 0.408
## F-statistic: 12.7 on 1 and 16 DF, p-value: 0.00257
ate.2 <- round(as.numeric(coefficients(m2)[2]*100), 1)
The effect size reduces by 39.5 percent to 12.1 percent.
In reality, however, the treatment effect probably disappears altogether if summary sheet uptime drops below some threshold. Let's say that threshold is 50 percent uptime. For sites below 50 percent uptime, we set the treatment effect to zero.
mydata$post3 <- mydata$post2
mydata$post3[mydata$mean<=50] <- mydata$pre[mydata$mean<=50]
m3 <- lm(post3 ~ trt, data=mydata)
summary(m3)
##
## Call:
## lm(formula = post3 ~ trt, data = mydata)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.08868 -0.05526 -0.00487 0.00168 0.25699
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.0479 0.0301 1.59 0.131
## trt 0.0810 0.0425 1.90 0.075 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.0903 on 16 degrees of freedom
## Multiple R-squared: 0.185, Adjusted R-squared: 0.134
## F-statistic: 3.63 on 1 and 16 DF, p-value: 0.075
ate.3 <- round(as.numeric(coefficients(m3)[2]*100), 1)
Here the effect size drops to 8.1 and becomes non-significant. At this point we become sad.
This simple example illustrates how CDSS performance could greatly attenuate the treatment effect. We can disagree about the assumptions, but there should be general agreement that if our treatment delivery vehicle is not working well, we will have less of a treatment effect. I can't say what “good enough” looks like in terms of CDSS performance, but we should discuss and then do everything we can to help AMPATH meet this bar.
.