Let's say we want to look at lost-to-followup. We need to consider four things:
(i) What is our primary outcome?
(ii) What is the situation today (aka, the baseline)?
(iii) How much does the outcome vary today across facilities that we might include in the study?
(iv) What is the smallest meaningful effect that we want to be able to detect?
Let's use lost-to-followup as an example primary outcome. Here's a look at two scenarios: 25 sites per arm (so N=50 overall) and 50 sites per arm (N=100). We set desired power to 0.80, alpha to 0.05, and assume a between-cluster coefficient of variation of 0.25 and 100 patients per site (i.e., cluster) for some reference period. We vary the average starting point for lost-to-followup. To make it easy, we actually take 100 minus lost-to-followup to get the percentage of patients NOT lost to follow-up (i.e., success).
Consider a baseline of 40 percent lost-to-followup. Stating this in terms of success (NOT lost-to-followup), the baseline is 100-40 = 60 percent. With 25 facilities per arm (red line), we could detect about a 12 percentage point shift or greater, from 60 percent success to 72 percent success. With 50 facilities per arm, we could detect an even smaller shift from 60 percent success to about 68 percent success.
But what if lost-to-followup is only 20 percent? This would mean success is 80 percent. As you can see from the graph, with 25 facilites per cluser, we could only detect a shift of 16 percentage points or larger, from 80 to 96 percent success. With 50 facilities per cluser, we could detect a shift of 11 percentage points or more, from 80 to 91 percent success.
The starting point matters. It's always harder to detect an effect at the tails of the distribution. If lost-to-followup is 20 percent (i.e., 80 percent success), we would likely need at least 100 facilities to have a fighting chance of detecting an effect.
But maybe there is a lot of variation in lost-to-followup between facilities. The more variation, the harder it is to detect an effect. The following plot assumes lost-to-followup is 20 percent (so 80 percent success or NOT lost-to-followup). With 50 clusters per arm, we would be in trouble if the between-cluster variation is greater than 0.25.
If we can answer points (i) to (iv) above, we can come up with a good estimate of sample size. For (ii) and (iii), we just need to get recent stats on lost-to-followup among all facilities with EMRs that we would consider including in our study.