Power Analysis

Overview

Power analysis is an important aspect of experimental design. It allows us to determine the sample size required to detect an effect of a given size with a given degree of confidence. If the probability is unacceptably low, we would be wise to alter or abandon the experiment.

The following four quantities have an intimate relationship:

  • sample size
  • effect size
  • significance level = P(Type I error) = probability of finding an effect that is not there
  • power = 1 - P(Type II error) = probability of finding an effect that is there Given any three, we can determine the fourth. ( from Quick-R )

pwr Package

pwr package is an implementation of Cohen 1998

Install Package

library(pwr)

Simple Power Analysis Test for a Two-Tailed T-Test

pwr.2p.test two proportions (equal n)

  • h = effect size, Cohen suggests that d values of 0.2, 0.5, and 0.8 represent small, medium, and large effect sizes respectively.
  • n = number of Observations
  • sig.level = Significance level P(Type I error)
  • Power = Power of test 1-P(Type II)
  • alternative = a character string specifying the alternative hypothesis, must be one of “two.sided” (default), “greater” or “less”
pwr.2p.test(h=0.2,n=NULL,sig.level=0.02,power=0.98,alternative = "two.sided") #n=959
## 
##      Difference of proportion power calculation for binomial distribution (arcsine transformation) 
## 
##               h = 0.2
##               n = 959.2624
##       sig.level = 0.02
##           power = 0.98
##     alternative = two.sided
## 
## NOTE: same sample sizes

Effect and Sample Size for Various Power Plots

# Plot sample size curves for detecting correlations of various sizes.
library(pwr)

# range of effect sizes
h <- seq(0.2, 0.8, 0.01)
nh <- length(h)

# power values
p <- seq(0.4, 0.9, 0.1)
np <- length(p)

# obtain sample sizes
samsize <- array(numeric(nh * np), dim = c(nh, np))
for (i in 1:np) {
    for (j in 1:nh) {
        result <- pwr.2p.test(n = NULL, h = h[j], sig.level = 0.05, power = p[i], 
            alternative = "two.sided")
        samsize[j, i] <- ceiling(result$n)
    }
}

# set up graph
xrange <- range(h)
yrange <- round(range(samsize))
colors <- rainbow(length(p))
plot(xrange, yrange, type = "n", xlab = "Effect Size (h)", ylab = "Sample Size (n)")

# add power curves
for (i in 1:np) {
    lines(h, samsize[, i], type = "l", lwd = 2, col = colors[i])
}

# add annotation (grid lines, title, legend)
title("Sample Size Estimation, Two-Tailed, Sig=0.5")  #=P(TypeI)= finding a false effect
legend("topright", title = "Power", as.character(p), fill = colors)
text(x = 0.65, y = 200, "Sig Level= P(TypeI)= Find a False Effect")
text(x = 0.65, y = 150, "Power= 1-P(TypeII)= Find Effect If There")