MT5762 Lecture 12

Power is intimately related to type 2 errors (false negatives)

It is our ability to detect differences when they actually exist

We look to address:

• What is power? (and why do we care)
• Intuitively how would you calculate power?
• How do you calculate power in practice?
• How can you improve power?

We've done much testing now.

• Type 1 error is the probability of incorrectly rejecting \( H_0 \) i.e. a false positive
• Type 2 error is the probability of incorrectly failing to reject \( H_0 \) i.e. a false negative

We control Type 1 in our tests by setting our threshold \( p \)-value (type I error = \( \alpha \)).

There is nothing explicitly controlling type II error (\( \beta \)) in our tests.

• Perhaps obviously type II is the harder one.
• Calculating it requires we know (or speculate) about what \( H_A \) is - something we're usually very vague about.

Power is \( 1-\beta \) - the probability we correctly reject \( H_0 \)

• To calcuate it, we have to specify an explicit \( H_A \).
• Similar to our other tests, our probability calculations are conditional on some hypothesised state being true.

Power calculations are often done to determine how much data to collect.

• How big an effect do you need/hope to detect?
• How much variability is in the system?
• What level of power is needed?

Armed with these, we can advise on \( n \) to meet specifications

Power calculations are often done to determine how much data to collect.

• The variability of the system is needed:
  • Perhaps this is unknown, so have to guess
  • If we estimate from some small study, precision on the variance is very low
• The recommended \( n \) is often large, so desired power and effect size might need revising

(Example modified from Larsen & Marx, 2001)

• There is a new fuel additive that is expected to increase the fuel efficiency for vehicles. The underlying fuel efficiency with the standard fuel is assumed to be 25 mpg.
• It is thought an improvement of at least 3% in fuel efficiency would be substantial enough for the additive to be taken to market.
• Hence we are particularly interested in establishing a change from 25 to 25.75 mpg.

• It is proposed that a sample size of 30 would be used to establish this improvement.
• The standard deviation of efficiency in cars is claimed to be 2.4 (\( \sigma \)).
• If the improvement was in fact 25.75 mpg, what is the power and Type II associated with this (one-sided) test scenario?

NB we've been given \( \sigma \) as known. If this were estimated, we'd have to use \( t \)-distributions.

• So we'd fail to find evidence of an improvement, if it were only 3%, almost half the time.
• These are not great odds! The obvious place to make an improvements to this poor power are:
  • Increasing precision through larger samples.
  • Increasing precision through controlling variability (e.g. testing on a standardised track or the like).
• Accept a higher Type I error.
• Have a fuel additive that is expected to have improvements much greater than 3%.

What happens if \( n \) is doubled?

• The decision boundary is now at:

decisionBound <- 25+1.644854*2.4/sqrt(60)

decisionBound

[1] 25.50964

With this in mind, consider multiple comparison adjustments

• We make our type 1 error boundaries more stringent (broader) i.e. lower threshold \( p \)-values
• We're trading errors at some level - type 2 errors increase, power goes down

Implications for cynical exploitations:

• If you don't want to find significant results, you want low power
• Take small samples, define big changes as important, do adjust for multiple comparisons, sloppy measurements to give large variances etc

We need this a lot:

\[ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \mathbf{e} \]

leading to things like this

\[ \hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\hat{\beta}} = \begin{bmatrix} 1 & x_{1,1} & x_{2,1}\\ 1 & x_{1,2} & x_{2,2}\\ \vdots & \vdots & \vdots\\ 1 & x_{1,n} & x_{2,n}\\ \end{bmatrix} \begin{bmatrix} \hat{\beta_0}\\ \hat{\beta_1}\\ \hat{\beta_2}\\ \end{bmatrix} \]