Hypothesis testing is a procedure for determining, using available sample data, whether or not there exists enough information to make or reject a claim about the greater population of that sample.

• We start with a null hypothesis (\(H_0\)): This is our normal claim, our status quo. What we assume is true.
• We consider an alternative hypothesis (\(H_1\)): This is what we think might be true instead.
• We collect data and compute a test statistic
• We use that statistic to compute a p-value, which tells us if \(H_0\) was true, how surprising would our sample data be? Or, another way, how unlikely would the sample data be if we assume that \(H_0\), our default Null Hypothesis, is true.

For our running example, suppose we want to know whether the average fuel efficiency of cars in a dataset differs from 20 miles per gallon. We set \(H_0\) to the stated claim or what we want to test against, and \(H_1\) to what we suspect might be true.

• \(H_0: \mu = 20\) (the true mean mpg is 20)
• \(H_1: \mu \neq 20\) (the true mean mpg is not 20)

This is a two-sided test which means that it tests both above and below \(H_1\) since we care to know if the mean mpg is either side of the \(H_0\).

When the population standard deviation is unknown, we can standardize the sample mean using the sample standard deviation. This measures how far our sample’s mean is from the hypothesized mean, and it’s provided in units of “standard error”

\[ t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} \]

where \(\bar{x}\) is the sample mean, \(\mu_0\) is the hypothesized mean, \(s\) is the sample standard deviation, and \(n\) is the sample size. Under \(H_0\), this statistic follows a \(t\)-distribution with \(n-1\) degrees of freedom.

The p-value is the probability of observing a test statistic at least as extreme as the one we computed, assuming \(H_0\) is true:

\[ p = P\big(|T| \ge |t_{\text{obs}}| \;\big|\; H_0 \text{ true}\big) \]

• A small p-value means our data would be unusual if \(H_0\) were true and is therefore strong evidence against \(H_0\)
• A large p-value means our data is consistent with \(H_0\) and therefore there is no strong evidence against it
• The p-value is not the probability that \(H_0\) is true, it is a statistical representation of how unusual it would be for \(H_1\) to be true if \(H_0\) was really true.

We compare the p-value to a significance level \(\alpha\). Typically we choose 0.05 unless otherwise stated.

• If \(p \le \alpha\): reject \(H_0\) then the result is “statistically significant”
• If \(p > \alpha\): fail to reject \(H_0\) then there is not enough evidence to reject it

We must always remember that rejecting \(H_0\) does not prove \(H_1\) and failing to reject \(H_0\) does not prove \(H_0\). We are not determining truth, we are only ever weighing the evidence from the sample data to make judgements about the population.

R conveniently has a MotorTrend Car Data-set, so we can use that to exercise the point. Continuing with our stated goal, we set \(H_0\) and \(H_1\) and work to determine if the hypothesis that the average fuel efficiency is 20 mpg is true, or if it’s likely that it is not true.

\[ H_0: \mu_{\text{mpg}} = 20 \qquad \text{vs.} \qquad H_1: \mu_{\text{mpg}} \neq 20 \]

##                    mpg
## Mazda RX4         21.0
## Mazda RX4 Wag     21.0
## Datsun 710        22.8
## Hornet 4 Drive    21.4
## Hornet Sportabout 18.7
## Valiant           18.1

We’ll run a one-sample \(t\)-test at our default \(\alpha = 0.05\).

result <- t.test(mtcars$mpg, mu = 20)
result

## 
##  One Sample t-test
## 
## data:  mtcars$mpg
## t = 0.08506, df = 31, p-value = 0.9328
## alternative hypothesis: true mean is not equal to 20
## 95 percent confidence interval:
##  17.91768 22.26357
## sample estimates:
## mean of x 
##  20.09062

This gives us the observed \(t\) statistic, degrees of freedom, and the p-value.

• This graph shows the possible data points you could have received, not the actual data that we reviewed. It shows a theoretical bell-curve that, if \(H_0\) were true and if you continually repeated the experiment, our test statistic would follow.

• The shaded regions at the tails are the areas where, had our data landed us with that test statistic, we would be forced to reject \(H_0\).

• This plot shows us all of our recorded sample data including our means and IQR.

• The p-value of a two-sided \(t\)-test depends on both the observed \(t\) statistic and the degrees of freedom. As a final diagram, plot shows how it changes across both. It is fully interactable, give it a spin!

• Notice the p-value drops sharply as \(|t|\) grows, and that the surface gets a bit steeper at low degrees of freedom (heavier tails). From this we can infer that lower sample sizes create more drastic curves.

• A p-value measures how compatible your data is with the null hypothesis, or not whether \(H_0\) is “true”. We do this by testing against a hypothesis that we believe could be true instead.
• Small p-values (below your chosen \(\alpha\)) lead you to reject \(H_0\), meaning that the sample data would be unusual given \(H_0\) was true.
• In our example, mtcars mpg had a p-value well above .05, meaning that it would be extremely likely for the sample data to show what it did if \(H_0\). We can confidently fail to reject \(H_0\).