Foundations for Inference

Sampling Methods

• Probability sample: each member of the population has a known probability of being selected
  • Random sample: everyone in the population has an equal probability of selection (uniform distribution)
  • Stratified random sample: population is divided into groups (strata) and some groups are “over-sampled” to increase the number of people
• Non-probability sample: probability of selection is unknown
  • Representative sample: sample reflects known characteristics of the population (i.e., demographics)
• Convenience sample: sample that is accessible at relatively low cost, but may not be representative of the population

CAUTION!

• Standard errors are only accurate if the sampling distribution is normally distributed
• Unfortunately, we cannot verify this if we only have one sample, so we assume that the sampling distribution is nearly normal if:
  1. The sample observations are independent
     • Drawing an observation for Sample A in no way affects the drawing of an observation for Sample B (and vice-versa)
  2. The sample size is greater than 30
     • Thank you central limit theorem!
  3. The distribution of the sample observations is roughly symmetric around the mean (not strongly skewed)
     • Always, always, always plot the distribution of sample observations!!

Hypothesis Testing with CIs

• Now that we know something about the population (based on the sample), we can use point estimates and confidence intervals to test hypotheses about the population
• We are skeptics—we assume that our hypotheses are incorrect unless the data strongly suggest otherwise; we do this by:
  1. Specifying two hypothesis:
     • a null hypothesis \((H_0)\)
     • an alternative hypothesis \((H_A)\)
  2. Assuming \(H_0\) is correct, unless the evidence in favor of \(H_A\) is extremely strong—if the evidence in favor of \(H_A\) is not extremely strong, we “fail to reject” the null hypothesis and assume that our hypothesis is incorrect

Hypothesis Testing with CIs

• How would we use point estimates and confidence intervals to test this hypothesis?
  1. Calculate the sample mean (\(\bar{x}\))
  2. Calculate the standard error of the sample mean (\(SE\))
  3. Calculate the confidence interval around the sample mean (\(95\% CI\))
  4. Does the confidence interval around the sample mean contain the null value?
     • If yes, we cannot reject the null hypothesis
     • If no, we can reject the null hypothesis

Hypothesis Testing with CIs

On a scale from zero to ten, where zero means you are not at all concerned and ten means you are extremely concerned, how concerned are you about the delivery and cost of healthcare in Oklahoma?

Hypothesis Testing with CIs

• In Texas, average concern about the delivery and cost of healthcare is 8.25; are Oklahomans more concerned than Texans?
  • \(H_0: \mu = 8.25\)
  • \(H_A: \mu > 8.25\)
• How would we use point estimates and confidence intervals to test this hypothesis?
  1. Calculate the sample mean (\(\bar{x} = 8.55\))
  2. Calculate the standard error of the sample mean (\(SE = \frac{1.93}{\sqrt{2530}} = 0.04\))
  3. Calculate the confidence interval around the sample mean (\(95\% CI = 8.55 \pm (0.04*1.96) = [8.47, 8.63]\))
  4. Does the confidence interval around the sample mean contain the null value?
     • If yes, we cannot reject the null hypothesis
     • If no, we can reject the null hypothesis

Decision Errors in Hypothesis Testing

• Statistical inference involves uncertainty, so hypothesis tests are never 100% accurate
  • Type I error (“false positive”): reject \(H_0\) when \(H_0\) is actually true
    • Detect a difference (or effect) that is not present
    • To reduce type I error, we increase our standard of evidence (i.e., from 95% to 99%)
  • Type II error (“false negative”): fail to reject \(H_0\) when \(H_A\) is actually true
    • Fail to detect a difference (or effect) that is present
    • To reduce type II error, we reduce our standard of evidence (i.e., from 95% to 90%)

Decision Errors in Hypothesis Testing

• Type I and II errors are zero-sum (reducing one increases the other)
• Usually, we err on the side of caution by reducing type I error at the expense of type II error
  • We avoid incorrect rejection of a true null hypothesis
• We do this by specifying an acceptable type I error rate (\(\alpha\)) before we test our hypotheses
  • Significance level: probability of the study rejecting the null hypothesis, given that it were true
• 5% (\(\alpha = 0.05\)) is the most common significance level
  • \(\alpha = 0.05 = 95\% CI\)
  • Levels should be set according to specific circumstances

Hypothesis Testing with P-values

• Hypothesis tests with confidence intervals can be crude
  • A null value that is just barley outside a CI is treated the same as a null value that is way outside a CI—in both instances we reject \(H_0\)
  • A null value that is just barley inside a CI is treated the same as a null value that is near the center of a CI—in both instances we fail to reject \(H_0\)
• P-values solve this problem by quantifying the strength of evidence against \(H_0\)
  • If \(H_0\) is true, how likely are your data (findings)?
    • The probability of a result at least as extreme as your finding, if \(H_0\) is correct
  • The smaller the p-value, the less likely it is that \(H_0\) is true
  • If p-value \(< \alpha\) (the significance level) we can reject \(H_0\)
• Remember, p-values can provide evidence against \(H_0\), but they say nothing about \(H_A\)

Hypothesis Testing with P-values

• In Texas, average concern about the delivery and cost of healthcare is 8.25; are Oklahomans more concerned than Texans?
  • \(H_0: \mu = 8.25\)
  • \(H_A: \mu > 8.25\)
• How would we use p-values to test this hypotheses?
  1. Define a significance level (\(\alpha\))
  2. Calculate the point estimate (\(\bar{x} = 8.55\))
  3. Calculate the standard error of the point estimate (\(SE = \frac{1.93}{\sqrt{2530}} = 0.04\))
  4. Calculate the z-score of the sample mean (\(\bar{x}\)) assuming the null value is the population mean (\(\mu\)) and the standard error is the standard deviation of the sampling distribution (\(\sigma\))
  • \(z = \frac{\bar{x} - \mu_0}{\frac{s}{\sqrt{n}}} = \frac{\bar{x} - \mu_0}{SE}\)
  5. Calculate the p-value
     • Use the standard normal distribution to calculate the probability of observing a value of \(z\) or higher

Hypothesis Testing with P-values

• If \(H_0\) (8.25) is the true population mean, then the probability of observing a sample mean of 8.55 or higher is < 0.05
• If \(\mu = 8.25\), then it is very unlikely that we would observe a sample mean of 8.55 or higher, which is 7.5 standard deviations away from the null value
• The data are consistent with the hypothesis that Oklahomans are more the cost and quality of healthcare than Texans
• The data provide statistically significant evidence that Oklahomans are more concerned cost and quality of healthcare than Texans
• CAUTION!!!
  • STATISTICAL SIGNIFICANCE \(\neq\) PRACTICAL SIGNIFICANCE
  • Is a difference of 0.3 on a 0-10 concern scale practically significant?

The ASA’s Statement on P-values

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.