Errors in Hypothesis Testing; Linear Regression

• Reading assignment for Monday: OpenIntro Stats, Chapter 7: Introduction to Linear Regression

Can parents distinguish their own children by smell alone? To investigate, Porter and Moore (1981) gave new T-shirts to children of nine mothers. Each child wore his or her shirt to bed for three consecutive nights. During the day, from waking until bedtime, the shirts were kept in individually sealed plastic bags. No scented soaps or perfumes were used during the study. Each mother was then given the shirt of her child and that of another, randomly chosen child and asked to identify her own by smell.

Discuss: What is the null hypothesis? alternative hypothesis?

Answer: With \( p \) the probability of choosing correctly,
\[ H_{0}: \ p = 0.5 \] \[ H_{A}: \ p \neq 0.5 \]

Definition: The test statistic is a number calculated from the data that is used to evaluate how compatible the data are with the result expected under the null hypothesis.

Definition: The null distribution is the sampling distribution of outcomes for a test statistic under the assumption that the null hypothesis is true.

Definition: A  \( P \)-value is the probability of obtaining the data (or data showing as great or greater difference from the null hypothesis) if the null hypothesis were true.

• State the hypotheses.
• Compute the test statistic.
• Determine the \( P \)-value.
• Draw the appropriate conclusions.

Can parents distinguish their own children by smell alone? To investigate, Porter and Moore (1981) gave new T-shirts to children of nine mothers. Each child wore his or her shirt to bed for three consecutive nights. During the day, from waking until bedtime, the shirts were kept in individually sealed plastic bags. No scented soaps or perfumes were used during the study. Each mother was then given the shirt of her child and that of another, randomly chosen child and asked to identify her own by smell. Eight of nine mothers identified their children correctly.

Discuss: What test statistic should you use?

Answer: The number of mothers with correct identifications.

So, P = 0.04. Is that good?

https://youtu.be/4ZB0KxqKN1I?t=1095

So, P = 0.04. Is that good?

Definition: The significance level, \( \alpha \), is the probability used as a criterion for rejecting the null hypothesis. If the \( P \)-value is less than or equal to \( \alpha \), then the null hypothesis is rejected. If the \( P \)-value is greater than \( \alpha \), then the null hypothesis is not rejected

Definition: A result is considered statistically significant when \( P \)-value \( < \alpha \).

Definition: A result is considered not statistically significant when \( P \)-value \( \geq \alpha \).

Can parents distinguish their own children by smell alone? To investigate, Porter and Moore (1981) gave new T-shirts to children of nine mothers. Each child wore his or her shirt to bed for three consecutive nights. During the day, from waking until bedtime, the shirts were kept in individually sealed plastic bags. No scented soaps or perfumes were used during the study. Each mother was then given the shirt of her child and that of another, randomly chosen child and asked to identify her own by smell. Eight of nine mothers identified their children correctly.

Discuss: Given \( \alpha = 0.05 \), \( \{H_{0}: \ p = 0.5\} \), and \( P \)-value of 0.04, what is the appropriate conclusion?

Answer: Reject \( H_{0} \). There is evidence that mothers consistently identify own children correctly by smell.

“We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance.”

Christie Aschwanden

http://fivethirtyeight.com/pvalue

“Belief that "statistical significance” can alone discriminate between truth and falsehood borders on magical thinking.“

Cohen

A Dirty Dozen: Twelve P-Value Misconceptions, by Steven Goodman
    [http://dx.doi.org/10.1053/j.seminhematol.2008.04.003]

Measure and report precision and effect size separately (the \( P \)-value is a summary measure that mixes them):

• Present the magnitude of effect through the use of measures such as rates, risk differences, and odds ratios.
• Report precision with standard errors or confidence intervals.

• Statistical significance is NOT the same as biological importance.
• Effect sizes are important. Large sample sizes can lead to statistically significant results, even though the effect size is small!

http://www.smbc-comics.com/comic/2011-11-23

https://xkcd.com/882/

https://xkcd.com/1478/

https://xkcd.com/892/

Power of a statistical test is a function of
     - Significance level \( \alpha \)
     - Variability of data
     - Sample size
     - Effect size

• Desired power is set by researcher (typically 80%)
• Significance level set by researcher
• Data variability and effect size can be estimated by previous studies or pilot studies
• Sample size is then calculated to achieve desired power given previous fixed attributes