To P or not to P?

M. Drew LaMar
April 29, 2016

“Increased quantification of scientific research and a proliferation of large, complex datasets in recent years have expanded the scope of applications of statistical methods.” From Wasserstein, Ronald L., and Nicole A. Lazar. “The ASA's statement on p-values: context, process, and purpose.” The American Statistician (2016).

Class announcements

  • Reading assignment for today is posted (you have till midnight)
  • Final Exam: Thursday, May 5, 9 am-12 noon
    • Designed for 2 hours
    • Cumulative
    • 8-10 questions
    • 3 questions from Chaps. 16-18
    • Know the same formulas as you needed for Exam #3
  • Projects
  • Solutions to HW will be up tomorrow

Design and Statistics

“Appropriately chosen techniques, properly conducted analyses and correct interpretation of statistical results also play a key role in ensuring that conclusions are sound and that uncertainty surrounding them is represented properly.”

Importance

“The issues touched on here affect not only research, but research funding, journal practices, career advancement, scientific education, public policy, journalism, and law.”

What is a P-value?

“Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data … would be equal to or more extreme than its observed value.”

Definition: A \( P \)-value is the probability of obtaining a statistical summary of the data or worse given that the null hypothesis is true.

Gotta have principles

Principle #1: P-values can indicate how incompatible the data are with a specified statistical model.

  • The most common context is a model, constructed under a set of assumptions, together with a so-called “null hypothesis.“
  • The null hypothesis postulates the absence of an effect.
  • This incompatibility can be interpreted as casting doubt on or providing evidence against the null hypothesis or the underlying assumptions.

Gotta have principles

Principle #2: P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

  • The P-value is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself.
  • Remember, a P-value is the probability of getting a specific statistical summary of the data or worse assuming the null hypothesis is true.

Gotta have principles

Principle #3: Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.

  • A conclusion does not immediately become “true” on one side of the divide and “false” on the other.
  • Researchers should bring many contextual factors into play to derive scientific inferences:
    • the design of a study,
    • the quality of the measurements,
    • the external evidence for the phenomenon under study,
    • and the validity of assumptions that underlie the data analysis.

Gotta have principles

Principle #4: Proper inference requires full reporting and transparency.

  • P-values and related analyses should not be reported selectively.
  • Researchers should disclose:
    • the number of hypotheses explored during the study,
    • all data collection decisions,
    • all statistical analyses conducted,
    • and all p-values computed.

Gotta have principles

Principle #5: A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

Gotta have principles

Principle #6: By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

  • A p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis.
  • A relatively large p-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data.

Now what?

Alternatives:

  • Estimation (e.g. confidence intervals)
  • Bayesian methods
  • alternative measures of evidence, such as likelihood ratios or Bayes Factors
  • decision-theoretic modeling
  • false discovery rates

The choice is yours

Remember to use statistics for good not evil!

One more thing...