• Low p-values imply high statistical significance.
• Low p-values do not mean it's actually important.
• Large samples are likely to yield low p-values, but the magnitude of the effect might be tiny (insignificant)
• Example: A p-value might be 0.000001 for a coefficient that tells us a trillion dollar increase in toothpick expenditures would reduce unemployment by .1%
  • Statistically significant but economically insignificant.

Commentary on social science (e.g. economics, psychology) has been worrying about "p-hacking."

• Is published research wrong? (video)

• Science isn't broken

• Failure is Moving Science Forward

Even people trained in statistics can be fooled by randomness.

Selection effects can lead to weird outcomes.

• Any one estimate should be taken with a grain of salt.

• When studying something you will try several different specifications, sometimes different datasets, etc.

• But you'll ultimately report on just a handful of models.

• You should give your audience some indication of what the rest of the models looked like.
  • Do your results hold up when outliers are excluded?
  • What about under different model specifications?
• It's better to say
  • "I find some evidence of X but the relationship doesn't always hold up," than
  • "I find evidence of X, please don't look behind the curtain."

• See: Ed Leamer
• The results of empirical work dont' always reflect the work that went into finding those results.
  • We need to find ways (including social norms) to keep ourselves honest.