• if you have internet, please install http://jasp-stats.org
• it's also on the sticks
• please also install https://www.r-project.org, https://www.rstudio.com for more comfort, and blavaan (open R, enter install.packages("blavaan") and run) to follow
• let's do it!
• Literature: Password "JRPC2016bayes": http://bit.ly/JRPC2016bayes_lit

Ziliak and McCloskey (2008)

"The textbooks are wrong. The teaching is wrong. The seminar you just attended is wrong. The most prestigious journal in your scientific field is wrong."

creadata <- read.csv("crea_bayes.csv") # Load data
summary(lm(fluency ~ N + E + O, data = creadata)) # Multiple regression

• What does p tell us?

• The probability to get data this or more extreme in case that in the underlying population there is no association between openness and ideational fluency is less than 0.1%
  • Null hypothesis: association = 0;
  • Reject this null: apply 5% convention -> if p < .05.
  • If H0 is true, data very unlikely -> reject H0, accept H1
  • …do this same thing in all analyses…

• Does this sound strange to you?

• Frequentists, probability is the long-run relative frequency of events
  • prob. coin coming up heads: Proportion of heads in infinite amount of tosses
  • Asking about probability of next coin toss: Nonsensical
  • Next toss: Either heads, or not
  • Single events can't be assigned probability
• Repeatability becomes a crucial ingredient
• Fisher: Biologist, did lots of repeatable experiments

• Frequentists cannot talk about…

• …the probability of a 3rd world war
• …climate change
• …you failing your next exam
• …Jimmy C. having a beer tonight
• …any non repeatable event

• Sneak peek: Single events are about uncertainty

• Probability conceptualized as long-run average frequency: No answer to essential question in science "what is the probability that my hypothesis / theory is true?"

p…

• …disproves the null hypothesis of no difference.
• …proves the alternative hypothesis.
• …indicates the probability of the null hypothesis being true.
• …indicates the probability of the alternative hypothesis being true.
• …indicates the probability of the experimental hypothesis being true.
• …indicates the probability of a wrong decision in case we reject the null hypothesis.
• …indicates that in 99% of cases we would obtain a significant result when repeating the experiment.
• …indicates the probability of the data or more extreme data given that the null hypothesis is true.

p…

• …disproves the null hypothesis of no difference.
• …proves the alternative hypothesis.
• …indicates the probability of the null hypothesis being true.
• …indicates the probability of the alternative hypothesis being true.
• …indicates the probability of the experimental hypothesis being true.
• …indicates the probability of a wrong decision in case we reject the null hypothesis.
• …indicates that in 99% of cases we would obtain a significant result when repeating the experiment.
• …indicates the probability of the data or more extreme data given that the null hypothesis is true.

(Wagenmakers, 2007)

• Just like dropping a smelly shoe.

• A frequentist needs this sampling distribution
• Often, properties of the sampling distribution can be derived analytically from the sample data
• Variance of the sampling distribution of the sample mean is \(\sigma^2 = \frac{s^2}{N}\)
• One reason why frequentism got so much traction: It is computationally trivial! Nice, or not?

• result of statistical analysis depends on the intention of the researcher
• because those intentions define the space of all possible (unobserved) data (Wagenmakers, 2007)

• suppose I ask you 26 questions on Bayesian stats, and you get 8 right
• the last question you answered, you answered correctly
• were you better than chance? \(H_0: \theta = 0.5\)
• there are two possible sampling plans
• ask 26 questions, and see how many you answered correctly (binomial)
• ask questions so long until you answer 8 correctly (negative binomial)
• the data are the data are the data (example from Kruschke, 2010, Ch. 11)

N <- 26
k <- 8

binomial <- pbinom(k, N, 0.5) * 2
negbinomial <- pnbinom(k, N, 0.5) * 2

binomial # fail to reject

## [1] 0.0755187

negbinomial # science paper!!

## [1] 0.002935056

• optional stopping (Rouder, 2014; Sanborn & Hills, 2014)
• look at data, test, gather more data, test, etc.

source("http://rynesherman.com/phack.r") # read in the p-hack function

res <- phack(initialN=30, hackrate=5, grp1M=0, grp2M=0, grp1SD=1, grp2SD=1,
   maxN=200, alpha=.05, alternative="two.sided", graph=TRUE, sims=2000)

• inflated \(\alpha\) (p-hacking!)
• assume you tested 20 participants, and get \(p = 0.057\)
• What do you do with this? Does it tell you nothing?

• don't quantify statistical evidence; Wagenmakers (2007)
• \(p = 0.04, n = 10\) is more evidence than \(p = 0.04, n = 1000\)
• in fact, the latter is support for \(H_0\)!
• are violently biased against \(H_0\)

• violently biased against \(H_0\)
• underlying logic of p-values: either the null hypothesis is false, or a rare event has occured
• following example taken from Rouder et al. (in press)

• the following is logically correct:
• (Premise): If Hypothesis \(H\) is true, then event \(X\) will not occur.
• (Premise): Event \(X\) occured.
• (Conclusion): Hypothesis \(H\) is not true.

• this does not translate to probabilistic settings
• (Premise): If Hypothesis \(H\) is true, then event \(X\) is unlikely.
• (Premise): Event \(X\) occured.
• (Conclusion): Hypothesis \(H\) is probably not true.

• take this example as demonstration:
• (Premise): If Jimmy is a German, then it will be unlikely that he is a JRP stakeholder.
• (Premise): Jimmy is a JRP staleholder.
• (Conclusion): Jimmy is probably not German.
• by only looking at \(p(D|H_0)\), p-values are violently biased against \(H_0\)

• p-value gives you \(p(\text{D or more extreme}|H_0)\)
• what we want is \(p(H|D)\), the probability that our hypothesis is true!
• there is a subtle, but important difference
• \(p(\text{you are crying} | \text{someone throws a potato on your foot})\) is very high :(
• \(p(\text{someone threw a potato on your foot} | \text{you are crying})\) is very low!

• quick question: how old do you think is modern statistics?

• Statistical Methods for Research Workers (1925), Design of Experiments (1935)

• His first approach to statistical inference
  • Null hypothesis \(H_0\), \(\alpha\)
  • no alternative hypothesis.
  • p > .05: ignore results.

• p-value indicates strength of evidence
  • \(p = .001\) is better than \(p = .049\)

Revision: "No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas."

• R.A. Fisher (1956), as cited in (Gigerenzer & Marewski, 2015, p. 1)

• Introduced the alternative hypothesis, \(H_1\)
  • Concept of a \(\beta\) error; statistical power
• \(p < \alpha\), or \(p > \alpha\)
• Binary cut-off; not statistical evidence
• They did behavioral statistics

no test based on probability theory can provide evidence of truth or falsehood. BUT it might govern our behavior, in following which we ensure that in the long run of experience, we shall not be too often wrong.

• Neyman and Pearson (1933), as cited in (Johansson, 2011, p. 118)

• 1940s/1950s
  • Psychologists (Guilford) combine both approaches in statistics textbooks
  • Ignoring differences in names (Fisher vs. Neyman & Pearson) and fundamentals (alternative hypothesis vs. none, fixed alpha vs. not)

• Current statistical practice: An evilish hybrid between those two incompatible paradigms

The Result? Mindless Statistics

• 1a: setup a statistical hypothesis of no difference or zero correlation
• 1b: don't specify the predictions of your research hypothesis
• 2: Use 0.05 as a convention for rejecting H0
• 3: always use this procedure (Gigerenzer, 2004)

statisticians "have already overrun every branch of science with a rapidity of conquest rivalled only by Attila, Mohammed, and the Colorado beetle."

• Did Piaget ever compute a p-value?
• Did Skinner? what's with Köhler? Pavlov?

• Effect Size
  • A measure of the strength of an effect
  • e.g., a correlation of r = .35, a difference between experimental conditions of Cohen's d = 0.30
• Confidence interval
  • The area within which in 95% of replications (same experiment/measures, same sample size drawn from the same population) the real parameter (effect) will be
• Example:
  • There was a significant difference between the control group and the intervention group, p = .002, d = 0.34 [0.22, 0.44]

• Cumming, 2014
• "We need to make substantial changes to how we conduct research"

• Replicate (x)

• Adapt meta-analytical thinking (x)

• Avoid NHST (x)

• Don't trust any p-value (x)

• BUT

• ES are the main research outcome (…?)
• The CI tells us the precision of a study (…?)
  • …a much better approach than declaring the result "statistically significant" (…?)
• "Enjoy the benefits!" (…?)

• So, what does a SINGLE confidence interval tell us?

• Nothing. :D

Part I: Lee, 2014

(Lee, here)

• "There are three things in life you can be sure of. Death, taxes, and that, when the APS makes an editorial pronouncement dealing with statistical theory, they will botch it."

"…it relies on fundamentally flawed statistical theory. Since confidence intervals share this theory, the new statistics are also fundamentally flawed."

Part II: Morey et al., 2014

• "For psychological science to be a healthy science, both estimation and hypothesis testing are needed."
• Estimation -> pretheoretical work, theory revision.
• Hypothesis testing -> testing the quantitative predictions of theories.
• None is more informative than the other. They answer different questions.
• Estimation alone -> massive catalogue devoid of theoretical content.
• Hypothesis testing alone -> researchers might miss rich, meaningful structure in data.
• It is crucial for estimation and hypothesis testing to be advocated side by side.

Part III: Hoekstra et al., 2014

Part III: Hoekstra et al., 2014

• Effect size estimation is no solution.

• Apparently, frequentist statistics has something flawed inherent.

• We still need a solution to the pervasive problem of p-values. (Wagenmakers, 2007)

• "What should be done about this? Easy. Use Bayesian methods." (Lee, 2014)

• what is the probability of my hypothesis, p(\(H\)), given the data, \(\textbf{y}\), I have collected? \[ p(H|\textbf{y}) = \frac{p(\textbf{y}|H)p(h)}{p(\textbf{y})} \]

• p-value gives you \(p(\text{D or more extreme}|H_0)\)
• what we want is \(p(H|D)\), the probability that our hypothesis is true!
• there is a subtle, but important difference
• \(p(\text{you are dead} | \text{shark has bitten off your head})\) is very high :(
• \(p(\text{shark has bitten off your head} | \text{you are dead})\) is very low!
• We should compare different hypotheses.

• both Clark's babies died, where \(p(\text{baby dies}) = \frac{1}{8543}\)
• probability both babies died: 1 in 73 million -> \(p < 0.00001\) -> alternative: Clark probably killed her babies?!
• indeed, in November 1999, a jury found Sally Clark guilty of double murder.

"The jury needs to weigh up two competing explanations for the babies' deaths: SIDS or murder. The fact that two deaths by SIDS is quite unlikely is, taken alone, of little value. Two deaths by murder may well be even more unlikely. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation."

• President of the Royal Statistical Society (2002)

• in fact, \(p(\text{baby dies} | \text{sudden infant death})\) is higher than \(p(\text{baby dies} | \text{murder})\)
• the ratio of these two - the likelihood ratio - is the proper measure of statistical evidence
• statistical evidence is always relative; there is no free lunch

• in conditional probability, say \(p(D|\theta)\), the hypothesis is fixed and the data vary
• in likelihoods, \(L(\theta|D)\), the data is fixed and the hypothesis varies
• \(L(\theta|D) \propto K \times p(D|\theta)\), need not sum to one

• law of likelihood: data supports \(H_0\) over \(H_1\) iff \(p(D|H_0) > p(D|H_1)\)
• likelihood principle: all information relevant for statistical evidence is contained in the likelihood function
• Difference to p-value: For both the binomial and the negative binomial case, the likelihood is \(\theta^k (1 - \theta)^{n-k}\)
  • Everything is in the data, no need to fix the sampling plan.
  • Intentions do not matter
  • There is no issue with optional stopping! see Royall (1997)

• The factor by which we update our prior beliefs to obtain posterior beliefs
• \(\frac{p(H_0)}{p(H_1)}\)
• \(\frac{p(H_0|D)}{p(H_1|D)}\)
• \(\frac{p(D|H_0)}{p(D|H_1)}\)
• thus: \(\frac{p(H_0|D)}{p(H_1|D)} = \frac{p(D|H_0)}{p(D|H_1)} \times \frac{p(H_0)}{p(H_1)}\)

• it's just conditional probability: \[ \begin{align} P(\text{spotted}, \text{red}) &= P(\text{spotted}|\text{red})P(\text{red}) \\[1ex] P(\text{spotted}, \text{red}) &= P(\text{red}|\text{spotted})P(\text{spotted}) \\[1ex] \end{align} \]

• which yields Bayes' Rule:

\[ \begin{align*} P(\text{spotted}|\text{red})P(\text{red}) &= P(\text{red}|\text{spotted})P(\text{spotted}) \\[1ex] P(\text{spotted}|\text{red}) &= \frac{P(\text{spotted}) \times P(\text{red}|\text{spotted})}{P(\text{red})} \end{align*} \]

• what is the posterior probability of my hypothesis, p(\(H\)), given the data, \(\textbf{y}\), I have collected?
• It is the data weighted with the prior knowledge I had before collecting it. \[ p(H|\textbf{y}) = \frac{p(H)p(\textbf{y}|H)}{p(\textbf{y})} \]

• "What's the probability that it will rain tomorrow"?
  • Frequentist: "That is a non-sensical question."
  • Bayesian: "I am in England and it's August. I guess about 30%."
• if probabilities are subjective, don't we have a problem?
• how do we quantify them, then?

John M. Keynes (1921)

Frank P. Ramsey (1926)

Frank P. Ramsey (1926)

• Psychology (perception of degree of belief, DoB) was the basis for modern probability calculus (and decision making theory)!

Does the train come? Make your bet! Set your prior

• \(\frac{p(H_0|D)}{p(H_1|D)} = \frac{p(D|H_0)}{p(D|H_1)} \times \frac{p(H_0)}{p(H_1)}\)
• \(\text{posterior odds} = \text{Bayes factor} \times {\text{prior odds}}\)
• important distinction: prior on parameters, prior on models (BEM)
  • Parameter estimation vs. hypothesis testing

• demo!

• computationally hard
• Savage-Dickey! (Wagenmakers, Lodewyckx, Kuriyal, & Grasman, 2010)

Interpretation.

- from Vandekerckhove, Matzke, & Wagenmakers (2014)

• real confidence intervals
• you can be 95% confident that the parameter lies within this range!

• Tackle the frequentist problems
• Direct information about hypotheses
• Estimating parameters (Credibility Interval) and models (Bayes Factor)

• "Why can't we all just be Bayesian?" (Lee, here)
• We can now.
• Computationally intensive
  • Nowadays, no problem with MCMC
• Subjectivity of priors
  • A "good" thing or a "bad" thing?
• What "type of model" is this?
  • Exactly the same as the models you know!
• Software
  • R: BUGS/JAGS/BayesFactor/blavaan
  • JASP
  • Mplus
  • Even AMOS (SPSS-kiddie)

“Hidden meaning transforms unparalleled abstract beauty.”!

"In 20 years, our children will ask us why we have been frequentist, and the answer will be difficult."

• Executive functions, intelligence, personality, creativity + insight

• N = 230 university students

Structural Equation Modeling

• Analysis of (variance) covariance (mean) structures
  • Compare whether "theoretically allowed" parameter estimates suffice to approximate empirical data

Structural Equation Modeling

• Analysis of (variance) covariance (mean) structures
  • Compare whether "theoretically allowed" parameter estimates suffice to approximate empirical data

Structural Equation Modeling

• Analysis of (variance) covariance (mean) structures
  • Compare whether "allowed" parameter estimates suffice to approximate empirical data
• Emphasis on latent variables
  • Non-observed variables representing psychological constructs
  • Get rid of measurement error
• Highly flexible approach

# Define model
model1 <- '# Intercepts: Estimate mean (and var)
fluency~1'
# Estimate model parameters: ML
fit_model1 <- cfa(model1,
                  data=crea)
summary(fit_model1) # Show model results
coef(fit_model1)[1] # Show estimated Mean
mean(fluency) # Compare to sample Mean
sqrt(coef(fit_model1)[2]) # Show estimated SD
sd(fluency) # Compare to sample SD

# Draw SEM model figure
semPaths(fit_model1,
         style = "lisrel",
         "estimates",
         rotation = 4,
)

Latent growth model

approximate MI -> van de Schoot et al., 2013B

• Mean comparisons

• Multigroup CFA

• Break out of schematic statistical testing
  • There is no sense in sticking to "t-test", "ANOVA", "MANOVA", "ANCOCA", "RMANOVA"
  • Cross the borders, build your own model

bfit5 <- bsem(parTable(fit_model5), data=risk)
summary(bfit5)
fitMeasures(bfit5)

• Item Response Theory
  • Same model as confirmatory factor analysis (CFA) in SEM
  • With categorical indicators
  • Valid estimation possible in Mplus, lavaan (R)
  • Use weighted least squares (WLS) estimator
  • or Bayes (blavaan)

• Hu & Bentler, 1999: Rules of thumb
  • But see Heene, 2013
• RMSEA
  • Chi-square/number estimated parameters
  • "Should be" <.06
• CFI
  • Comparison to strongly constrained null model
  • "Should be" >.90
• SRMR
  • Average residual variance/covariance
  • "Should be" <.11

• ppp: Posterior predictive p-value
  • Simlaute data from specified model: Where do yours fall?
• DIC: Deviance information criterion
  • Rational like classical information criteria AIC, BIC
• BF*: Bayes factor approximation
  • Log Laplace-appproximated ratio of marginal likelihoods

• Prior choice (informative/non-informative, rationales)
• Graphical model (SEM-type/cognitive science-type)
• Convergence: Trace plots (for important parameters)
  • see also van de Schoot et al., 2013

(From most user-friendly to most elaborate)

• JASP http://jasp-stats.org (easy cheesy) <->

• BayesFactor http://bayesfactorpcl.r-forge.r-project.org (easy cheesy/moderate)

• blavaan (R) (moderate) <->

• JAGS http://mcmc-jags.sourceforge.net (tough)

• Mplus http://statmodel.com/ (moderate)
  • Kaplan & Depaoli, 2012; van de Schoot et al., 2013

• Stan http://mc-stan.org (tough)

• AMOS (easy cheesy), WinBUGS (tough)

Bayesian Statistics: What is it and why do we need it?

Bayesian Statistics: Why and how?

How to become a Bayesian in eight easy steps: An annotated reading list

Structural equation modeling: What is it, what does it have in common with hippie music, and why does it eat cake to get rid of measurement error?

• Bayes:
• McElreath (2015) Statistical Rethinking

• Lee & Wagenmkaers (2013) Bayesian Cognitive Modeling

• SEM:
• Beaujean (2014) Latent Variable Modeling Using R

• Kline (2015) Principles and Practice of Structural Equation Modeling

• Bayesian SEM:
• Kaplan (2013) Bayesian Structural Equation Modeling

• Kaplan & Depaoli (2013) Bayesian Structural Equation Modeling

• Beaujean, A. A., (2014). Latent Variable Modeling Using R: A Step-by-Step guide. Routledge.
• Byrne, B. (2011). Structural Equation Modeling with Mplus: Concepts, Applications, and Programming. Routledge & Chapman.
• Geiser, C. (2012). Data Analysis with Mplus. New York: Guilford Press.
• Hoyle, Rick H. Handbook of Structural Equation Modeling. New York: Guilford Press.
• Hu, Li-tze, and Peter M. Bentler. "Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives." Structural Equation Modeling: A Multidisciplinary Journal, 1-55. doi:10.1080/10705519909540118.
• Kline, R. B. (2011). Principles and Practice of Structural Equation Modeling (3rd ed.). New York: Guilford Press.
• Little, T. (2013). Longitudinal Structural Equation Modeling. New York
• McArdle, J. J., & Nesselroade, J. R. (2014). Longitudinal Data Analysis Using Structural Equation Models. American Psychological Association.

Barker, D. H., Rancourt, D., & Jelalian, E. (2014). Flexible Models of Change: Using Structural Equations to Match Statistical and Theoretical Models of Multiple Change Processes. Journal of Pediatric Psychology, 39(2), 233-245. http://doi.org/10.1093/jpepsy/jst082

Van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & van Aken, M. A. G. (2013). A Gentle Introduction to Bayesian Analysis: Applications to Developmental Research. Child Development, 85, 841-860. http://doi.org/10.1111/cdev.12169

Van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muth?n, B. (2013). Facing off with Scylla and Charybdis: a comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 4. http://doi.org/10.3389/fpsyg.2013.00770

JEPS Bulletin blog posts: http://blog.efpsa.org/2014/11/17/bayesian-statistics-what-is-it-and-why-do-we-need-it-2/ http://blog.efpsa.org/2015/08/03/bayesian-statistics-why-and-how/

Application in R and Mplus: van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & van Aken, M. A. (2013). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85, 841-860. doi:10.1111/cdev.12169

Application in R: Kruschke, J. K. (2014). Doing bayesian data analysis: A tutorial with R, JAGS, and Stan (2nd ed.). Academic Press.

Bayesian SEM in Mplus: Kaplan & Depaoli (2012). Bayesian Structural Equation Modeling. In: Hoyle, R. H.(Ed): Handbook of Structural Equation Modeling. pp.650-673. Guilford Press.