DropboxLink

A model is a conceptualization of a system, often a hypothesis of how we think a system works.

"All models are wrong, but some are useful." - George Box, 1976 & 1978

…is a mathematical representation of relationships between a population's variables.

• Facilitate understanding of a stochastic process
• …and/or predict the outcome of a stochastic process

• To understand a process, we construct simple models that are easy to interpret, but do not always make very accurate predictions.
• To predict an outcome, we construct complex models that can be difficult to interpret, but make more accurate predictions.

\[y = f(x) + \varepsilon\]

• \(y\) is a specific value of the response or outcome variable, \(Y\)
• \(x\) is a specific value of the explanitory or predictor variable, \(X\)
• \(\varepsilon\) describes residual error unexplained by the model
• \(f(x)\), describes how \(Y\) and \(X\) are related

(We used to call \(Y\) and \(X\) dependent and independent variables, but statisticians no longer recommend this nomenclature.)

\(f(x) = \beta_{0} + \beta_{1} x\), and \(\varepsilon\) follows a normal distribution:

\[y = \beta_{0} + \beta_{1} x + \varepsilon\]

…using various methods.

• Ordinary Least Squares Regression
• Maximum Likelihood Estimation
• Bayesian Estimation
• …etc.

…can be modeled by including higher order terms.

\[y = \beta_{0} + \beta_{1} x + \beta_{2} x^2 + \dots + \beta_{z} x^z + \varepsilon\]

GLMs describe categorical and/or non-linear relationships by assigning a link function, \(f(y)\), and letting the residuals follow non-normal distributions.

\[f(y) = \beta_{0} + \beta_{1} x + \varepsilon\]

This common GLM uses the logit link function and binomial \(\varepsilon\) to model the probability of a binary outcome.

\[\ln \left(\frac{p}{1-p}\right) = \beta_{0} + \beta_{1} x + \varepsilon\]

Finally, the linear model can allow any number of predictor variables.

\[y = \beta_{0} + \beta_{1} x_{1} + \beta_{2} x_{2} + \dots + \beta_{k} x_{k} + \varepsilon\]

• Does \(Y\) change with \(X\)? …for a specific \(f(x)\).
  • "Significance"
• How much variability in \(Y\) is explained by a specific \(f(x)\)? …i.e. "fit."
  • \(R^2\)
• What \(f(x)\) best describes \(Y\)?
  • Machine Learning

Ronald Fisher went to war with Jerzy Neyman and Egon Pearson (son of Karl Pearson) during the 1930's over the most philosophically sound method for statistical hypothesis testing.

Our modern null hypothesis significance test, NHST, is a hybrid of the two camps' recommendations.

Hypotheses can never proven true, they can only ever be proven false.

• The hypothesis "all raves are black" can never be proven true because you can't possibly be assured that all raves have been observed.
• However, finding one white raven disproves "all ravens are black."

This philosophy of science says that a hypothesis is scientific only if it is falsifiable.

Following this philosophy, NHST attempts to falsify a null hypothesis.

1. Make null and alternate hypotheses, \(\mathrm{H}_0\) and \(\mathrm{H}_\mathrm{a}\).
2. Calculate a test statistic measuring our sample's deviation from the null condition.
3. Assume \(\mathrm{H}_0\) is true.
4. Determine the probability of observing a deviation larger than our sample's deviation by chance alone, the p value.
5. If this probability…
   • …is small, conclude \(\mathrm{H}_0\) is false, therefore \(\mathrm{H}_\mathrm{a}\) must be true.
   • …is not small, conclude \(\mathrm{H}_0\) could be true.

• \(\mathrm{H}_{0}\) describes a null condition, i.e. that our sample statistic is similar to some hypothesized population parameter.
• \(\mathrm{H}_\mathrm{a}\) describes every other possible condition other than the null condition.

Examples:

Test statistics are standardized measures of deviation from the null-hypothesized condition.

Examples:

By chance alone, most samples will exhibit small differences from the population, some will exhibit larger differences, and a very few will exhibit very large differences.

The larger the test statistic, the less likely we are to observe that value due to chance alone when \(\mathrm{H}_0\) is true.

The integral of the appropriate probability density function outside of our test statistic (shaded) gives us \(p\), the probability of getting a test statistic larger than our observed test statistic by chance alone when \(\mathrm{H}_0\) is true.

Alternatively, we could use randomization methods to find \(p\).

• If p is small, reject \(\mathrm{H}_0\), and conclude \(\mathrm{H}_{\mathrm{a}}\) is true.
• If p is not small, fail to reject \(\mathrm{H}_0\), and conclude…
  • …our data is consistant with the null hypothesis.
  • …the null hypothesis is true.

• How small of a \(p\) is "small enough?" By convention, \(\alpha = 0.05\), but only by convention.
• Degrees of freedom are scaling parameters for the probability density functions, and are derived from the sample size and experimental design.
• The critical value is the smallest value of the test statistic that would still be considered "significant." These were important before desktop computers, but largely irrelevant today.

We could phrase our question several ways…

• Are men taller than women?
• Is there an effect of sex on height?
• Does height vary by sex, \(\mathrm{h} = f(\mathrm{s})\), where \(f(\mathrm{s})\) is a linear function parametized through OLS regression?

\(\mathrm{H}_0:\) \(\mu_\mathrm{F} = \mu_\mathrm{M}\)
\(\mathrm{H}_a:\) \(\mu_\mathrm{F} \neq \mu_\mathrm{M}\)

\(t = \frac{\bar{x}_1-\bar{x}_2}{{SE}_{1,2}}=\) -11.09, with 192.86 degrees of freedom.

\(p \approx\) 0 \(<0.05\), therefore reject \(\mathrm{H}_0\)

The probability of observing a \(t\) larger than -11.09 with 192.86 degrees of freedom by chance when the null hypothesis is true is less than 0.05. Therefore, we reject the null hypothesis in favor of the alternate hypothesis, and conclude that women (exhibiting here a mean of 63.8 inches) are shorter than men (exhibiting here a mean of 68.7 inches).

• Decision making criteria
• Well known
• Intended for small to moderately sized data

• \(p\neq\) the probability that \(\mathrm{H}_0\) is true
• Can never conclude \(\mathrm{H}_0\) is true
• Never assures a correct conclusion
  • \(\alpha\) = Type I error rate
  • \(\beta\) = Type II error rate
• Not applicable to Big Data
• Does not assess model fit
• Questionable model selection
• P hacking
  • Model and \(\alpha\) must be chosen a priori

Significant just means "measurable," not "important."

How much variability in \(Y\) is explained by \(f(x)\)?

\[ R^2 = \frac{SS_{regression}}{SS_{total}}\]

\(R^2\) is the proportion of variability explained by the regression, relative to the total variation in the data.

What \(f(x)\) best describes \(Y\)?

1. Fit many different \(f(x)\) to a set of training data
2. Evaluate the fit of those models on a new set of testing data
3. Pick \(f(x)\) that minimizes Testing Mean Squared Error

We call this cross-validation

Very flexible models can bring training error to almost 0.

…but those models become hyper-specific to the training set, and don't do a good job at predicting new data.

The \(f(x)\) that minimizes test error is the best at predicting new \(Y\).

• A statistical model is an approximation of a stochastic system.
• Simple models help us to understand a system.
• Complex models help us to predict a system.
• The linear model is a powerful model that can be either simple or complex.
• NHST answers the question "are \(Y\) and \(X\) related?"
• \(R^2\) answers the question "how much of \(Y\) is explained by my \(f(x)\)?"
• Machine learning answers the question "what is the best \(f(x)\) to predict \(Y\)?"