Learning Objectives

1. Understand what generalized linear models are
2. Understand the use of logistic models and its assumptions
3. Know how to estimate logistic models in R, check assumptions and interpret results
4. Basic knowledge of Receiver Operating Characteristic (ROC) Curves and understand what AUC is (this may show up as extra credit on an exam)

What is a generalized linear model (GLM)?

• GLMs were formally introduced in a seminal paper by John Nelder and Robert Wedderburn in 1972

• They broadened the applicability of linear models to other types of dependent variables with error distributions that were not normally distributed

  Dependent variable nature	Example	Model
  Binary (2 levels)	Diabetes (yes, no)	Logisitic
  Multinomial (2+ levels)	Stage at diagnosis (I-IV)	Multinomial
  Ordinal (Ordered levels)	Likert scales (1-5)	Ordinal
  Count outcome	Number of colds, number of cancer cases	Poisson, negative binomial

  

Link function

• Every GLM has a link function

• Transforms the dependent variable to a linear form

The logistic model

• Why not use the linear model for a binary outcome variable that has the values of 0 and 1 to predict the probability of the outcome as a function of the X variables?
• It has several limitations including:
  • Predicted probabilities fall outside of [0,1]
    • How do you interpret a negative probability of the outcome?
  • Violation of assumptions
    • Homoscedasticity that assumes constant variance, which causes biased standard errors
    • Linearity assumption between x variables and the outcome
    • Errors are not normally distributed because y can only take the values of 0 and 1

The probability can be constrained between 0 and 1 by taking the natural log of the odds, which has a range from -∞ to ∞ solving our problem

The logistic model

• Link function: logit

• We model the log of the probability of success divided by the probability of failure (i.e. the odds)

• The intercept in the logistic model is the baseline log odds when all predictors are zero

• If you exponentiate the intercept you get the odds of the outcome when all predictors are zero

• \(\beta_1\) tells you the log odds of the outcome for a one unit increase in X

• If you exponentiate \(\beta_1\) you get fold change in the odds of the outcome for a one unit increase in X (i.e. the Odds ratio)

Predicted probabilities

• You can also calculate the predicted probabilities from this model.

Research questions and hypotheses

• Does age, ethnicity, or BMI influence the probability of being diagnosed with heart disease?

  Hypothesis

• \(H_0\): The probability of having heart disease is not associated with BMI.

• \(H_a\): The probability of having heart disease is associated with BMI.

Assessing model fit for the logistic model

1. Likelihood ratio test
2. Wald test
3. Pseudo R2
4. Hosmer and Lemeshow GOF test
5. Wald test for individual variables

1. Likelihood Ratio Test (LR test or LRT)

What it is: Compares the fit of two nested models (a full model and a reduced model with the same number of observations). The reduced model is a special case of the full model with one or more variables omitted.

Purpose: To test whether adding one or more predictors significantly improves the models ability to explain the data.

How it works:

• Calculate the log-likelihoods of the reduced and full models

• Compute the test statistic (\(G^2\)=−2(lnLreduced​−lnLfull​)

This statistic follows a chi-square distribution with degrees of freedom equal to the difference in the number of estimated parameters (df between the models).

Significant p-value: The full model provides a better fit than the reduced model

LR test example

• Compare two models to determine if sex improves the model fit for the log odds of heart disease in association with income using the LRT:

Reduced model: log odds of heart disease= \(\beta_0\) + \(\beta_1\)*income

We could then ask if sex improves the model fit:

Full model: log odds of heart disease= \(\beta_0\) + \(\beta_1\)*income + \(\beta_2\)*sex

The reduced and full model can be compared using the LRT. A significant p-value indicates an improvement in model fit.

2. Wald test (for more than one predictor)

What it is: Compares whether a set of regression coefficients are all equal to zero. Can test blocks of predictors.

Purpose: To determine if a group of predictors improves the model’s ability to predict the outcome.

• \(H_0\): The parameters of the predictor variables are equal to zero, there is no change in the log odds in association with the predictors for \(\beta_1\) to \(\beta_i\)

• \(H_a\): At least one of the parameters is not equal to zero

How it works:

• Estimate the coefficients and their variance-covariance matrix

• Compute the Wald test statistic

• The statistic follows a chi-square distribution with degrees of freedom equal to the number of coefficients tested

Significant p-value: At least one of coefficient is not zero (the block improves model fit)

3. Pseudo \(R^2\)

What it is: There is no \(R^2\) like in linear regression that gives the percent of variance explained by the model. Instead, there is a pseudo R2 with several variations (McFadden’s, Cox and Snell’s, Nagelkerke’s)

Purpose: To mimic the \(R^2\)

• McFadden’s R2 is calculated as below and has the bounds of 0 to <1

Interpretation of Pseudo \(R^2\): higher values indicate better fit. There is no interpretation like \(R^2\) in linear regression

Suggest not using (more discussion on that can be found at https://statisticalhorizons.com/r2logistic/#)

4. Hosmer and Lemeshow Goodness of Fit test

What it is: A statistical test that checks how well the model’s predicted probabilities agree with the actual observed outcomes.

Purpose: To evaluate whether the logistic regression model provides an adequate overall fit to the data.

How it works:

1. Sort subjects by predicted probability of the outcome.

2. Divide them into groups (often 10 groups, or “deciles”)

3. For each group, compare the number of observed events vs. the number expected from the model.

4. Calculate a chi-square statistic across all groups.

Interpretation:

High p-value (e.g., >0.05): The predicted and observed outcomes match reasonably well. There is no evidence of poor fit.

Low p-value (e.g., <0.05): The model predictions differ from what was observed. The model may not fit adequately.

Note: In very large samples, even small differences can lead to a significant result.

5. Wald test (for one predictor)

What it is: Tests whether an individual regression coefficient is equal to zero.

Purpose: To evaluate whether a single predictor has a statistically significant association with the outcome after adjusting for all other predictors in the model.

How it works:

1. Estimate \(\beta_i\) and its standard error
2. Compute the Wald statistic according to the formula:

\(Z\) = \(\beta\)/\(SE\)

3. Square of this statistic follows a chi-square distribution with 1 degree of freedom

Interpretation:

Significant p-value: The predictor is associated with the outcome.

Assumptions

Testing assumptions

Linearity assumption: In linear regression, each continous predictor is assumed to have a linear association with the outcome. In logistic regression, each continuous predictor is assumed to have a linear association with the log odds of the outcome.

Box Tidwell test can check this

• For each continuous term, add an interaction term with the natural log of its outcome to the model.

• A non-significant coefficient indicates no violation of linearity.

• A significant coefficient indicates violation of linearity.

Multicollinearity: This happens when predictors are highly correlated with each other (i.e. they measure similar or the same thing). This makes it difficult to estimate their individual effects on the outcome.

The VIF test can check this.

• VIF = 1: no correlation

• Several cutoffs of concern are suggested: 3, 4, 5, and 10. VIF values > cutoff suggest multicollinearity problems.

• Use GVIF (output by R VIF function) for categorical variables

Testing assumptions

Influential observations

• Cook’s Distance: calculates a measure of influence for each observation that can be plotted against the observation number. May be useful for identifying erroneous data.

• DFFITS: measures change in predicted values

• DFBETAs: measures change in model estimates

Typically, one should not remove influential values unless there is a very good reason to do so. Avoid “cherry-picking” and “p-hacking”. Be very clear about what data was removed and why it was removed when reporting results.

Interpreting odds ratios

• Odds ratios are calculated by exponentiating the b estimates
• Odds ratios > 1 are interpreted as an increase in the odds given a one unit change in x.
• Odds ratio = 1, no difference
• Odds ratios < 1 are interpreted as a decrease in the odds given a one unit change in x
• OR = 2: 2 times greater odds
• OR = 1.2: 20% greater odds
• OR = 0.6: 0.6 times lower odds or 40% lower odds

Example interpretations

Categorical exposure

OR = 5.65 and 95% CI (3.98 to 8.36): Compared to those who were categorized as underweight, the odds of reporting diabetes was 5.65 (95% CI 3.98 to 8.36) times higher for those who were categorized as obese.

The odds of diabetes in individuals in the obese BMI category is 5.65 (95% CI 3.98 to 8.36) times higher than the odds of diabetes in individuals in the underweight category.

OR = 0.18 and 95% CI (0.12 to 0.25): The odds of diabetes in those who are underweight is 0.18 (95% CI 0.12 to 0.25) times the odds of diabetes in those who are obese.

The odds of diabetes in individuals in the underweight bmi category is 82% (OR = 0.18, 95% CI 0.12 to 0.25) lower than the odds of diabetes in individuals in the obese category.

Continuous exposure

OR = 1.09 (95% CI 1.08 to 1.10). For every one unit increase in BMI, the odds of diabetes increases by 9% (OR = 1.09, 95% CI 1.08 to 1.10).

For every one unit increase in BMI, the odds of diabetes increases by 1.09 (95% CI 1.08 to 1.10) times.

OPTIONAL MATERIAL TO REVIEW ON YOUR OWN (or with Kim or Kyle in an office hour appointment)

Predictive modeling

• Very hot topic in machine learning
• Will only use logistic regression as an example here
• Motivating question: How well does a model predict an outcome based on one or more variables?
• The is in contrast to explanatory modeling where we are more interested in whether a single variable is associated with the dependent variable after controlling for confounding.
• Explanatory modeling has the goal of making causal inferences.
• Predictive modeling has the goal of predicting individual risk based on a set of variables.

Recall some basic relevant concepts from Epidemiology

• Sensitivity: The probability of someone with a condition will test positive for that condition.

• Specificity: The probability of someone without a condition will test negative for that condition.

• We can apply these concepts to a logistic model.

  • The gold standard: The actual data

  • The test: The logistic model

• From actual outcome and model predicted outcomes for each individual, we can calculate sensitivity and specificity.

Predicted probabilities and model predicted outcomes

• We can calculate predicted probabilities by plugging in an individual’s values for variables into the logistic model formula that we obtain after running a regression

• The predicted probability of the outcome for each individual will range from 0-1 (0-100%).

• We can set a threshold to determine the model-predicted outcome (1 = yes, 0 = no) based on each individual’s predicted probability.

• For example, using a 0.5 threshold, individuals with predicted probabilities of 0.5 (50%) or higher are given a model predicted outcome of 1. Those with a value of <0.5 (<50%) are given a model predicted outcome of 0.

• We can calculate the sensitivity and specificity of the model using the predicted outcomes for each individual for different thresholds.

• Example: For a threshold of 0.5, our model correctly predicts 10 of the 100 individuals who have the outcome and 90 of the 100 individuals who do not have the outcome. Sensitivity = 10%, Specificity = 90%

  	Actual outcome =1	Actual outcome = 0
  Model outcome = 1	10	10
  Model outcome = 0	90	90
  	100	100

• If we change this threshold, we can change the sensitivity and specificity of the model

• Example: For a threshold of 0.3, our model now correctly predicts 20 of the 100 individuals who have the outcome and 80 of the 100 individuals who do not have the outcome. Sensitivity = 20%, Specificity = 80%.

  	Actual outcome =1	Actual outcome = 0
  Model outcome = 1	20	20
  Model outcome = 0	80	80
  	100	100

• If the threshold is set at a predicted probability of zero (meaning all individuals will be classified as having the outcome), the model will be 100% sensitive. If set at 1, the model will be be 100% specific

Receiver Operating Characteristic (ROC) curves

• ROC curves plot the sensitivity against the false positive rate (1- specificity) across thresholds from 0-1 (or 0 to 100).
• The blue line is an example of the resulting curve
• The diagonal dashed line indicates a random guess line
• A ROC curve overlapping with the diagonal line means your model is no better at predicting the outcome for individuals in your dataset than chance (i.e. flipping a fair coin)

Area under the curve (AUC)

• We can calculate the AUC under our ROC curve to provide us one summary measure of how well our model predicts the outcome.
• The AUC is a measure of the model’s performance across all possible classification thresholds.
• It can be interpreted as the probability that a randomly chosen true positive case will be ranked higher than a randomly chosen true negative case.
• The AUC for the dashed line is 0.5.
• An AUC of 0.5 is no better than a random guess
• The higher the AUC, the better it is at predicting cases
• Example interpretation: AUC= 0.85: “The probability that a randomly chosen HIV-positive person will be ranked higher by the model than a randomly chosen HIV-negative person is 85%”

AUC value interpretation

• From: https://pmc.ncbi.nlm.nih.gov/articles/PMC8831439/

0.9 ≤ AUC Excellent

0.8 ≤ AUC < 0.9 Good

0.7 ≤ AUC < 0.8 Fair

0.6 ≤ AUC < 0.7 Poor

0.5 ≤ AUC < 0.6 Fail

• AUC must be > 0.5 for a meaningful diagnostic test.
• Generally, AUC ≥ 0.8 is acceptable

ROC Curve Application Examples

• Spam filters (https://arxiv.org/pdf/0910.2540)

• Disease markers (https://www.e-epih.org/journal/view.php?doi=10.4178%2Fepih.e2022088&utm_source=chatgpt.com)

• Diagnostic tests (https://www.ahajournals.org/doi/10.1161/circulationaha.105.594929?utm_source=chatgpt.com)