Interpreting the Output of a Logistic Regression Model

Introduction

This article looks at how to interpret the output of the glm() R function using the Titanic train dataset.

A note on the p-value: the p-value is a test of significance for the null hypothesis \(H_0\) that

• there is no difference in the log-odds of the outcome between the reference group (captured by the intercept) and the explanatory variable (or one of its categories), or that the difference between the two groups equals zero: \(H_0: b_1 = 0\) and \(H_a: b_1 \neq 0\).

If p<0.5, we reject \(H_0\) as we have evidence to suggest that the difference between the two groups does not equal zero.

Log-odds are not the most intuitive to interpret. Instead of discussing the change in the log-odds, we can calculate the odds ratio for a given variable by exponentiating the coefficient. Odds ratio is read “have x times the odds of the outcome of interest compared to thos in the reference group”.

Reference group of the outcome variable: by default, R creates uses the lowest coded group as the reference. The reference category can ce changed by using the ‘relevel()’. Here, ‘not survived’ (coded 0) is the reference group!

Relatonship between Odds and Probabilities:

• Odds=P/(1-P)
• P=odds/(1+odds)
• Odds=exp(log-odds)
• P=exp(log-odds)/(1+exp(log-odds))

Categorical Explanatory Variable (2 Categories)

• Survived is modelled as a function of Sex.
• Reference group is female¹.

model1 <- glm(Survived ~ Sex, family=binomial(link='logit'), data=data)
summary(model1)

Call:
glm(formula = Survived ~ Sex, family = binomial(link = "logit"), 
    data = data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.646  -0.647  -0.647   0.772   1.826  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)    1.057      0.129    8.19  2.6e-16 ***
Sexmale       -2.514      0.167  -15.04  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1186.7  on 890  degrees of freedom
Residual deviance:  917.8  on 889  degrees of freedom
AIC: 921.8

Number of Fisher Scoring iterations: 4

• Intercept: The log-odds of Survival for women is 1.057.
• Sexmale:
  • The difference in the log-odds of survival between men and women is -2.514 i.e. the chance of survival is lower for men than for women.
  • Given \(p < 0.5\), we can reject the null hypothesis (\(b_1=0\)) that there is no difference in the log-odds between men and women.
  • Men have 0.081² times the odds (i.e. -92 %)³ of survival than women.
• Null deviance = SSTot
• Residual deviance = SSResid

Categorical Explanatory Variable (More than 2 Categories)

• Survived is modeled as a function of AgeGroup.
• Note that this model is produced from the data without missing Age rows.
• The default reference group is 0-19.

model2 <- glm(Survived ~ AgeGroup, family=binomial(link='logit'), 
              data=na.omit(data))
summary(model2)

Call:
glm(formula = Survived ~ AgeGroup, family = binomial(link = "logit"), 
    data = na.omit(data))

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
 -1.15   -0.99   -0.99    1.36    1.62  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)  
(Intercept)    -0.0732     0.1563   -0.47    0.639  
AgeGroup20-39  -0.3842     0.1879   -2.04    0.041 *
AgeGroup40-59  -0.3567     0.2345   -1.52    0.128  
AgeGroup60+    -0.9253     0.4689   -1.97    0.048 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 964.52  on 713  degrees of freedom
Residual deviance: 957.93  on 710  degrees of freedom
AIC: 965.9

Number of Fisher Scoring iterations: 4

• Intercept: the log-odds of survival for someone aged 0-19 (reference group) is -0.073.
• For each of the other age groups, the coefficient tells us that the log-odds of survival for a given group is smaller than that of the reference group.
• Where \(p>0.5\), the null hypothesis \(b_k=0\) cannot be rejected i.e. there is insufficient statistical evidence that the chance of survival is significantly smaller compared to the reference group.

Now let’s change the reference group to 20-39:

• this is still nased on data without missing age rows.

levels(AgeGroup)

[1] "0-19"  "20-39" "40-59" "60+"  

data$AgeGroupR <- relevel(AgeGroup, ref=2)
levels(data$AgeGroupR) # new col is not attached so must use data$

[1] "20-39" "0-19"  "40-59" "60+"  

model3 <- glm(Survived ~ AgeGroupR, family=binomial(link='logit'), 
              data=na.omit(data))
summary(model3)

Call:
glm(formula = Survived ~ AgeGroupR, family = binomial(link = "logit"), 
    data = na.omit(data))

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
 -1.15   -0.99   -0.99    1.36    1.62  

Coefficients:
               Estimate Std. Error z value Pr(>|z|)    
(Intercept)     -0.4574     0.1043   -4.38  1.2e-05 ***
AgeGroupR0-19    0.3842     0.1879    2.04    0.041 *  
AgeGroupR40-59   0.0276     0.2036    0.14    0.892    
AgeGroupR60+    -0.5411     0.4543   -1.19    0.234    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 964.52  on 713  degrees of freedom
Residual deviance: 957.93  on 710  degrees of freedom
AIC: 965.9

Number of Fisher Scoring iterations: 4

• The findings are “worse”. Now there are two groups with \(p>0.5\).

Now let’s try with a different categorical age group variable AgeGroup2029:

levels(AgeGroup2029)

[1] "0" "1"

model2b <- glm(Survived ~ AgeGroup2029, family=binomial(link='logit'), 
              data=na.omit(data))
summary(model2b)

Call:
glm(formula = Survived ~ AgeGroup2029, family = binomial(link = "logit"), 
    data = na.omit(data))

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.062  -1.062  -0.928   1.297   1.449  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)   
(Intercept)    -0.2771     0.0908   -3.05   0.0023 **
AgeGroup20291  -0.3420     0.1680   -2.04   0.0418 * 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 964.52  on 713  degrees of freedom
Residual deviance: 960.32  on 712  degrees of freedom
AIC: 964.3

Number of Fisher Scoring iterations: 4

• Intercept: The log-odds of Survival for passengers that are not in the 20-29 age group is -0.277.
• AgeGroup20291:
  • The difference in the log-odds of survival between passengers in the 20-29 age group and those who are not is -0.342 i.e. the chance of survival is lower for passenger in the 20-29 age group.
  • Given \(p < 0.5\), we can reject the null hypothesis (\(b_1=0\)) that there is no difference in the log-odds between the two groups.
  • Passengers in the 20-29 age group have 0.081 times the odds (i.e. -29%) of survival than thos who are not in that age group.

Continuous Explanatory Variable

• Survived is modelled as a function of Age.
• There is no reference group as such.
• R “can deal with” missing data for a continuous variable i.e.throws no error du to missing data in the continuous variable.

model4 <- glm(Survived ~ Age, family=binomial(link='logit'), data=data)
summary(model4)

Call:
glm(formula = Survived ~ Age, family = binomial(link = "logit"), 
    data = data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.149  -1.036  -0.954   1.316   1.591  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept) -0.05672    0.17358   -0.33     0.74  
Age         -0.01096    0.00533   -2.06     0.04 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 964.52  on 713  degrees of freedom
Residual deviance: 960.23  on 712  degrees of freedom
  (177 observations deleted due to missingness)
AIC: 964.2

Number of Fisher Scoring iterations: 4

• Intercept: The log-odds of Survival when \(Age = 0\) is -0.057.
• Age:
  • For every unit increase in Age the log-odds of survival decrease by -0.011 i.e. the chances of survival decrease as passenger age increases.
  • Given \(p < 0.5\), we can reject the null hypothesis \(b_1=0\) that one unit increase in age does not affect chances of survival.
  • For every unit increase in Age, the odds of survival are 0.989 times the odds of those with one Age unit less (i.e. -1.09 %).

Now let’s try the same with imputed ages:

data.impute <- data %>% kNN() # Note: there will be imputed ages < 1 ...
model5 <- glm(Survived ~ Age, family=binomial(link='logit'), data=data.impute)
summary(model5)

Call:
glm(formula = Survived ~ Age, family = binomial(link = "logit"), 
    data = data.impute)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.003  -0.988  -0.977   1.381   1.424  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)   
(Intercept) -0.42433    0.16229   -2.61   0.0089 **
Age         -0.00172    0.00518   -0.33   0.7393   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1186.7  on 890  degrees of freedom
Residual deviance: 1186.5  on 889  degrees of freedom
AIC: 1191

Number of Fisher Scoring iterations: 4

• The model has worsened considerably. Given \(p > 0.5\), we cannot reject the null hypothesis \(b_1=0\) that one unit increase in age does not chnage the chances of survival.

Estimating the Odds of an outcome

To estimate the odds of an outcome from pluig in the relevant values into the model equation \(ln(odds) = b_0 + b_1x_1 + ... + b_kx_k\)

For example let’s estimate the odds of survival for

• a passenger who is aged between 20-29, and
• for a passenger who is not in that age group

model6 <- glm(Survived ~ AgeGroup2029, family=binomial(link='logit'), data=na.omit(data))
model6$coefficients

  (Intercept) AgeGroup20291 
       -0.277        -0.342 

• Reference group: AgeGroup20290 (i.e. those who are NOT aged 20-29).
• Recall that, for categorical explanatory variables, the intercept represents the log odds of the outcome for the reference group. Thus, in this case, the logg-odds of survival, if the passenger is not aged 20-29 are ln(odds)=\(b_0=\) -0.277 and the odds of survival is \(e^{b_0}\): 0.758.
• The log-odds of survival of those that are in the age range 20 to 29 are ln(odds)=\(b_0 + b_1\)= -0.619 and the odds of survival (aka odds ratio) is \(e^{b_0+b_1}\): 0.538.

Now, in terms of probabilities:

• The probability of not surviving for passengers aged 20-29 is P=exp(b_0)/(1+exp(b_0)): 0.431%.
• The probability of not surviving for passengers not aged 20-29 is P=exp(log-odds)/(1+exp(b_0+b_1)): 0.35%.

---

1. By default R assigns the reference group based on alphabetical order↩

2. log-odds to odds-ratio: exp(log-odds)↩

3. odds-ratio to %: (OR-1) * 100↩