This homework is designed to give you practice fitting a logistic regression and working with statistical/philosophical measures of fairness. We will work with the titanic dataset which we have previously seen in class in connection to decision trees.

Below I will preprocess the data precisely as we did in class. You can simply refer to data_train as your training data and data_test as your testing data.

#this is all of the preprocessing done for the decision trees lecture.  

path <- 'https://raw.githubusercontent.com/guru99-edu/R-Programming/master/titanic_data.csv'
titanic <-read.csv(path)
head(titanic)
##   x pclass survived                                            name    sex
## 1 1      1        1                   Allen, Miss. Elisabeth Walton female
## 2 2      1        1                  Allison, Master. Hudson Trevor   male
## 3 3      1        0                    Allison, Miss. Helen Loraine female
## 4 4      1        0            Allison, Mr. Hudson Joshua Creighton   male
## 5 5      1        0 Allison, Mrs. Hudson J C (Bessie Waldo Daniels) female
## 6 6      1        1                             Anderson, Mr. Harry   male
##      age sibsp parch ticket     fare   cabin embarked
## 1     29     0     0  24160 211.3375      B5        S
## 2 0.9167     1     2 113781   151.55 C22 C26        S
## 3      2     1     2 113781   151.55 C22 C26        S
## 4     30     1     2 113781   151.55 C22 C26        S
## 5     25     1     2 113781   151.55 C22 C26        S
## 6     48     0     0  19952    26.55     E12        S
##                         home.dest
## 1                    St Louis, MO
## 2 Montreal, PQ / Chesterville, ON
## 3 Montreal, PQ / Chesterville, ON
## 4 Montreal, PQ / Chesterville, ON
## 5 Montreal, PQ / Chesterville, ON
## 6                    New York, NY
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
#replace ? with NA
replace_question_mark <- function(x) {
  if (is.character(x)) {
    x <- na_if(x, "?")
  }
  return(x)
}

titanic <- titanic %>%
  mutate_all(replace_question_mark)

set.seed(678)
shuffle_index <- sample(1:nrow(titanic))
head(shuffle_index)
## [1]   57  774  796 1044  681  920
titanic <- titanic[shuffle_index, ]
head(titanic)
##         x pclass survived                                             name
## 57     57      1        1                       Carter, Mr. William Ernest
## 774   774      3        0                                 Dimic, Mr. Jovan
## 796   796      3        0                          Emir, Mr. Farred Chehab
## 1044 1044      3        1                      Murphy, Miss. Margaret Jane
## 681   681      3        0                                Boulos, Mr. Hanna
## 920   920      3        0 Katavelas, Mr. Vassilios ('Catavelas Vassilios')
##         sex  age sibsp parch ticket   fare   cabin embarked     home.dest
## 57     male   36     1     2 113760    120 B96 B98        S Bryn Mawr, PA
## 774    male   42     0     0 315088 8.6625    <NA>        S          <NA>
## 796    male <NA>     0     0   2631  7.225    <NA>        C          <NA>
## 1044 female <NA>     1     0 367230   15.5    <NA>        Q          <NA>
## 681    male <NA>     0     0   2664  7.225    <NA>        C         Syria
## 920    male 18.5     0     0   2682 7.2292    <NA>        C          <NA>
library(dplyr)
# Drop variables
clean_titanic <- titanic %>%
select(-c(home.dest, cabin, name, x, ticket)) %>% 
#Convert to factor level
    mutate(pclass = factor(pclass, levels = c(1, 2, 3), labels = c('Upper', 'Middle', 'Lower')),
    survived = factor(survived, levels = c(0, 1), labels = c('No', 'Yes'))) %>%
na.omit()
#previously were characters
clean_titanic$age <- as.numeric(clean_titanic$age)
clean_titanic$fare <- as.numeric(clean_titanic$fare)
glimpse(clean_titanic)
## Rows: 1,043
## Columns: 8
## $ pclass   <fct> Upper, Lower, Lower, Middle, Lower, Middle, Lower, Lower, Upp…
## $ survived <fct> Yes, No, No, No, No, No, No, No, Yes, No, Yes, No, No, Yes, N…
## $ sex      <chr> "male", "male", "male", "male", "female", "female", "male", "…
## $ age      <dbl> 36.0, 42.0, 18.5, 44.0, 19.0, 26.0, 23.0, 28.5, 64.0, 36.5, 4…
## $ sibsp    <int> 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0…
## $ parch    <int> 2, 0, 0, 0, 0, 1, 0, 0, 2, 2, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0…
## $ fare     <dbl> 120.0000, 8.6625, 7.2292, 13.0000, 16.1000, 26.0000, 7.8542, …
## $ embarked <chr> "S", "S", "C", "S", "S", "S", "S", "S", "C", "S", "S", "S", "…
create_train_test <- function(data, size = 0.8, train = TRUE) {
    n_row = nrow(data)
    total_row = size * n_row
    train_sample <- 1: total_row
    if (train == TRUE) {
        return (data[train_sample, ])
    } else {
        return (data[-train_sample, ])
    }
}
data_train <- create_train_test(clean_titanic, 0.8, train = TRUE)
data_test <- create_train_test(clean_titanic, 0.8, train = FALSE)

Create a table reporting the proportion of people in the training set surviving the Titanic. Do the same for the testing set. Comment on whether the current training-testing partition looks suitable.

train_survivors_prop <- mean(data_train$survived == "Yes")

test_survivors_prop <- mean(data_test$survived == "Yes")

survivors_table <- data.frame(
  Dataset = c("Training", "Testing"),
  Proportion_of_Survivors = c(train_survivors_prop, test_survivors_prop)
)

survivors_table
##    Dataset Proportion_of_Survivors
## 1 Training               0.3980815
## 2  Testing               0.4449761

student input

Use the glm command to build a logistic regression on the training partition. survived should be your response variable and pclass, sex, age, sibsp, and parch should be your response variables.

logit_model <- glm(survived ~ pclass + sex + age + sibsp + parch, data = data_train, family = binomial)

summary(logit_model)
## 
## Call:
## glm(formula = survived ~ pclass + sex + age + sibsp + parch, 
##     family = binomial, data = data_train)
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   3.903165   0.409280   9.537  < 2e-16 ***
## pclassMiddle -1.291506   0.257421  -5.017 5.25e-07 ***
## pclassLower  -2.404084   0.262022  -9.175  < 2e-16 ***
## sexmale      -2.684206   0.200130 -13.412  < 2e-16 ***
## age          -0.036776   0.007494  -4.907 9.24e-07 ***
## sibsp        -0.395584   0.118587  -3.336  0.00085 ***
## parch         0.032494   0.111916   0.290  0.77155    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1121.27  on 833  degrees of freedom
## Residual deviance:  757.87  on 827  degrees of freedom
## AIC: 771.87
## 
## Number of Fisher Scoring iterations: 5

We would now like to test whether this classifier is fair across the sex subgroups. It was reported that women and children were prioritized on the life-boats and as a result survived the incident at a much higher rate. Let us see if our model is able to capture this fact.

Subset your test data into a male group and a female group. Then, use the predict function on the male testing group to come up with predicted probabilities of surviving the Titanic for each male in the testing set. Do the same for the female testing group.

male_test_data <- subset(data_test, sex == "male")
female_test_data <- subset(data_test, sex == "female")


male_predicted_probabilities <- predict(logit_model, newdata = male_test_data, type = "response")
female_predicted_probabilities <- predict(logit_model, newdata = female_test_data, type = "response")

mean(male_predicted_probabilities)
## [1] 0.1979176
mean(female_predicted_probabilities)
## [1] 0.7710022

Now recall that for this logistic regression to be a true classifier, we need to pair it with a decision boundary. Use an if-else statement to translate any predicted probability in the male group greater than \(0.5\) into Yes (as in Yes this individual is predicted to have survived). Likewise an predicted probability less than \(0.5\) should be translated into a No.

Do this for the female testing group as well, and then create a confusion matrix for each of the male and female test set predictions. You can use the confusionMatrix command as seen in class to expidite this process as well as provide you necessary metrics for the following questions.

library(caret)
## Warning: package 'caret' was built under R version 4.3.3
## Loading required package: ggplot2
## Loading required package: lattice
male_test_data$survived <- factor(male_test_data$survived, levels = c("No", "Yes"))
female_test_data$survived <- factor(female_test_data$survived, levels = c("No", "Yes"))

male_predictions <- ifelse(male_predicted_probabilities > 0.5, "Yes", "No")
female_predictions <- ifelse(female_predicted_probabilities > 0.5, "Yes", "No")

male_confusion_matrix <- confusionMatrix(factor(male_predictions), reference = male_test_data$survived)
female_confusion_matrix <- confusionMatrix(factor(female_predictions), reference = female_test_data$survived)


male_confusion_matrix
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction No Yes
##        No  93  28
##        Yes  4   4
##                                           
##                Accuracy : 0.7519          
##                  95% CI : (0.6682, 0.8237)
##     No Information Rate : 0.7519          
##     P-Value [Acc > NIR] : 0.5473          
##                                           
##                   Kappa : 0.1119          
##                                           
##  Mcnemar's Test P-Value : 4.785e-05       
##                                           
##             Sensitivity : 0.9588          
##             Specificity : 0.1250          
##          Pos Pred Value : 0.7686          
##          Neg Pred Value : 0.5000          
##              Prevalence : 0.7519          
##          Detection Rate : 0.7209          
##    Detection Prevalence : 0.9380          
##       Balanced Accuracy : 0.5419          
##                                           
##        'Positive' Class : No              
## 
female_confusion_matrix
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction No Yes
##        No   4   2
##        Yes 15  59
##                                           
##                Accuracy : 0.7875          
##                  95% CI : (0.6817, 0.8711)
##     No Information Rate : 0.7625          
##     P-Value [Acc > NIR] : 0.354209        
##                                           
##                   Kappa : 0.2325          
##                                           
##  Mcnemar's Test P-Value : 0.003609        
##                                           
##             Sensitivity : 0.2105          
##             Specificity : 0.9672          
##          Pos Pred Value : 0.6667          
##          Neg Pred Value : 0.7973          
##              Prevalence : 0.2375          
##          Detection Rate : 0.0500          
##    Detection Prevalence : 0.0750          
##       Balanced Accuracy : 0.5889          
##                                           
##        'Positive' Class : No              
## 

We can see that indeed, at least within the testing groups, women did seem to survive at a higher proportion than men (24.8% to 76.3% in the testing set). Print a summary of your trained model and interpret one of the fitted coefficients in light of the above disparity.

summary(logit_model)
## 
## Call:
## glm(formula = survived ~ pclass + sex + age + sibsp + parch, 
##     family = binomial, data = data_train)
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   3.903165   0.409280   9.537  < 2e-16 ***
## pclassMiddle -1.291506   0.257421  -5.017 5.25e-07 ***
## pclassLower  -2.404084   0.262022  -9.175  < 2e-16 ***
## sexmale      -2.684206   0.200130 -13.412  < 2e-16 ***
## age          -0.036776   0.007494  -4.907 9.24e-07 ***
## sibsp        -0.395584   0.118587  -3.336  0.00085 ***
## parch         0.032494   0.111916   0.290  0.77155    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1121.27  on 833  degrees of freedom
## Residual deviance:  757.87  on 827  degrees of freedom
## AIC: 771.87
## 
## Number of Fisher Scoring iterations: 5

The coefficient for the sexmale variable is -2.684206, which means that when all other variables are held constant, being male is associated with a decrease in the log odds of survival by approximately 2.684206 units compared to being female. This coefficient indicates that being male was a significant predictor of death on the Titanic and highlights the gender disparity in survival on the Titanic.

Now let’s see if our model is fair across this explanatory variable. Calculate five measures (as defined in class) in this question: the Overall accuracy rate ratio between females and males, the disparate impact between females and males, the statistical parity between females and males, and the predictive equality as well as equal opportunity between females and males (collectively these last two comprise equalized odds). Set a reasonable \(\epsilon\) each time and then comment on which (if any) of these five criteria are met.

male_predictions <- ifelse(male_predicted_probabilities > 0.5, 1, 0)
female_predictions <- ifelse(female_predicted_probabilities > 0.5, 1, 0)

male_true_labels <- ifelse(male_test_data$survived == "Yes", 1, 0)
female_true_labels <- ifelse(female_test_data$survived == "Yes", 1, 0)

# Overall accuracy ratio
overall_accuracy_ratio <- sum(female_predictions == female_true_labels) / sum(male_predictions == male_true_labels)

# Disparate impact
disparate_impact <- sum(female_predictions) / sum(male_predictions)

# Statistical parity
statistical_parity <- abs((sum(female_predictions) / sum(male_predictions)) - 1)

# False Positive Rate (FPR) for males and females
male_fpr <- sum(male_predictions == 1 & male_true_labels == 0) / sum(male_true_labels == 0)
female_fpr <- sum(female_predictions == 1 & female_true_labels == 0) / sum(female_true_labels == 0)

# Predictive equality
predictive_equality <- abs(female_fpr - male_fpr)

# Equal opportunity
male_fnr <- sum(male_predictions == 0 & male_true_labels == 1) / sum(male_true_labels == 1)
female_fnr <- sum(female_predictions == 0 & female_true_labels == 1) / sum(female_true_labels == 1)
equal_opportunity <- abs(female_fnr - male_fnr)


overall_accuracy_ratio
## [1] 0.6494845
disparate_impact
## [1] 9.25
statistical_parity
## [1] 8.25
predictive_equality
## [1] 0.7482366
equal_opportunity
## [1] 0.8422131

The model does not meet the criteria for fairness. The disparate impact, statistical parity, and overall accuracy rate ratio suggest potential disparities in predictions between females and males. The predictive equality and equal opportunity measures show relatively lower disparities in false positive and false negative rates between the two groups, but it’s not enough to reduce make the model fair..

It is always important for us to interpret our results in light of the original data and the context of the analysis. In this case, it is relevant that we are analyzing a historical event post-facto and any disparities across demographics identified are unlikely to be replicated. So even though our model fails numerous of the statistical fairness criteria, I would argue we need not worry that our model could be misused to perpetuate discrimination in the future. After all, this model is likely not being used to prescribe a preferred method of treatment in the future.

Even so, provide a philosophical notion of justice or fairness that may have motivated the Titanic survivors to act as they did. Spell out what this philosophical notion or principle entails?

Virtue ethics may have motivated survival outcomes on the Titanic because people believed it was a man’s duty to protect his wife and children which meant sacrificing his own life so they would survive. The virtues of duty and chivalry were pretty universal, especially at that time, so women and children were saved first.

Note that the echo = FALSE parameter was added to the code chunk to prevent printing of the R code that generated the plot.