This homework is designed to give you practice fitting a logistic
regression and working with statistical/philosophical measures of
fairness. We will work with the titanic dataset which we
have previously seen in class in connection to decision trees.
Below I will preprocess the data precisely as we did in class. You
can simply refer to data_train as your training data and
data_test as your testing data.
#this is all of the preprocessing done for the decision trees lecture.
path <- 'https://raw.githubusercontent.com/guru99-edu/R-Programming/master/titanic_data.csv'
titanic <-read.csv(path)
head(titanic)
## x pclass survived name sex
## 1 1 1 1 Allen, Miss. Elisabeth Walton female
## 2 2 1 1 Allison, Master. Hudson Trevor male
## 3 3 1 0 Allison, Miss. Helen Loraine female
## 4 4 1 0 Allison, Mr. Hudson Joshua Creighton male
## 5 5 1 0 Allison, Mrs. Hudson J C (Bessie Waldo Daniels) female
## 6 6 1 1 Anderson, Mr. Harry male
## age sibsp parch ticket fare cabin embarked
## 1 29 0 0 24160 211.3375 B5 S
## 2 0.9167 1 2 113781 151.55 C22 C26 S
## 3 2 1 2 113781 151.55 C22 C26 S
## 4 30 1 2 113781 151.55 C22 C26 S
## 5 25 1 2 113781 151.55 C22 C26 S
## 6 48 0 0 19952 26.55 E12 S
## home.dest
## 1 St Louis, MO
## 2 Montreal, PQ / Chesterville, ON
## 3 Montreal, PQ / Chesterville, ON
## 4 Montreal, PQ / Chesterville, ON
## 5 Montreal, PQ / Chesterville, ON
## 6 New York, NY
library(dplyr)
##
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
##
## filter, lag
## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union
#replace ? with NA
replace_question_mark <- function(x) {
if (is.character(x)) {
x <- na_if(x, "?")
}
return(x)
}
titanic <- titanic %>%
mutate_all(replace_question_mark)
set.seed(678)
shuffle_index <- sample(1:nrow(titanic))
head(shuffle_index)
## [1] 57 774 796 1044 681 920
titanic <- titanic[shuffle_index, ]
head(titanic)
## x pclass survived name
## 57 57 1 1 Carter, Mr. William Ernest
## 774 774 3 0 Dimic, Mr. Jovan
## 796 796 3 0 Emir, Mr. Farred Chehab
## 1044 1044 3 1 Murphy, Miss. Margaret Jane
## 681 681 3 0 Boulos, Mr. Hanna
## 920 920 3 0 Katavelas, Mr. Vassilios ('Catavelas Vassilios')
## sex age sibsp parch ticket fare cabin embarked home.dest
## 57 male 36 1 2 113760 120 B96 B98 S Bryn Mawr, PA
## 774 male 42 0 0 315088 8.6625 <NA> S <NA>
## 796 male <NA> 0 0 2631 7.225 <NA> C <NA>
## 1044 female <NA> 1 0 367230 15.5 <NA> Q <NA>
## 681 male <NA> 0 0 2664 7.225 <NA> C Syria
## 920 male 18.5 0 0 2682 7.2292 <NA> C <NA>
library(dplyr)
# Drop variables
clean_titanic <- titanic %>%
select(-c(home.dest, cabin, name, x, ticket)) %>%
#Convert to factor level
mutate(pclass = factor(pclass, levels = c(1, 2, 3), labels = c('Upper', 'Middle', 'Lower')),
survived = factor(survived, levels = c(0, 1), labels = c('No', 'Yes'))) %>%
na.omit()
#previously were characters
clean_titanic$age <- as.numeric(clean_titanic$age)
clean_titanic$fare <- as.numeric(clean_titanic$fare)
glimpse(clean_titanic)
## Rows: 1,043
## Columns: 8
## $ pclass <fct> Upper, Lower, Lower, Middle, Lower, Middle, Lower, Lower, Upp…
## $ survived <fct> Yes, No, No, No, No, No, No, No, Yes, No, Yes, No, No, Yes, N…
## $ sex <chr> "male", "male", "male", "male", "female", "female", "male", "…
## $ age <dbl> 36.0, 42.0, 18.5, 44.0, 19.0, 26.0, 23.0, 28.5, 64.0, 36.5, 4…
## $ sibsp <int> 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0…
## $ parch <int> 2, 0, 0, 0, 0, 1, 0, 0, 2, 2, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0…
## $ fare <dbl> 120.0000, 8.6625, 7.2292, 13.0000, 16.1000, 26.0000, 7.8542, …
## $ embarked <chr> "S", "S", "C", "S", "S", "S", "S", "S", "C", "S", "S", "S", "…
create_train_test <- function(data, size = 0.8, train = TRUE) {
n_row = nrow(data)
total_row = size * n_row
train_sample <- 1: total_row
if (train == TRUE) {
return (data[train_sample, ])
} else {
return (data[-train_sample, ])
}
}
data_train <- create_train_test(clean_titanic, 0.8, train = TRUE)
data_test <- create_train_test(clean_titanic, 0.8, train = FALSE)
Create a table reporting the proportion of people in the training set surviving the Titanic. Do the same for the testing set. Comment on whether the current training-testing partition looks suitable.
train_survivors_prop <- mean(data_train$survived == "Yes")
test_survivors_prop <- mean(data_test$survived == "Yes")
survivors_table <- data.frame(
Dataset = c("Training", "Testing"),
Proportion_of_Survivors = c(train_survivors_prop, test_survivors_prop)
)
survivors_table
## Dataset Proportion_of_Survivors
## 1 Training 0.3980815
## 2 Testing 0.4449761
student input
Use the glm command to build a logistic regression on
the training partition. survived should be your response
variable and pclass, sex, age,
sibsp, and parch should be your response
variables.
logit_model <- glm(survived ~ pclass + sex + age + sibsp + parch, data = data_train, family = binomial)
summary(logit_model)
##
## Call:
## glm(formula = survived ~ pclass + sex + age + sibsp + parch,
## family = binomial, data = data_train)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 3.903165 0.409280 9.537 < 2e-16 ***
## pclassMiddle -1.291506 0.257421 -5.017 5.25e-07 ***
## pclassLower -2.404084 0.262022 -9.175 < 2e-16 ***
## sexmale -2.684206 0.200130 -13.412 < 2e-16 ***
## age -0.036776 0.007494 -4.907 9.24e-07 ***
## sibsp -0.395584 0.118587 -3.336 0.00085 ***
## parch 0.032494 0.111916 0.290 0.77155
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1121.27 on 833 degrees of freedom
## Residual deviance: 757.87 on 827 degrees of freedom
## AIC: 771.87
##
## Number of Fisher Scoring iterations: 5
We would now like to test whether this classifier is fair across the sex subgroups. It was reported that women and children were prioritized on the life-boats and as a result survived the incident at a much higher rate. Let us see if our model is able to capture this fact.
Subset your test data into a male group and a female group. Then, use
the predict function on the male testing group to come up
with predicted probabilities of surviving the Titanic for each male in
the testing set. Do the same for the female testing group.
male_test_data <- subset(data_test, sex == "male")
female_test_data <- subset(data_test, sex == "female")
male_predicted_probabilities <- predict(logit_model, newdata = male_test_data, type = "response")
female_predicted_probabilities <- predict(logit_model, newdata = female_test_data, type = "response")
mean(male_predicted_probabilities)
## [1] 0.1979176
mean(female_predicted_probabilities)
## [1] 0.7710022
Now recall that for this logistic regression to be a true
classifier, we need to pair it with a decision boundary. Use an
if-else statement to translate any predicted probability in
the male group greater than \(0.5\)
into Yes (as in Yes this individual is predicted to have
survived). Likewise an predicted probability less than \(0.5\) should be translated into a
No.
Do this for the female testing group as well, and then create a
confusion matrix for each of the male and female test set predictions.
You can use the confusionMatrix command as seen in class to
expidite this process as well as provide you necessary metrics for the
following questions.
library(caret)
## Warning: package 'caret' was built under R version 4.3.3
## Loading required package: ggplot2
## Loading required package: lattice
male_test_data$survived <- factor(male_test_data$survived, levels = c("No", "Yes"))
female_test_data$survived <- factor(female_test_data$survived, levels = c("No", "Yes"))
male_predictions <- ifelse(male_predicted_probabilities > 0.5, "Yes", "No")
female_predictions <- ifelse(female_predicted_probabilities > 0.5, "Yes", "No")
male_confusion_matrix <- confusionMatrix(factor(male_predictions), reference = male_test_data$survived)
female_confusion_matrix <- confusionMatrix(factor(female_predictions), reference = female_test_data$survived)
male_confusion_matrix
## Confusion Matrix and Statistics
##
## Reference
## Prediction No Yes
## No 93 28
## Yes 4 4
##
## Accuracy : 0.7519
## 95% CI : (0.6682, 0.8237)
## No Information Rate : 0.7519
## P-Value [Acc > NIR] : 0.5473
##
## Kappa : 0.1119
##
## Mcnemar's Test P-Value : 4.785e-05
##
## Sensitivity : 0.9588
## Specificity : 0.1250
## Pos Pred Value : 0.7686
## Neg Pred Value : 0.5000
## Prevalence : 0.7519
## Detection Rate : 0.7209
## Detection Prevalence : 0.9380
## Balanced Accuracy : 0.5419
##
## 'Positive' Class : No
##
female_confusion_matrix
## Confusion Matrix and Statistics
##
## Reference
## Prediction No Yes
## No 4 2
## Yes 15 59
##
## Accuracy : 0.7875
## 95% CI : (0.6817, 0.8711)
## No Information Rate : 0.7625
## P-Value [Acc > NIR] : 0.354209
##
## Kappa : 0.2325
##
## Mcnemar's Test P-Value : 0.003609
##
## Sensitivity : 0.2105
## Specificity : 0.9672
## Pos Pred Value : 0.6667
## Neg Pred Value : 0.7973
## Prevalence : 0.2375
## Detection Rate : 0.0500
## Detection Prevalence : 0.0750
## Balanced Accuracy : 0.5889
##
## 'Positive' Class : No
##
We can see that indeed, at least within the testing groups, women did seem to survive at a higher proportion than men (24.8% to 76.3% in the testing set). Print a summary of your trained model and interpret one of the fitted coefficients in light of the above disparity.
summary(logit_model)
##
## Call:
## glm(formula = survived ~ pclass + sex + age + sibsp + parch,
## family = binomial, data = data_train)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 3.903165 0.409280 9.537 < 2e-16 ***
## pclassMiddle -1.291506 0.257421 -5.017 5.25e-07 ***
## pclassLower -2.404084 0.262022 -9.175 < 2e-16 ***
## sexmale -2.684206 0.200130 -13.412 < 2e-16 ***
## age -0.036776 0.007494 -4.907 9.24e-07 ***
## sibsp -0.395584 0.118587 -3.336 0.00085 ***
## parch 0.032494 0.111916 0.290 0.77155
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1121.27 on 833 degrees of freedom
## Residual deviance: 757.87 on 827 degrees of freedom
## AIC: 771.87
##
## Number of Fisher Scoring iterations: 5
The coefficient for the sexmale variable is -2.684206, which means that when all other variables are held constant, being male is associated with a decrease in the log odds of survival by approximately 2.684206 units compared to being female. This coefficient indicates that being male was a significant predictor of death on the Titanic and highlights the gender disparity in survival on the Titanic.
Now let’s see if our model is fair across this explanatory variable. Calculate five measures (as defined in class) in this question: the Overall accuracy rate ratio between females and males, the disparate impact between females and males, the statistical parity between females and males, and the predictive equality as well as equal opportunity between females and males (collectively these last two comprise equalized odds). Set a reasonable \(\epsilon\) each time and then comment on which (if any) of these five criteria are met.
male_predictions <- ifelse(male_predicted_probabilities > 0.5, 1, 0)
female_predictions <- ifelse(female_predicted_probabilities > 0.5, 1, 0)
male_true_labels <- ifelse(male_test_data$survived == "Yes", 1, 0)
female_true_labels <- ifelse(female_test_data$survived == "Yes", 1, 0)
# Overall accuracy ratio
overall_accuracy_ratio <- sum(female_predictions == female_true_labels) / sum(male_predictions == male_true_labels)
# Disparate impact
disparate_impact <- sum(female_predictions) / sum(male_predictions)
# Statistical parity
statistical_parity <- abs((sum(female_predictions) / sum(male_predictions)) - 1)
# False Positive Rate (FPR) for males and females
male_fpr <- sum(male_predictions == 1 & male_true_labels == 0) / sum(male_true_labels == 0)
female_fpr <- sum(female_predictions == 1 & female_true_labels == 0) / sum(female_true_labels == 0)
# Predictive equality
predictive_equality <- abs(female_fpr - male_fpr)
# Equal opportunity
male_fnr <- sum(male_predictions == 0 & male_true_labels == 1) / sum(male_true_labels == 1)
female_fnr <- sum(female_predictions == 0 & female_true_labels == 1) / sum(female_true_labels == 1)
equal_opportunity <- abs(female_fnr - male_fnr)
overall_accuracy_ratio
## [1] 0.6494845
disparate_impact
## [1] 9.25
statistical_parity
## [1] 8.25
predictive_equality
## [1] 0.7482366
equal_opportunity
## [1] 0.8422131
The model does not meet the criteria for fairness. The disparate impact, statistical parity, and overall accuracy rate ratio suggest potential disparities in predictions between females and males. The predictive equality and equal opportunity measures show relatively lower disparities in false positive and false negative rates between the two groups, but it’s not enough to reduce make the model fair..
It is always important for us to interpret our results in light of the original data and the context of the analysis. In this case, it is relevant that we are analyzing a historical event post-facto and any disparities across demographics identified are unlikely to be replicated. So even though our model fails numerous of the statistical fairness criteria, I would argue we need not worry that our model could be misused to perpetuate discrimination in the future. After all, this model is likely not being used to prescribe a preferred method of treatment in the future.
Even so, provide a philosophical notion of justice or fairness that may have motivated the Titanic survivors to act as they did. Spell out what this philosophical notion or principle entails?
Virtue ethics may have motivated survival outcomes on the Titanic because people believed it was a man’s duty to protect his wife and children which meant sacrificing his own life so they would survive. The virtues of duty and chivalry were pretty universal, especially at that time, so women and children were saved first.
Note that the echo = FALSE parameter was added to the
code chunk to prevent printing of the R code that generated the
plot.