Comparing Classification Methods to Predict the Development of Diabetes
Analyst: Raine Huebner Gloege
Date: December 5th, 2025
1. Background of the Data
This comprehensive analysis uses the Pima Indian Diabetes dataset from the “mlbench” R package. The dataset contains 8 features about patients including: the number of pregnancies, glucose levels, blood pressure, thickness of tricep skin fold, insulin, BMI, the diabetes pedigree function (measure genetic predisposition for developing diabetes), and age. We will use these features in combination with five different classification methods to build a model to predict the ninth feature in the dataset, whether a patient develops diabetes.
2. Research Questions
The goal of this analysis was to compare the performance of five different classification methods on predicting a diabetes diagnosis. The five classifiers compared where logistic regression, KNN, CART, Neural Network, and Random Forest.
In our comparison we aiming to answer the following questions:
Based on accuracy, recall, precision, and F1-score, which model has the best overall performance?
Which method results in the highest accuracy on the test set, is it the same as the method with the best overall performance?
Which classifier has the highest AUC?
3. Analysis Methods
3.1 Data Preprocessing
The data were split into test and training sets, with a 20% to 80% split, respectively. Both sets were then normalized. Each classifier was trained on the same normalized set and validated with the same normalized test set.
3.2 Determining the Optimal Logistic Classifier
To ensure the optimal logistic classifier was chosen to be compared with the other methods, ordinary logistic regression and penalized regression methods were explored.
3.3 Data Visualization
The results of this analysis were visualized using a bar plot to compare the accuracies of the different methods and an overlaid ROC plot of each classifier that includes AUC values. The colors used in the graphs where from the Okabe-Ito color palette, a colorblind-friendly palette that is widely recommended for use in scientific figures.
3.4 Packages Used
The R packages used in this analysis included: caret, gt, mlbench, neuralnet, pROC, rpart, randomForest, and tidyverse.
4. Results
4.1 Logistic Regression Model
Trying to fit an ordinary linear regression model using step wise selection resulted in separation, so we opted to use a penalized regression method. Ridge, lasso, and elastic net were compared using a re-sampling table to decide which method is best, based on accuracy.
##
## Call:
## summary.resamples(object = ., metric = "Accuracy")
##
## Models: ridge, lasso, elastic
## Number of resamples: 10
##
## Accuracy
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## ridge 0.7419355 0.7759577 0.7938508 0.8058468 0.8064516 0.9354839 0
## lasso 0.6562500 0.7560484 0.7938508 0.7902218 0.8344254 0.8709677 0
## elastic 0.6451613 0.7822581 0.8094758 0.7960685 0.8424899 0.8709677 0Ridge regression was selected as our logistic classifier because it had the highest mean, maximum, and minimum accuracy, in addition to being the most stable, with the smallest spread between the first and third quartile.
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 43 12
## 1 9 14
##
## Accuracy : 0.7308
## 95% CI : (0.6184, 0.825)
## No Information Rate : 0.6667
## P-Value [Acc > NIR] : 0.1393
##
## Kappa : 0.3762
##
## Mcnemar's Test P-Value : 0.6625
##
## Sensitivity : 0.5385
## Specificity : 0.8269
## Pos Pred Value : 0.6087
## Neg Pred Value : 0.7818
## Prevalence : 0.3333
## Detection Rate : 0.1795
## Detection Prevalence : 0.2949
## Balanced Accuracy : 0.6827
##
## 'Positive' Class : 1
## The confusion matrix of the ridge regression model has an accuracy of 73.08 percent but fails to demonstrate an accuracy that is statistically significant compared to the no information rate (denoted by P-Value [Acc > NIR]).
4.2 KNN Model
The KNN model was tuned using values 1 through 20 for k. It also has an accuracy of 73.08 percent but still doesn’t have an accuracy significantly higher than the no information rate. The KNN model also has worse balanced accuracy than the ridge regression model.
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 45 14
## 1 7 12
##
## Accuracy : 0.7308
## 95% CI : (0.6184, 0.825)
## No Information Rate : 0.6667
## P-Value [Acc > NIR] : 0.1393
##
## Kappa : 0.3505
##
## Mcnemar's Test P-Value : 0.1904
##
## Sensitivity : 0.4615
## Specificity : 0.8654
## Pos Pred Value : 0.6316
## Neg Pred Value : 0.7627
## Prevalence : 0.3333
## Detection Rate : 0.1538
## Detection Prevalence : 0.2436
## Balanced Accuracy : 0.6635
##
## 'Positive' Class : 1
## 4.3 CART Model
The CART (decision trees) model has the best accuracy of the methods tried so far (75.64%). Its accuracy is very close to being significantly higher than the no information rate, but isn’t quite there with a p-value of 0.05651. This model also has the best balanced accuracy observed so far.
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 41 8
## 1 11 18
##
## Accuracy : 0.7564
## 95% CI : (0.646, 0.8465)
## No Information Rate : 0.6667
## P-Value [Acc > NIR] : 0.05651
##
## Kappa : 0.4673
##
## Mcnemar's Test P-Value : 0.64636
##
## Sensitivity : 0.6923
## Specificity : 0.7885
## Pos Pred Value : 0.6207
## Neg Pred Value : 0.8367
## Prevalence : 0.3333
## Detection Rate : 0.2308
## Detection Prevalence : 0.3718
## Balanced Accuracy : 0.7404
##
## 'Positive' Class : 1
## 4.4 Neural Network Model
The neural network model was trained with 2 hidden layers with 10 and 5 nodes, respectively. This model has the worst accuracy of all the methods tried so far (69.23%). The lower end of its confidence interval (57.76%) for accuracy is much worse than the no information rate (66.67%).
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 40 12
## 1 12 14
##
## Accuracy : 0.6923
## 95% CI : (0.5776, 0.7919)
## No Information Rate : 0.6667
## P-Value [Acc > NIR] : 0.3638
##
## Kappa : 0.3077
##
## Mcnemar's Test P-Value : 1.0000
##
## Sensitivity : 0.5385
## Specificity : 0.7692
## Pos Pred Value : 0.5385
## Neg Pred Value : 0.7692
## Prevalence : 0.3333
## Detection Rate : 0.1795
## Detection Prevalence : 0.3333
## Balanced Accuracy : 0.6538
##
## 'Positive' Class : 1
## 4.5 Random Forest Model
The random forest model has the same accuracy as the ridge regression and KNN models. However, it has a better balanced accuracy than both the ridge and KNN models.
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 40 9
## 1 12 17
##
## Accuracy : 0.7308
## 95% CI : (0.6184, 0.825)
## No Information Rate : 0.6667
## P-Value [Acc > NIR] : 0.1393
##
## Kappa : 0.4112
##
## Mcnemar's Test P-Value : 0.6625
##
## Sensitivity : 0.6538
## Specificity : 0.7692
## Pos Pred Value : 0.5862
## Neg Pred Value : 0.8163
## Prevalence : 0.3333
## Detection Rate : 0.2179
## Detection Prevalence : 0.3718
## Balanced Accuracy : 0.7115
##
## 'Positive' Class : 1
## 4.6 Model Comparison
Table 1. Comparison of performance metrics across all five classifiers.
In comparing accuracy, recall, precision, and F1 score across all classifiers we can see that the CART model performs best on all metrics, except precision. Although the KNN model has the highest precision, its recall is much lower resulting in imbalance shown in the F1 score.
Figure 1. Bar plot of accuracies of all five classifiers.
The bar chart above shows the accuracies of all the classifiers in ascending order. As seen earlier, the CART model has the highest accuracy, whereas the neural network model has the worst accuracy. Ridge, KNN, and Random Forest all have the same accuracy, although they differ in other performance metrics as seen in Table 1 and the confusion matrices of the models on the test set.
Figure 2. Overlaid ROC curves of all five classifiers with AUC values included in legend.
The figure above depicts ROC curves for each classifier and their corresponding AUC values. All classifiers have a better than random performance AUC > 0.5. Surprisingly, the CART model has the second highest AUC, despite better performance according to Table 1. The ridge regression’s AUC of 0.834 suggests it is able to distinguish between classes best across all classification thresholds.
5. Conclusion
Overall, the CART model demonstrated an ability to predict diabetes in patients better than the other classification methods. However, all models had an accuracy that was not significantly higher than the no information rate, including the CART model, even if only by a slim margin. This could be because the classifiers may require more data to have better and more confident accuracies. After removing NA values form the Pima Indian Diabetes data set, there where only 392 observations remaining. This meant that an 80/20 training and test split resulted in a training set of 314 observations and a test set of 78 observations. Thus, the classifiers may be suffering from a test or training set that is too small to capture the structure of the data efficiently.
6. Appendices
6.1 Setup Code
knitr::opts_chunk$set(echo = TRUE)
library(caret)
library(gt)
library(mlbench)
library(neuralnet)
library(pROC)
library(rpart)
library(randomForest)
library(tidyverse)
# Make dataset available
data("PimaIndiansDiabetes2")
# Remove missing values
pima_data <- PimaIndiansDiabetes2 |> na.omit()
set.seed(123)
train_index <- createDataPartition(pima_data$diabetes, p = 0.8, list = FALSE)
# Split predictors and target
X_train <- pima_data[train_index, ] %>% select(-diabetes)
X_test <- pima_data[-train_index, ] %>% select(-diabetes)
y_train <- pima_data$diabetes[train_index]
y_test <- pima_data$diabetes[-train_index]
y_train <- ifelse(y_train == "pos", 1, 0)
y_test <- ifelse(y_test == "pos", 1, 0)
# Fit preprocess on training predictors only
pre_proc <- preProcess(X_train, method = c("center", "scale"))
# Apply preprocess transformation to both training and test sets
X_train_scaled <- predict(pre_proc, X_train)
X_test_scaled <- predict(pre_proc, X_test)6.2 Logistic Regression Model Section 4.1 Code
# Build model matrices
train_df <- cbind(X_train_scaled, diabetes = y_train)
test_df <- cbind(X_test_scaled, diabetes = y_test)
train_df$diabetes <- factor(train_df$diabetes, levels = c(0,1))
test_df$diabetes <- factor(test_df$diabetes, levels = c(0,1))
x_train <- model.matrix(diabetes ~ ., data = train_df)[, -1]
x_test <- model.matrix(diabetes ~ ., data = test_df)[, -1]
# Setup lambda grid
lambda_grid <- 10^seq(-3, 3, length = 100)
# Train ridge model
set.seed(234)
ridge <- train(
diabetes ~ .,
data = train_df,
method = "glmnet",
family = "binomial",
tuneGrid = expand.grid(alpha = 0, lambda = lambda_grid),
trControl = trainControl(method = "cv", number = 10)
)
# Make preditions on the test set
ridge_pred_prob <- predict(ridge, newdata = test_df, type = "prob")[,2]
ridge_pred_class <- factor(ifelse(ridge_pred_prob > 0.5, 1, 0), levels = c(0,1))
actual_class <- factor(y_test, levels = c(0,1))
# Train lasso model
set.seed(345)
lasso <- train(
diabetes ~ .,
data = train_df,
method = "glmnet",
family = "binomial",
tuneGrid = expand.grid(alpha = 1, lambda = lambda_grid),
trControl = trainControl(method = "cv", number = 10)
)
# Train elastic net
set.seed(456)
elastic <- train(
diabetes ~ .,
data = train_df,
method = "glmnet",
family = "binomial",
tuneLength = 10, # caret will choose alpha + lambda combinations
trControl = trainControl(method = "cv", number = 10)
)
# Create re-sampling table
models <- list(ridge = ridge, lasso = lasso, elastic = elastic)
resamples(models) %>% summary(metric = "Accuracy")
# Confusion matrix of predictions on test set
cm_ridge <- confusionMatrix(ridge_pred_class, actual_class, positive = "1")
cm_ridge6.3 KNN Model Section 4.2 Code
# Define values of k to test
k_values <- seq(from = 1, to = 20, by = 1)
# Train control with cross-validation
ctrl <- trainControl(method = "cv", number = 5, classProbs = FALSE, savePredictions = TRUE)
# Train KNN model
set.seed(567)
knn_model <- train(diabetes ~ ., data = train_df, method = "knn", trControl = ctrl, tuneGrid = expand.grid(k = k_values), metric = "Accuracy")
# Make predictions on test set
test_pred.knn <- predict(knn_model, newdata = test_df)
# Confusion matrix of predictions on test set
cm_knn<- confusionMatrix(test_pred.knn, test_df$diabetes, positive = "1")
cm_knn6.4 CART Model Section 4.3 Code
# Train CART model
cart_model <- rpart(diabetes ~ ., data = train_df, method = "class")
# Make predictions on test set
cart_pred <- predict(cart_model, newdata = test_df, type = "class")
# Confusion matrix of predictions on test set
cm_cart <- confusionMatrix(cart_pred, test_df$diabetes, positive = "1")
cm_cart6.5 Neural Network Model Section 4.4 Code
# Train Neural Network
set.seed(678)
neural_net_model <- neuralnet(
diabetes ~ .,
data = train_df,
hidden = c(10, 5),
rep = 1,
act.fct = "logistic",
linear.output = FALSE,
stepmax = 2000,
)
# Predict probabilities (two columns: prob_0, prob_1)
predicted_probs <- predict(neural_net_model, X_test_scaled)
# Extract probability of class 1
prob_class1 <- predicted_probs[, 2]
# Convert to class labels
predicted_class <- factor(ifelse(prob_class1 > 0.5, 1, 0), levels = c(0,1))
# Confusion matrix of predictions on test set
cm_nnet <- confusionMatrix(predicted_class, test_df$diabetes, positive = "1")
cm_nnet6.6 Random Forest Model Section 4.5 Code
# Train Random Forest Model
set.seed(789)
random_forest_model <- randomForest(diabetes ~ ., data = train_df, ntree = 100, mtry = 2)
predictions_rf <- predict(random_forest_model, newdata = test_df)
# Confusion matrix of predictions on test set
cm_rf <- confusionMatrix(predictions_rf, test_df$diabetes, positive = "1")
cm_rf6.7 Model Comparison Section 4.6 Code
# Define a helper function to extract metrics from confusion matrices
get_metrics <- function(cm) {
acc <- cm$overall["Accuracy"]
rec <- cm$byClass["Sensitivity"]
prec <- cm$byClass["Precision"]
f1 <- cm$byClass["F1"]
return(c(Accuracy = acc, Recall = rec, Precision = prec, F1_score = f1))
}
# Calculate metrics for each classfier
ridge_metrics <- get_metrics(cm_ridge)
cart_metrics <- get_metrics(cm_cart)
knn_metrics <- get_metrics(cm_knn)
nnet_metrics <- get_metrics(cm_nnet)
rf_metrics <- get_metrics(cm_rf)
# Put metrics into data frame
evaluation_df <- rbind(
Logistic_Ridge = ridge_metrics,
CART = cart_metrics,
KNN = knn_metrics,
Neural_Network = nnet_metrics,
Random_Forest = rf_metrics
) %>%
as.data.frame()
colnames(evaluation_df) <- c("Accuracy", "Recall", "Precision", "F1_score")
# Identify the max value for each metric
max_vals <- sapply(evaluation_df, max)
# Create comparison table
comp_table <- evaluation_df %>%
gt(rownames_to_stub = TRUE) %>%
# Format numbers
fmt_number(
columns = everything(),
decimals = 3
) %>%
# Bold column names
tab_style(
style = cell_text(weight = "bold"),
locations = cells_column_labels(everything())
) %>%
# Bold row names (model names)
tab_style(
style = cell_text(weight = "bold"),
locations = cells_stub(rows = everything())
) %>%
# Highlight & bold max Accuracy
data_color(
columns = Accuracy,
fn = function(column) ifelse(column == max(column), "#b6f5b6", "white")
) %>%
tab_style(
style = cell_text(weight = "bold"),
locations = cells_body(
columns = Accuracy,
rows = Accuracy == max(Accuracy)
)
) %>%
# Highlight & bold max Recall
data_color(
columns = Recall,
fn = function(column) ifelse(column == max(column), "#b6f5b6", "white")
) %>%
tab_style(
style = cell_text(weight = "bold"),
locations = cells_body(
columns = Recall,
rows = Recall == max(Recall)
)
) %>%
# Highlight & bold max Precision
data_color(
columns = Precision,
fn = function(column) ifelse(column == max(column), "#b6f5b6", "white")
) %>%
tab_style(
style = cell_text(weight = "bold"),
locations = cells_body(
columns = Precision,
rows = Precision == max(Precision)
)
) %>%
# Highlight & bold max F1 score
data_color(
columns = F1_score,
fn = function(column) ifelse(column == max(column), "#b6f5b6", "white")
) %>%
tab_style(
style = cell_text(weight = "bold"),
locations = cells_body(
columns = F1_score,
rows = F1_score == max(F1_score)
)
) %>%
# Add header
tab_header(
title = "Model Evaluation Metrics",
subtitle = "Comparison of Classifiers on the Pima Indians Diabetes Dataset"
)
# Save as PNG
gtsave(comp_table, "model_comparison_table.png", vwidth = 500, vheight = 300)
# Extract accuracy from confusion matrices
acc_ridge <- cm_ridge$overall["Accuracy"]
acc_cart <- cm_cart$overall["Accuracy"]
acc_knn <- cm_knn$overall["Accuracy"]
acc_nnet <- cm_nnet$overall["Accuracy"]
acc_rf <- cm_rf$overall["Accuracy"]
# Build accuracy comparison data frame
accuracy_df <- data.frame(
Model = c("Logistic Regression (Ridge)", "CART", "KNN", "Neural Network", "Random Forest"),
Accuracy = c(acc_ridge, acc_cart, acc_knn, acc_nnet, acc_rf)
)
accuracy_df$Model <- factor(
accuracy_df$Model,
levels = accuracy_df$Model[order(accuracy_df$Accuracy)]
)
# Define distinct, yet colorblind accessible colors (Okabe-Ito Color Palette for Accessibility)
model_colors <- c(
"Logistic Regression (Ridge)" = "#56B4E9",
"CART" = "#009E73",
"KNN" = "#E69F00",
"Neural Network" = "#F0E442",
"Random Forest" = "#0072B2"
)
# Create model comparison bar chart
ggplot(accuracy_df, aes(x = Model, y = Accuracy, fill = Model)) +
geom_col(width = 0.7) +
geom_text(aes(label = sprintf("%.3f", Accuracy)), vjust = -0.75, size = 4) +
scale_fill_manual(values = model_colors) +
scale_y_continuous(limits = c(0, 1)) +
labs(
title = "Classifier Performance Comparison on Pima Indian Diabetes Dataset",
x = "Model",
y = "Accuracy"
) +
theme_minimal(base_size = 14) +
theme(
legend.position = "none",
axis.text.x = element_text(angle = 25, hjust = 1),
plot.title = element_text(size = 13)
)
# Get probabilities for calculating ROC
cart_pred_prob <- predict(cart_model, newdata = test_df, type = "prob")[,2]
knn_pred_prob <- predict(knn_model, newdata = test_df, type = "prob")[,2]
rf_pred_prob <- predict(random_forest_model, newdata = test_df, type = "prob")[,2]
# Calculate ROC for each classifier
roc_ridge <- roc(actual_class, ridge_pred_prob)
roc_cart <- roc(actual_class, cart_pred_prob)
roc_knn <- roc(actual_class, knn_pred_prob)
roc_nnet <- roc(actual_class, prob_class1)
roc_rf <- roc(actual_class, rf_pred_prob)
# Create a list of each classifer for plotting
roc_list <- list(
"Logistic Regression (Ridge)" = roc_ridge,
"CART" = roc_cart,
"KNN" = roc_knn,
"Neural Network" = roc_nnet,
"Random Forest" = roc_rf
)
# Create AUC labels for plotting
auc_labels <- sapply(roc_list, auc)
names(auc_labels) <- paste0(names(roc_list), " (AUC=", round(auc_labels, 3), ")")
# Create overlaid ROC curve plot
ggroc(roc_list, linewidth = 1.2, aes = c("colour"), legacy.axes = TRUE) +
scale_color_manual(
values = model_colors,
labels = names(auc_labels)
) +
geom_segment(aes(x = 0, xend = 1, y = 0, yend = 1), color="darkgrey", linetype="dashed") +
labs(
title = "ROC Curve Comparison Across Classifiers on Pima Indian Diabetes Dataset",
x = "False Positive Rate (1 - Specificity)",
y = "True Positive Rate (Sensitivity)",
color = "Model (AUC)"
) +
theme_minimal() +
theme(
legend.position = "bottom",
plot.title = element_text(size = 12),
legend.box = "vertical"
) +
guides(color = guide_legend(nrow = 3, byrow = TRUE))