Research Question: Which model—SVM, Random Forest, kNN, or LDA—most accurately classifies breast tumors as malignant or benign based on diagnostic features?
Author
Affiliation
Clyde Huang
Middlebury College
1. Introduction
Breast cancer is one of the most common cancers affecting millions of people worldwide, with early detection being crucial for improving survival rates. Analyzing features of cell nuclei, such as texture, radius, and smoothness, provides valuable insights into distinguishing between cancerous and non-cancerous breast masses.
In this project, I explored how statistical learning can help predict breast cancer based on features of cell nuclei. I developed four models, Support Vector Machines (SVM), Random Forest, k-Nearest Neighbors (kNN), and Linear Discriminant Analysis (LDA), with the goal of identifying the most accurate and reliable model for breast cancer prediction.
my research aims to aid early cancer diagnosis, support medical decision-making, and broaden public knowledge on malignant breast cancer characteristics.
2. Preliminary Work
2.1 Data Cleaning & Missing Value Check
Code
# load librariesrm(list=ls())library(tidyverse)library(caret)library(ggplot2)library(mice)library(janitor)library(cowplot)library(kableExtra)library(readr)archive_1_ <-read_csv("~/Documents/Statistical Learning/archive (1).zip")View(archive_1_)# read datacancer <- archive_1_cancer_clean <- cancer |># remove unnecessary columnselect(-...33) |># clean column namesclean_names()# factor the diagnosis columncancer_clean$diagnosis <-factor(cancer_clean$diagnosis)# missing value checkmd.pattern(cancer_clean)
/\ /\
{ `---' }
{ O O }
==> V <== No need for mice. This data set is completely observed.
\ \|/ /
`-----'
This dataset has no missing values, so no imputation is needed.
2.2 Variable Introduction
Features of breast mass are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.
Ten real-valued features are computed for each cell nucleus:
Radius: How far the edge of the cell is from the center on average.
Texture: How uneven the cell’s surface appears. A rougher surface means a higher texture value.
Perimeter: The total distance around the edge of the cell.
Area: How much space the cell takes up.
Smoothness: How smooth or bumpy the cell’s edges are. Smoother edges have lower values, while jagged edges have higher values.
Compactness: How closely packed the cell is. If the cell is more stretched or irregular, its compactness value will be higher.
Concavity: How deeply the edges of the cell curve inward. A more indented edge means a higher concavity value.
Concave Points: The number of inward curves (dents) along the cell’s edge. More dents mean a higher value.
Symmetry: How evenly balanced the cell looks. If both sides of the cell look similar, the symmetry value is higher.
Fractal Dimension: How complicated or detailed the cell’s edge is. A more jagged or intricate edge has a higher fractal dimension value.
For each of the feature above, statistical measurements, including mean, standard deviation, and mean of the “worst” (largest) values, were calculated to give a more complete picture of the cell nuclei. This resulted in a total of 30 features.
For example:
Mean Radius tells how large the cell nuclei are on average
Standard Error of Radius shows how much the cell sizes vary
Worst Radius points out the mean value of the three largest cell sizes found for each observation
2.3 Variable Distributions
All variables in this data set, except Diagnosis (malignant/benign), are continuous variables. Here are their distributions, based on the three statistical measurements of each feature.
Code
ggplot(cancer_clean, aes(x = diagnosis, fill = diagnosis)) +geom_bar() +scale_fill_manual(values =c("B"="skyblue", "M"="salmon")) +labs(title ="Distribution of Breast Cancer Diagnosis",x ="Diagnosis (B = Benign, M = Malignant)",y ="Count") +theme_minimal()
The Benign group, represented in blue, is significantly larger with over 350 cases, indicating a higher frequency of non-cancerous diagnoses. In contrast, the Malignant group, shown in red, contains approximately 200 cases. The imbalance in the dataset, with a higher number of benign (B) cases compared to malignant (M) cases, emphasizes the importance of careful handling in predictive modeling to prevent bias toward the majority class.
Code
mean_vars <-grep("_mean$", names(cancer_clean), value =TRUE)# function to create individual density plotscreate_plot <-function(data, variable) {ggplot(data, aes_string(x = variable)) +geom_density(colour ="salmon", fill ="salmon", alpha =0.3) +labs(title = variable, x = variable, y ="Density") +theme_minimal(base_size =8) +theme(plot.title =element_text(size =9, hjust =0.5))}# function to plot groups of variablesplot_group <-function(var_list, data, group_name) { plots <-lapply(var_list, function(var) {create_plot(data, var) }) combined_plot <-plot_grid(plotlist = plots, ncol =3) title <-ggdraw() +draw_label(group_name, fontface ='bold', size =14, hjust =0.5)plot_grid(title, combined_plot, ncol =1, rel_heights =c(0.1, 1))}# plot Mean Variablesmean_plot <-plot_group(mean_vars, cancer_clean, "Variable Distributions (Mean)")# display the plotsprint(mean_plot)
Key observations
Skewed Distributions: Many features (e.g., area mean, concavity mean) exhibit skewness, which may influence model performance (e.g. LDA)
Feature Importance: Variables like area_mean, concavity_mean, and concave_points_mean are right-skewed, which might hold critical information for distinguishing malignant and benign cases, as higher values in these features are often linked to irregular cell growth.
Code
se_vars <-grep("_se$", names(cancer_clean), value =TRUE)# function to create individual density plotscreate_plot <-function(data, variable) {ggplot(data, aes_string(x = variable)) +geom_density(colour ="skyblue", fill ="skyblue", alpha =0.3) +labs(title = variable, x = variable, y ="Density") +theme_minimal(base_size =8) +theme(plot.title =element_text(size =9, hjust =0.5))}# create density plots for SE Variablesse_plot <-plot_group(se_vars, cancer_clean, "Variable Distributions (Standard Deviation)")# display the plotsprint(se_plot)
Key observation
The right-skewed nature of the standard deviation features reflects that variations in cell nuclei characteristics are generally low for most samples, with only a small subset exhibiting higher variability.
Code
worst_vars <-grep("_worst$", names(cancer_clean), value =TRUE)# function to create individual density plotscreate_plot <-function(data, variable) {ggplot(data, aes_string(x = variable)) +geom_density(colour ="lightgreen", fill ="lightgreen", alpha =0.3) +labs(title = variable, x = variable, y ="Density") +theme_minimal(base_size =8) +theme(plot.title =element_text(size =9, hjust =0.5))}# create density plots for Worst Variablesworst_plot <-plot_group(worst_vars, cancer_clean, "Variable Distribution ('Worst')")# display the plotsprint(worst_plot)
Key observation
Compared to the Variable Distributions (Mean), the distributions of the worst values (mean of the three largest measurements per observation) exhibit greater right-skewness. This increased skewness may suggest more pronounced differences between malignant and benign breast cells, potentially highlighting key features that distinguish abnormal cell growth.
2.4 Exploratory Visualization: Tumor Class Separation
To identify potential classification boundaries and guide model selection, we could plot key diagnostic features, colored by diagnosis, to visualize the separation between malignant and benign tumors. To reduce dimensionality, I use principal components (PCs), which capture key patterns in the data.
Code
# PCApca_data <- cancer_clean %>%select(-id, -diagnosis)pca_result <-prcomp(pca_data, scale. =TRUE)# add the PC1, PC2 and diagnosis to a new dataframepca_df <-as.data.frame(pca_result$x[, 1:2]) %>%mutate(diagnosis = cancer_clean$diagnosis)# rename columnscolnames(pca_df) <-c("PC1", "PC2", "diagnosis")# plot PC1 vs. PC2ggplot(pca_df, aes(x = PC1, y = PC2, color = diagnosis)) +geom_point(size =1, alpha =0.8) +scale_color_manual(values =c("B"="skyblue", "M"="salmon")) +labs(title ="PCA Scatter Plot: Breast Tumor Diagnosis",x ="Principal Component 1 (PC1)",y ="Principal Component 2 (PC2)",color ="Diagnosis" ) +theme_minimal()
Key observation
A clear, likely linear separation between benign and malignant tumors is visible, suggesting that linear classification models such as Support Vector Machine might be effective for this project.
3. Model Selection
3.1 Linear Support Vector Machine
3.1.1 Model Intro
Support Vector Machine (SVM) is a supervised learning algorithm used for classification tasks. It tries to find the best possible boundary to separate different classes of data. For my breast cancer dataset, the SVM will learn to distinguish between Benign (B) and Malignant (M) cases based on features like the size, shape, and texture of cell nuclei.
3.1.2 Model Mechanism
When I use a Linear SVM, the algorithm assumes that the classes (Benign and Malignant) can be separated using a straight line (in two dimensions) or a flat plane (in higher dimensions). The SVM identifies the boundary that:
The distance between the boundary (hyperplane) and the closest data points (support vectors) is maximized.
It tries to correctly classify as many data points as possible.
In my data set, the features like radius_mean, concavity_mean, and smoothness_worst provide numerical values describing the shape and smoothness of cell nuclei.The algorithm will combine these features linearly to create the best possible boundary that separates Benign (B) and Malignant (M) cases.
A linear model is a great starting point for my project because it is fast and easy to interpret. It also helps us establish a baseline performance before exploring more complex models.
3.1.3 Linear SVM Modeling Steps:
Training the Model: the algorithm learned the boundary using the training data set while being validated using cross-validation for reliable results.
Data Pre-processing: the numerical features were centered and scaled so that all features have equal importance, preventing features with larger ranges (e.g., area_mean) from dominating the model.
Tuning Parameter C: the C parameter controls how flexible the boundary is. A small C allows some misclassifications to maximize the margin, while a large C prioritizes classifying every point correctly. I tested values of C from 1 to 10 to find the best-performing model.
After training, the model achieved high accuracy (97.54%), correctly identifies 98.31% of benign tumors (sensitivity = 98.31%) and 96.23% of malignant tumors (specificity = 96.23%). These metrics demonstrate that the model effectively separates benign and malignant cases, though further evaluation is needed on unseen data.
3.2 Radial Basis Function (RBF) Kernel Special Vector Machine
3.2.1 Model Intro
Support Vector Machine with a Radial Basis Function (RBF) is a supervised learning algorithm that can capture non-linear relationships in the data. While a Linear SVM separates the data using a straight boundary, the Radial SVM can create curved boundaries to better classify Benign (B) and Malignant (M) breast cancer cases.
3.2.2 Model Mechanism
Radial SVM uses a kernel trick to map the data into a higher-dimensional space where it becomes easier to find a boundary that separates the two classes. In the context of breast cancer prediction, the original features, such as radius_mean, may not perfectly separate benign and malignant cases with a straight line. The RBF kernel helps the SVM find a curved boundary that better captures the relationships between these features, improving classification performance.
3.2.3 Radial SVM Modeling Steps
Data Pre-processing: All numerical features were centered and scaled to ensure consistent influence on the model.
Model Training and Tuning: I trained the Radial SVM using cross-validation and tuned the C and sigma parameters using a grid search.
The model achieved an accuracy of 96.13%, with balanced sensitivity (96.07%) and specificity (96.23%), which illustrates poorer performance compared to linear SVM, which might be explained by Radial SVM’s sensitivity to the C and gamma parameters.
3.3 k-Nearest Neighbors
3.3.1 Model Intro
K-Nearest Neighbors (kNN) is a straightforward and effective non-parametric classification method that predicts the class of a data point based on the majority vote of its nearest neighbors. It assumes that similar cases are close to each other in the feature space. kNN is particularly useful for its simplicity and effectiveness in scenarios where the relationships between data points can be captured through their proximity. In this project, kNN will be utilized to classify breast tumors as benign or malignant by analyzing their diagnostic features and comparing them with known classifications in the dataset.
3.3.2 kNN Model Mechanism
The k-Nearest Neighbors algorithm operates by identifying the k closest training examples to a new data point and predicting its class based on the most common class among these neighbors. The proximity between examples is calculated using a distance metric, typically Euclidean distance. For breast cancer classification, kNN looks at the k nearest instances of a given tumor based on diagnostic features such as radius, texture, and perimeter, and classifies it as either benign or malignant depending on the predominant category of those neighbors. This method is effective for capturing complex, non-linear relationships without needing to define a specific model form, making it highly adaptable to varied data distributions.
Model Performance Metrics: k-Nearest Neighbors Model
Value
Accuracy
0.9665493
Sensitivity
0.9831461
Specificity
0.9386792
Kappa
0.9280226
3.3.3 Performance Metrics
The kNN model achieved an accuracy of 96.65%, which, while commendable, is not as high as the Linear SVM’s accuracy of 97.54%. The model’s sensitivity at 98.31% is on par with the Linear SVM, which effectively identifies benign tumors. However, its specificity at 93.87% is lower than that of both the Linear SVM (96.23%) and Radial SVM (96.23%), indicating a relative weakness in correctly classifying malignant tumors. The Kappa statistic of 0.9280, though indicating excellent agreement, further shows that kNN’s overall performance is slightly inferior to that of the SVM models, particularly in distinguishing malignant cases more accurately. This suggests that while kNN is a good classifier for benign tumors, it may require adjustments.
3.4 Linear Discriminant Analysis
3.4.1 LDA Model Introduction
Linear Discriminant Analysis (LDA) is used for classification and dimensionality reduction. It seeks to find a linear combination of features that best separates two or more classes of objects or events. In this project, LDA will be employed to classify breast tumors as benign or malignant. By projecting the diagnostic features onto a lower-dimensional space that maximizes the separation between the two classes, LDA not only simplifies the classification task but also enhances interpretability and computational efficiency.
3.4.2 LDA Model Mechanism
LDA functions by calculating linear decision boundaries between classes based on the mean and variance of each feature within each class. The algorithm focuses on maximizing the ratio of between-class variance to within-class variance in any particular data set, ensuring that the classes are as distinct as possible. For breast cancer classification, LDA analyzes the diagnostic features such as radius, texture, and area to find a projection that best separates benign from malignant tumors. This is achieved by deriving a linear combination of features that has the best ability to differentiate between the two classes. The resultant decision boundary is linear, making LDA particularly effective when the underlying data distribution approximates normality and the classes are linearly separable.
Model Performance Metrics: Linear Discriminant Analysis
Value
Accuracy
0.9612676
Sensitivity
0.9971910
Specificity
0.9009434
Kappa
0.9156041
3.4.3 Performance Metrics
The Linear Discriminant Analysis (LDA) model achieved an accuracy of 96.13%, placing its performance on par with the Radial SVM but below the Linear SVM. The model demonstrates exceptionally high sensitivity at 99.72%, showing us an excellent capability to correctly identify benign tumors with nearly perfect precision. However, its specificity at 90.09% is the lowest among the models discussed, including the k-Nearest Neighbors and both SVM models. This lower specificity reflects LDA’s relative difficulty in accurately classifying malignant tumors compared to benign ones. The Kappa statistic of 0.9156, while indicating substantial agreement, shows the model’s challenges in achieving balanced accuracy across both classes. This means that while LDA is highly effective in identifying benign cases, it may need refinement to improve its detection of malignant cases, especially when aiming for a balanced performance in medical diagnostic scenarios.
3.5 Random Forest
3.5.1 Random Forest Model Introduction
Random Forest is a learning method that operates by constructing a multitude of decision trees at training time and outputting the class that is the majority vote of the individual trees. This model is well-suited for classification tasks like distinguishing between benign and malignant breast tumors due to its robustness to overfitting and its ability to handle high-dimensional data effectively. Random Forest not only offers insights into feature importance but also improves prediction accuracy through its ensemble approach. By aggregating the results of various decision trees, it reduces the risk of errors from any single tree, making it highly reliable for complex classification problems where both accuracy and interpretability are important.
3.5.2 The Random Forest model mechanism
rf model combines multiple decision trees to improve classification accuracy and control overfitting. Each tree in the forest is constructed using a different random sample of the data and a random subset of features at each decision point, which ensures diversity in the model’s predictions. During the training phase, the algorithm builds numerous trees and integrates their outcomes to decide on the final classification.
The Random Forest model demonstrated a high accuracy of 97.01%, surpassing the performance of the k-Nearest Neighbors, Linear Discriminant Analysis, and matching closely with the Linear SVM. It shows excellent sensitivity at 98.31%, effectively identifying nearly all benign tumors correctly. The specificity at 94.81% is also robust, indicating a strong ability to correctly classify malignant tumors. The Kappa statistic of 0.9357, which measures inter-rater agreement for qualitative items, suggests a very high level of consistency in the model’s predictions beyond what would be expected by chance alone. Despite these strong results, the SVM still holds a slight edge, maintaining the best overall performance among all the models tested, particularly in terms of achieving the highest accuracy and balanced sensitivity and specificity. This makes the Linear SVM the best model for breast cancer classification in this study, combining high predictive accuracy with effective discrimination between benign and malignant tumors.
