The objective of this third assignment is to analyze the Boston Housing data set using the SVM algorithm. In Assignment II, (which revised version is provided as an addendum to this assignment) I trained and analyzed two distinct decision tree models and a random forest model on the same data set. The present analysis will focus on applying SVM to compare its findings with those from the previous assignment, enriched by insights from relevant journal articles and my own perspective as someone working in the field of public safety. The Boston Housing data set used in both assignments is accessible in R through the MASS and mlbench packages, as well asonline .

Assignment III: Analysis of the Boston Housing data set using the SVM algorithm

Set up the environment: Load required libraries

Load the Boston Housing data

# Load the dataset
data(BostonHousing, package = "mlbench")

# Preview the data
head(BostonHousing)
##      crim zn indus chas   nox    rm  age    dis rad tax ptratio      b lstat
## 1 0.00632 18  2.31    0 0.538 6.575 65.2 4.0900   1 296    15.3 396.90  4.98
## 2 0.02731  0  7.07    0 0.469 6.421 78.9 4.9671   2 242    17.8 396.90  9.14
## 3 0.02729  0  7.07    0 0.469 7.185 61.1 4.9671   2 242    17.8 392.83  4.03
## 4 0.03237  0  2.18    0 0.458 6.998 45.8 6.0622   3 222    18.7 394.63  2.94
## 5 0.06905  0  2.18    0 0.458 7.147 54.2 6.0622   3 222    18.7 396.90  5.33
## 6 0.02985  0  2.18    0 0.458 6.430 58.7 6.0622   3 222    18.7 394.12  5.21
##   medv
## 1 24.0
## 2 21.6
## 3 34.7
## 4 33.4
## 5 36.2
## 6 28.7

Data Splitting

# Set seed for reproducibility
set.seed(25)

# Split the data set (70% training, 30% testing)
train_idx <- createDataPartition(BostonHousing$medv, p = 0.7, list = FALSE)
train_data <- BostonHousing[train_idx, ]
test_data <- BostonHousing[-train_idx, ]

Preprocessing and Feature Scaling

SVMs are sensitive to the scale of the features, so it’s essential to preprocess the data by centering and scaling the features. I will use caret’s preProcess function.

# Pre - processing: Centering and scaling
# Check for missing values in the training data
sum(is.na(train_data))  # Check for missing values
## [1] 0
# Impute missing values using knnImpute and scale features
preprocess_params <- preProcess(train_data[, -ncol(train_data)], method = c("center", "scale", "knnImpute"))
train_data_scaled <- predict(preprocess_params, train_data)

# Apply the same preprocessing to the test data
test_data_scaled <- predict(preprocess_params, test_data)

# Check the data types and convert to numeric if necessary
sapply(train_data_scaled[, -ncol(train_data_scaled)], class)
##      crim        zn     indus      chas       nox        rm       age       dis 
## "numeric" "numeric" "numeric"  "factor" "numeric" "numeric" "numeric" "numeric" 
##       rad       tax   ptratio         b     lstat 
## "numeric" "numeric" "numeric" "numeric" "numeric"
train_data_scaled[] <- lapply(train_data_scaled, as.numeric)  # Ensure all columns are numeric

# Convert 'chas' to numeric in both train and test datasets
train_data_scaled$chas <- as.numeric(train_data_scaled$chas) - 1  # Convert factor to numeric (0 and 1 for Yes/No)
test_data_scaled$chas <- as.numeric(test_data_scaled$chas) - 1

# Verify the changes
sapply(train_data_scaled, class)  # Check the class of columns in the train set
##      crim        zn     indus      chas       nox        rm       age       dis 
## "numeric" "numeric" "numeric" "numeric" "numeric" "numeric" "numeric" "numeric" 
##       rad       tax   ptratio         b     lstat      medv 
## "numeric" "numeric" "numeric" "numeric" "numeric" "numeric"
sapply(test_data_scaled, class)   # Check the class of columns in the test set
##      crim        zn     indus      chas       nox        rm       age       dis 
## "numeric" "numeric" "numeric" "numeric" "numeric" "numeric" "numeric" "numeric" 
##       rad       tax   ptratio         b     lstat      medv 
## "numeric" "numeric" "numeric" "numeric" "numeric" "numeric"
# Check for missing values
sum(is.na(train_data_scaled))
## [1] 0
sum(is.na(test_data_scaled))
## [1] 0
# Optionally, impute or remove missing values
train_data_scaled <- na.omit(train_data_scaled)
test_data_scaled <- na.omit(test_data_scaled)

Tuning the SVM Model

Now, I will define the tuning grid for SVM and prepare the training and test data.

# Prepare training data
x_train <- train_data_scaled[, -ncol(train_data_scaled)]  # Excluding target variable
y_train <- train_data_scaled$medv  # Target variable

# Define hyperparameter grid for SVM
svm_tune_grid <- expand.grid(
  C = c(0.1, 1, 10),
  sigma = c(0.05, 0.1, 0.5)
)
svm_tune_grid 
##      C sigma
## 1  0.1  0.05
## 2  1.0  0.05
## 3 10.0  0.05
## 4  0.1  0.10
## 5  1.0  0.10
## 6 10.0  0.10
## 7  0.1  0.50
## 8  1.0  0.50
## 9 10.0  0.50

Model Training with Cross-Validation

I will train the SVM model using 10-fold cross-validation.

# Inspect resampling results
svm_model$results
##      C sigma     RMSE  Rsquared      MAE   RMSESD RsquaredSD     MAESD
## 1  0.1  0.05 5.454954 0.7060641 3.298712 1.571617  0.1538727 0.6268930
## 4  1.0  0.05 4.064550 0.7990762 2.432428 1.503616  0.1323317 0.5459065
## 7 10.0  0.05 3.600356 0.8324552 2.212215 1.605980  0.1234792 0.5561868
## 2  0.1  0.10 5.798039 0.6763649 3.448526 1.581269  0.1369369 0.6711215
## 5  1.0  0.10 4.179611 0.7935399 2.441804 1.466413  0.1268217 0.5080439
## 8 10.0  0.10 3.821718 0.8147058 2.353427 1.548253  0.1251301 0.6223799
## 3  0.1  0.50 7.504872 0.4837591 4.736004 1.589874  0.1419899 0.7587745
## 6  1.0  0.50 5.384242 0.6802821 3.165722 1.505935  0.1349455 0.7010626
## 9 10.0  0.50 4.981986 0.7055873 3.121265 1.466014  0.1415976 0.7419203

Prediction on Test Data:

# Prepare test data for prediction
x_test <- test_data_scaled[, -ncol(test_data_scaled)]  # Excluding target variable
y_test <- test_data_scaled$medv  # True target values

# Make predictions
y_pred <- predict(svm_model, x_test)

y_pred
##   [1] 31.036653 31.779608 15.535656 18.631727 17.937642 18.200395 13.808811
##   [8] 15.167201 21.311864 21.142253 22.455817 22.369693 22.659189 20.909077
##  [15] 19.652605 21.620457 33.088592 29.434326 20.475860 18.475579 17.117429
##  [22] 26.237222 22.106462 25.050734 22.688515 21.029360 23.706222 26.770453
##  [29] 28.613400 23.741221 43.114874 34.709126 24.514177 19.159355 19.694064
##  [36] 18.685706 20.449740 21.337509 17.483977 18.820600 20.329160 20.787260
##  [43] 17.545490 19.344977 15.148719 15.252697 16.343218 17.809292 24.077477
##  [50] 12.862837 15.856164 25.571825 30.773691 43.656989 20.008808 45.815917
##  [57] 17.728726 43.660687 30.092501 21.939482 44.164439 19.556964 23.297672
##  [64] 25.382580 25.105533 27.540755 25.870331 41.596634 42.860240 30.021068
##  [71] 28.141728 23.128364 25.840250 25.664930 21.317557 16.491642 20.428362
##  [78] 29.342721 41.415479 21.381290 51.179872 25.533140 31.924219 26.640623
##  [85] 44.686339 32.498567 21.452155 23.866377 33.054304 26.252445 20.843068
##  [92] 23.059658 27.768082 34.895220 24.965191 20.454625 19.602671 20.294935
##  [99] 21.805562 18.742235 21.217456 23.820380 19.436988 19.751583 19.470343
## [106] 24.286173 28.314813 20.988413 18.384635 23.846661 24.335062 48.038942
## [113] 11.880940 19.240190  9.275025 14.664321 15.415606  9.178585 11.353057
## [120]  6.768795  8.850718 19.940419 13.336939 14.851273 18.942603 14.021179
## [127] 11.370853 12.629059  9.343995  8.448676 12.204939 14.546611 15.329917
## [134] 11.148758 13.978464 15.132102 16.199509 15.395522 18.318705 14.124134
## [141] 16.159785 18.814243 25.331377 24.517931 23.063246 10.196145 20.453155
## [148] 22.394021 17.691064 19.032405
# Visualize the results
plot(y_pred, y_test, main = "Predicted vs Actual", xlab = "Predicted", ylab = "Actual")
abline(0, 1, col = "red")  # Add reference line y = x

Evaluate Model Results

Model Evaluation on the Test Data:

# Calculate the residuals
residuals <- y_test - y_pred

# RMSE (Root Mean Squared Error)
rmse <- sqrt(mean(residuals^2))
cat("RMSE:", rmse, "\n")
## RMSE: 2.895167
# R-squared (R²)
ss_total <- sum((y_test - mean(y_test))^2)
ss_residual <- sum(residuals^2)
rsq <- 1 - (ss_residual / ss_total)
cat("R-squared:", rsq, "\n")
## R-squared: 0.9017955
# MAE (Mean Absolute Error)
mae <- mean(abs(residuals))
cat("MAE:", mae, "\n")
## MAE: 2.057908

Model Evaluation Using Caret’s postResample Function:

# Use postResample function for evaluation
metrics <- postResample(pred = y_pred, obs = y_test)
cat("Performance Metrics:\n")
## Performance Metrics:
print(metrics)
##      RMSE  Rsquared       MAE 
## 2.8951668 0.9042938 2.0579076

The RMSE of 2.895 indicates that, on average, the model’s predictions deviate from the actual values by approximately 2.895 units. An R-squared value of 0.9042 indicates that about 90.18% of the variability in the response variable is explained by the model. This suggests a very strong fit, meaning the SVM model is capturing the patterns in the data well. The MAE of 2.058 suggests that the average absolute difference between the model’s predictions and the actual values is about 2.058 units. Compared to RMSE, MAE is less sensitive to outliers.

Visualize model performance

library(kernlab)
library(ggplot2)

# Ensure there are no NAs or infinite values in the training and test sets
train_data_clean <- na.omit(train_data_scaled)  # Remove NAs from training data
test_data_clean <- na.omit(test_data_scaled)  # Remove NAs from test data

# Train the SVM model again, adjusting hyperparameters if necessary
svm_model <- ksvm(medv ~ ., data = train_data_clean, kernel = "rbfdot", C = 1, sigma = 0.1)

# Verify the model summary
summary(svm_model)
## Length  Class   Mode 
##      1   ksvm     S4
# Predict on the cleaned test set
y_pred <- predict(svm_model, newdata = test_data_clean)

# Check for NAs in the predicted values
if (any(is.na(y_pred))) {
  stop("Predictions returned NA values.")
}

# Check if predicted and actual values have the same length
if (length(y_pred) != length(test_data_clean$medv)) {
  stop("Predicted and actual values have different lengths.")
}

# Visualize predicted vs actual values
plot(y_pred, test_data_clean$medv, main = "Predicted vs Actual Values",
     xlab = "Predicted", ylab = "Actual", col = "gray", pch = 16)

# Add reference line y = x (diagonal line)
abline(0, 1, col = "red")

Benchmarking: Comparing SVM metrics against decision trees and random forests metrics (displayed in Assignment II, below) to identify the best-performing method:

# Given RMSE value
rmse_svm <- 2.895167

# Calculate MSE from RMSE
mse_svm <- rmse_svm^2

# Print the result
print(mse_svm)
## [1] 8.381992
# Define the model names, MSE values, and performance descriptions
model_names <- c("SVM", "Random Forest", "Decision Tree 1", "Decision Tree 2")
mse_values <- c(8.3821, 11.4742, 19.3019, 23.1812)
performance_description <- c(
  "Best performance: lowest error, highest prediction accuracy.",
  "Second best: slightly higher error but still quite accurate.",
  "Moderate performance: higher error than SVM and Random Forest.",
  "Worst performance: highest error, indicating poor accuracy."
)

# Create a data frame
performance_table <- data.frame(
  Model = model_names,
  MSE = mse_values,
  `Relative Performance` = performance_description
)

# Print the table
print(performance_table)
##             Model     MSE
## 1             SVM  8.3821
## 2   Random Forest 11.4742
## 3 Decision Tree 1 19.3019
## 4 Decision Tree 2 23.1812
##                                             Relative.Performance
## 1   Best performance: lowest error, highest prediction accuracy.
## 2   Second best: slightly higher error but still quite accurate.
## 3 Moderate performance: higher error than SVM and Random Forest.
## 4    Worst performance: highest error, indicating poor accuracy.

SVM vs. Random Forest: The SVM model outperforms the Random Forest model with a significantly lower MSE (8.3821 vs. 11.4742). Both models are strong performers, but the SVM demonstrates better accuracy and less variance in predictions. SVM vs. Decision Trees: The Decision Tree models have much higher MSE values (19.3019 and 23.1812), suggesting less accurate and more variable predictions. This aligns with the expectation that single decision trees are often less powerful than ensemble models like Random Forest or sophisticated models like SVM. Random Forest vs. Decision Trees: The Random Forest significantly outperforms both decision trees, highlighting the advantage of ensemble methods in reducing variance and improving predictive performance.

Overall,the SVM model demonstrates the best predictive accuracy among the models based on MSE. This suggests that it captures the underlying patterns in the data more effectively than both Random Forest and Decision Trees. Random Forest performs well but is slightly outperformed by SVM, while the Decision Trees show the weakest performance.If computational resources and model complexity are not concerns, SVM is the preferred model based on these metrics. However, if simplicity or interpretability is prioritized, Random Forest could be a strong alternative.

Discussion and Comparative Evaluation

In this study, I explored the application of machine learning models, specifically Support Vector Machines (SVM), Random Forest (RF), and simple decision trees, to predict housing prices in the Boston Housing dataset. Although the primary context of the analysis was housing prices, the implications and methods used are highly relevant to fields like public safety and emergency management, where predictive modeling plays a critical role.

The results of my analysis indicate that SVM outperformed both Random Forest and decision trees in terms of predictive accuracy. This finding aligns with trends observed in several academic studies, where SVM and RF have been evaluated in different contexts and applications. The ability of machine learning algorithms to model complex relationships and predict outcomes accurately is essential in public safety and disaster management scenarios, such as predicting emergency response times or the spread of natural disasters.

Methods Used in My Analysis

For my analysis of the Boston Housing dataset, the goal was to predict the continuous target variable of housing prices. However, the methods tested here, including SVM, RF, and decision trees, offer valuable insights that could be applied in disaster management and public safety contexts:

Support Vector Machine (SVM): SVM is a robust machine learning algorithm known for its ability to handle high-dimensional spaces and model complex, non-linear relationships through kernel functions. In my analysis, SVM showed superior accuracy compared to both Random Forest and decision trees. This aligns with findings in disaster management studies, where SVM has been applied to predict complex outcomes, such as the severity of incidents or the effectiveness of emergency responses. SVM’s ability to manage complex, multi-dimensional datasets could prove highly beneficial in environments with diverse and intricate data, such as those found in public safety data. Random Forest (RF): Random Forest, an ensemble learning method, constructs multiple decision trees to improve prediction accuracy. In the context of disaster management, RF could be particularly useful in handling datasets with high variance, noise, or missing data—conditions often encountered in real-time emergency scenarios. Although RF performed well, SVM was more accurate, suggesting that while Random Forest is a powerful tool, it might not always outperform SVM in situations where data is high-dimensional or non-linearly separable. Decision Trees: While decision trees are simple and interpretable, their performance was not as strong as the ensemble methods. This is particularly relevant in public safety applications, where model interpretability is crucial for decision-making in high-stakes environments. Decision trees might be useful for providing quick, understandable insights, but they may lack the predictive power needed when complex, multi-dimensional data is at play. The performance metrics used to evaluate the models included Mean Squared Error (MSE), which is commonly used for regression tasks to assess prediction accuracy. The same approach can be adopted in public safety for evaluating the reliability of predictive models, such as forecasting emergency response needs or resource allocation in disaster situations.

Discussion of the Results in Light of Academic Literature

The superior performance of SVM in my analysis is consistent with several studies that have applied machine learning models to complex datasets in public safety and emergency management. For instance, SVM has been used to predict the spread of wildfires or assess the severity of natural disasters, where data often involves high-dimensional features and non-linear relationships. My results corroborate these findings, suggesting that SVM’s ability to handle high-dimensional spaces is beneficial in such predictive tasks.

On the other hand, Random Forest has been applied successfully in various domains, including predicting the severity of emergency situations and assessing risks in disaster management. The study by Ahmad et al. (2019) highlights RF’s ability to manage noisy or imbalanced data, a common characteristic of public safety data. However, my analysis suggests that RF’s performance might not always match the precision of SVM, especially when dealing with high-dimensional datasets where the relationships between features are complex.

This finding echoes the results of Mohammed et al. (2021), which showed that while RF might outperform SVM in some cases (e.g., predicting shear strength in concrete beams), SVM is competitive and shows strong results in other regression tasks. This highlights the fact that the performance of SVM and RF may vary based on the task at hand, just as it would in emergency management scenarios where data characteristics change depending on the incident type.

Which Algorithm is Recommended for More Accurate Results?

Based on my analysis, SVM is generally recommended for tasks involving high-dimensional, non-linear regression data, such as predicting the severity of a disaster or the time to respond to emergencies. SVM’s strength lies in its ability to model complex relationships, making it well-suited for scenarios where the data does not follow simple linear patterns. However, RF may still be a better choice in environments with large, noisy datasets or when interpretability is key, as it provides more straightforward explanations for decision-making.

Is SVM Better for Classification or Regression Scenarios?

While SVM is commonly associated with classification tasks, its variant, Support Vector Regression (SVR), is highly effective for regression problems. In my analysis, SVM was applied to a regression task (predicting housing prices), where it demonstrated superior performance. This highlights SVM’s versatility, as it can be successfully applied to both classification and regression tasks—an attribute that is valuable in disaster management contexts, where different types of outcomes need to be predicted, ranging from the likelihood of an event to continuous variables like response time.

Do I Agree with the Recommendations?

I agree with the recommendations from the academic literature regarding the choice of algorithm. For high-dimensional, non-linear regression tasks, SVM tends to outperform other models due to its ability to effectively handle complex relationships between features. This is particularly relevant in disaster management and public safety applications, where predicting events with multiple influencing factors often requires such a robust modeling approach.Reliable SVM predictions can greatly enhance preparedness and response strategies for effective public safety and disaster management. While Random Forest remains a strong option, especially when handling noisy data or when interpretability is important, SVM’s ability to provide more accurate and precise predictions makes it a valuable tool in situations where human lives are at stake.

In conclusion, while Random Forest and Decision Trees are valuable in many machine learning applications, SVM proved to be the most accurate for predicting housing prices in my analysis. This result aligns with academic literature and further confirms the potential of SVM in complex, high-dimensional tasks. The applicability of these findings to public safety and disaster management demonstrates the broad utility of machine learning models in forecasting and decision-making, particularly in environments where data is multifaceted and outcomes are critical.

Revised Assignment II (for review):

Analysis of the Boston Housing data set using Decision Trees and Random Forest Models

Load the data and define utility functions

# Set seed for reproducibility
set.seed(25)

# Load the data set

data(BostonHousing, package = "mlbench")
# Define a reusable function for data splitting
split_data <- function(data, response, test_size = 0.3) {
  train_idx <- createDataPartition(data[[response]], p = 1 - test_size, list = FALSE)
  train_data <- data[train_idx, ]
  test_data <- data[-train_idx, ]
  return(list(train = train_data, test = test_data))
}
set.seed(25)
# Define the reusable function for training and evaluating models
train_and_evaluate <- function(model_func, train_data, test_data, response, ...) {
  # Train the model
  formula <- as.formula(paste(response, "~ ."))
  model <- model_func(formula, data = train_data, ...)
  
  # Make predictions
  predictions <- predict(model, test_data)
  
  # Compute MSE
  mse <- mean((predictions - test_data[[response]])^2)
  
  return(list(model = model, mse = mse))
}

# Run the function with a linear regression model
result <- train_and_evaluate(
  model_func = lm, 
  train_data = train_data, 
  test_data = test_data, 
  response = "medv"
)

# Access the MSE from the result object
mse <- result$mse
print(mse)
## [1] 22.12246
set.seed(25)
# Define a reusable function for training and evaluating models
train_and_evaluate <- function(model_func, train_data, test_data, response, ...) {
  # Train the model
  formula <- as.formula(paste(response, "~ ."))
  model <- model_func(formula, data = train_data, ...)
  # Make predictions
  predictions <- predict(model, test_data)
# Compute MSE
  mse <- mean((predictions - test_data[[response]])^2)
  return(list(model = model, mse = mse))
}
mse
## [1] 22.12246
set.seed(25)

# Define a reusable function for training and evaluating models
train_and_evaluate <- function(model_func, train_data, test_data, response, ...) {
  # Train the model
  formula <- as.formula(paste(response, "~ ."))
  model <- model_func(formula, data = train_data, ...)
  
  # Make predictions
  predictions <- predict(model, test_data)
  
  # Compute MSE
  mse <- mean((predictions - test_data[[response]])^2)
  
  return(list(model = model, mse = mse))
}

Split Data for Training and Testing

# Split the dataset
split <- split_data(BostonHousing, "medv", test_size = 0.3)
train_data <- split$train
test_data <- split$test

Train Decision Tree Models

set.seed(25)
# Decision Tree Using rm as Primary Predictor
# Train and evaluate decision tree model 1 (primary predictor: rm)
dt1_result <- train_and_evaluate(
  rpart,
  train_data,
  test_data,
  "medv",
  control = rpart.control(cp = 0.01)
)
dt1_result
## $model
## n= 356 
## 
## node), split, n, deviance, yval
##       * denotes terminal node
## 
##  1) root 356 29907.8500 22.46461  
##    2) lstat>=9.725 205  4917.7990 17.20732  
##      4) crim>=6.91188 62   757.1021 12.21129 *
##      5) crim< 6.91188 143  1942.1990 19.37343  
##       10) lstat>=16.04 49   602.8412 16.65510 *
##       11) lstat< 16.04 94   788.5414 20.79043 *
##    3) lstat< 9.725 151 11631.7700 29.60199  
##      6) rm< 7.433 131  5778.5270 27.37481  
##       12) dis>=1.9704 122  3257.3460 26.37541  
##         24) rm< 6.594 70   727.7377 23.09429 *
##         25) rm>=6.594 52   761.5369 30.79231 *
##       13) dis< 1.9704 9   747.5356 40.92222 *
##      7) rm>=7.433 20   947.2380 44.19000  
##       14) ptratio>=16.65 8   462.0588 38.73750 *
##       15) ptratio< 16.65 12    88.7825 47.82500 *
## 
## $mse
## [1] 23.18119
set.seed(25)
# Decision Tree Using Categorized age Variable

# Create a categorized version of 'age' for Decision Tree 2
train_data$age_cat <- cut(train_data$age, breaks = c(0, 25, 50, 75, 100), 
                          labels = c("0-25", "26-50", "51-75", "76-100"))
test_data$age_cat <- cut(test_data$age, breaks = c(0, 25, 50, 75, 100), 
                         labels = c("0-25", "26-50", "51-75", "76-100"))

# Train and evaluate decision tree model 2
dt2_result <- train_and_evaluate(
  rpart,
  train_data,
  test_data,
  "medv",
  control = rpart.control(cp = 0.01, maxdepth = 4)
)
dt2_result
## $model
## n= 356 
## 
## node), split, n, deviance, yval
##       * denotes terminal node
## 
##  1) root 356 29907.8500 22.46461  
##    2) lstat>=9.725 205  4917.7990 17.20732  
##      4) crim>=6.91188 62   757.1021 12.21129 *
##      5) crim< 6.91188 143  1942.1990 19.37343  
##       10) lstat>=16.04 49   602.8412 16.65510 *
##       11) lstat< 16.04 94   788.5414 20.79043 *
##    3) lstat< 9.725 151 11631.7700 29.60199  
##      6) rm< 7.433 131  5778.5270 27.37481  
##       12) dis>=1.9704 122  3257.3460 26.37541  
##         24) rm< 6.594 70   727.7377 23.09429 *
##         25) rm>=6.594 52   761.5369 30.79231 *
##       13) dis< 1.9704 9   747.5356 40.92222 *
##      7) rm>=7.433 20   947.2380 44.19000  
##       14) ptratio>=16.65 8   462.0588 38.73750 *
##       15) ptratio< 16.65 12    88.7825 47.82500 *
## 
## $mse
## [1] 23.18119

Train A Random Forest Model

set.seed(25)
# Train and evaluate random forest model
rf_result <- train_and_evaluate(
  randomForest,
  train_data,
  test_data,
  "medv",
  ntree = 500,       # Number of trees in the forest
  mtry = 3           # Number of variables tried at each split
)
rf_result
## $model
## 
## Call:
##  randomForest(formula = formula, data = train_data, ntree = 500,      mtry = 3) 
##                Type of random forest: regression
##                      Number of trees: 500
## No. of variables tried at each split: 3
## 
##           Mean of squared residuals: 12.76518
##                     % Var explained: 84.81
## 
## $mse
## [1] 11.47418

Summarize Results

# Consolidate results in a summary table
results_table <- data.frame(
  Model = c("Decision Tree 1", "Decision Tree 2", "Random Forest"),
  MSE = c(dt1_result$mse, dt2_result$mse, rf_result$mse)
)

# Print the summary table
print(results_table)
##             Model      MSE
## 1 Decision Tree 1 23.18119
## 2 Decision Tree 2 23.18119
## 3   Random Forest 11.47418

Visualize Models

Decision trees:

# Visualize the decision trees side by side
par(mfrow = c(1, 2)) # Arrange plots in a single row
rpart.plot(dt1_result$model, main = "Decision Tree 1: Primary Predictor 'rm'")
rpart.plot(dt2_result$model, main = "Decision Tree 2: Primary Predictor 'age_cat'")

Random Forest Variable Importance:

# Random Forest Variable Importance
varImpPlot(rf_result$model, main = "Random Forest Variable Importance")

# Extract variable importance and convert to a data frame
importance_df <- as.data.frame(randomForest::importance(rf_result$model))
importance_df$Variable <- rownames(importance_df)
rownames(importance_df) <- NULL

# Arrange by importance and select top 10 variables
importance_df <- importance_df |> 
  dplyr::arrange(desc(IncNodePurity)) |> 
  dplyr::slice(1:10)  # Select top 10 variables

# Plot using ggplot2
library(ggplot2)

ggplot(importance_df, aes(x = reorder(Variable, IncNodePurity), y = IncNodePurity)) +
  geom_bar(stat = "identity", fill = "steelblue", color = "black") +
  coord_flip() +  # Flip axes for better readability
  labs(
    title = "Top 10 Important Variables in Random Forest",
    x = "Variable",
    y = "Increase in Node Purity"
  ) +
  theme_minimal() +
  theme(
    text = element_text(size = 12),
    axis.text.x = element_text(angle = 45, hjust = 1)
  )

IV. Findings & Discussion:

Decision trees are widely recognized for their strengths and weaknesses. Their advantages lie in their intuitive design, which mirrors human thought processes, making them easy to use for decision-making. By breaking down complex issues into smaller, logical branches, they enhance clarity, understanding, and documentation. Their flexibility allows for continuous optimization, resulting in strategic and effective solutions that save time and provide actionable insights. However, decision trees also have notable limitations. They can become overly complex and difficult to maintain, especially for topics influenced by subjective factors. Their large, repetitive diagrams can reduce usability, and their reliance on familiarity with specific elements may alienate some users. Simplifying tree creation, offering clean interfaces, and supporting collaboration are crucial to overcoming these challenges. Conventional methods further complicate usability and accessibility, as static diagrams and coding skills are often required. Modern solutions should focus on mobile compatibility, intuitive editing, integration with other systems, and features like multimedia attachments and metrics to keep decision trees dynamic, interactive, and user-friendly.

In this analysis, I explored the relationship between various features of the Boston Housing dataset and the median value of homes (medv) using decision trees and a random forest regression models. The primary goal was to assess how different predictors influence model performance and compare the results obtained from these models, paying particular attention to bias and variance. The first decision tree model was constructed using rm (the average number of rooms per dwelling) as the primary predictor. The output indicated that the first split occurred at a threshold of 6.941 rooms, leading to two distinct branches of the tree. The deviance values indicated that the model successfully segmented the dataset, with predictions of the median home values reflecting significant variance depending on the rooms available. The nodes further split based on lstat (percentage of lower status of the population) and nox (nitric oxides concentration), highlighting how this predictor interacted with others to refine predictions. The final predictions varied significantly at terminal nodes, suggesting a complex relationship where lower median values were predicted for homes with fewer rooms and higher levels of lstat.

In contrast, when constructing the second decision tree model, I included age (the proportion of owner-occupied units built prior to 1940) as the primary predictor. This model, like the previous one, began with rm but took a different path after the initial split. The results illustrated how changing the primary predictor influenced the tree’s structure and the median values predicted. The nodes reflected how lstat and rm interacted differently, leading to alternative outcomes at terminal nodes. This change in features significantly affected the model’s bias and variance. The bias, which represents the error due to overly simplistic assumptions in the learning algorithm, appeared reduced when utilizing predictors that capture the underlying data variability more effectively. Conversely, variance increased as the model became more sensitive to fluctuations in the training dataset. The random forest model, leveraging an ensemble of decision trees, produced more reliable predictions than the individual trees. The output demonstrated a mean of squared residuals of 10.97856, with an impressive 86.68% of variance explained. This model benefitted from lower bias due to its ensemble nature while also managing variance through averaging multiple decision trees, reducing the risk of overfitting present in single tree models.

The performance improvement observed in the random forest model highlights the advantage of combining multiple trees, which can enhance model robustness and reduce variance significantly. In contrast, individual decision trees showed greater susceptibility to overfitting, especially when influenced by specific features like rm, age, or lstat. In addressing concerns related to changes and version control of decision trees, I would implement tools that facilitate continuous improvement. Version control systems, such as Git, can help track changes in model architecture and features over time, ensuring that the decision trees do not lose their effectiveness. Additionally, I would establish a standardized approach for creating and evaluating decision trees, utilizing

established performance metrics and validation techniques to maintain model integrity. To manage complexity, employing automated pipelines for data preprocessing, model training, and evaluation would streamline the process, ensuring consistency and repeatability. Regular audits of model performance against new data, alongside retraining protocols when necessary, would further enhance reliability. These strategies will help ensure that the decision tree models remain effective over time, adapting to new data and maintaining accuracy in predictions.

12345

  1. Sheykhmousa, M., Mahdianpari, M., Ghanbari, H., Mohammadimanesh, F., Ghamisi, P., & Homayouni, S. (2020). Support vector machine versus random forest for remote sensing image classification: A meta-analysis and systematic review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, 6308–6325. https://doi.org/10.1109/JSTARS.2020.2997795↩︎

  2. Lamas Piñeiro, J., & Wong Portillo, L. (2022). Web architecture for URL-based phishing detection based on Random Forest, Classification Trees, and Support Vector Machine. Inteligencia Artificial, 25(69), 107–121. https://doi.org/10.4114/intartif.2022.6963↩︎

  3. Comparative Analysis of Supervised Classification Algorithms for Residential Water End Uses Heydari, Zahra ; Stillwell, Ashlynn S. Water resources research, 2024-06, Vol.60 (6)↩︎

  4. Kamwa, Innocent et al (2012). On the Accuracy Versus Transparency Trade-Off of Data-Mining Models for Fast-Response PMU-Based Catastrophe Predictors↩︎

  5. A comparative study of forecasting corporate credit ratings using neural networks, support vector machines, and decision trees Golbayani, Parisa ; Florescu, Ionuţ ; Chatterjee, Rupak. The North American journal of economics and finance, 2020-11, Vol.54, p.101251, Article 101251↩︎