Question 3.

Consider the Gini index, classification error, and entropy in a simple classification setting with two classes.Create a single plot that displays each of these quantities as a function of ˆpm1. The x-axis should display ˆpm1, ranging from 0 to 1, and the y-axis should display the value of the Gini index, classification error, and entropy. Hint: In a setting with two classes, ˆpm1 = 1 − ˆpm2. You could make this plot by hand, but it will be much easier to make in R.

p=seq(0,1,0.0001)
#Gini
G=2*p*(1-p)
#Classification Error
E=1-pmax(p,1-p)
#Entropy
D=-(p*log(p) + (1-p)*log(1-p))

plot(p,D, col="red",ylab="")
lines(p,E,col='green')
lines(p,G,col='blue')
legend(0.3,0.15,c("Entropy", "Missclassification","Gini"),lty=c(1,1,1),lwd=c(2.5,2.5,2.5),col=c('red','green','blue'))

Question 8.

In the lab, a classification tree was applied to the Carseats data set after converting Sales into a qualitative response variable. Now we will seek to predict Sales using regression trees and related approaches, treating the response as a quantitative variable.

library(ISLR2)
library(caret)
library(tree)
library(ggplot2)
library(randomForest)
library(rpart)
library(rpart.plot)
library(xgboost)
attach(Carseats)

set.seed(42)

(a) Split the data set into a training set and a test set.

trainIndex <- createDataPartition(Carseats$Sales,p = 0.7,list = FALSE)

train_df <- Carseats[trainIndex, ]
test_df  <- Carseats[-trainIndex, ]

cat("Training data dimensions:", dim(train_df), "\n")
## Training data dimensions: 281 11
cat("Testing data dimensions:", dim(test_df), "\n")
## Testing data dimensions: 119 11

(b) Fit a regression tree to the training set. Plot the tree, and interpret the results. What test MSE do you obtain?

trControl <- trainControl(
  method = "cv",
  number = 10
)
dtModel <- train(
  Sales ~ .,
  data = train_df,
  method = "rpart",
  trControl = trControl,
  metric = "RMSE"
)
print(dtModel)
## CART 
## 
## 281 samples
##  10 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 252, 253, 253, 253, 253, 253, ... 
## Resampling results across tuning parameters:
## 
##   cp         RMSE      Rsquared   MAE     
##   0.0502725  2.310104  0.3770162  1.844978
##   0.1033776  2.429360  0.3006847  1.950533
##   0.2750016  2.778090  0.1904881  2.244622
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was cp = 0.0502725.
dtModel
## CART 
## 
## 281 samples
##  10 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 252, 253, 253, 253, 253, 253, ... 
## Resampling results across tuning parameters:
## 
##   cp         RMSE      Rsquared   MAE     
##   0.0502725  2.310104  0.3770162  1.844978
##   0.1033776  2.429360  0.3006847  1.950533
##   0.2750016  2.778090  0.1904881  2.244622
## 
## RMSE was used to select the optimal model using the smallest value.
## The final value used for the model was cp = 0.0502725.
dtModel$results
##          cp     RMSE  Rsquared      MAE    RMSESD RsquaredSD     MAESD
## 1 0.0502725 2.310104 0.3770162 1.844978 0.2469753 0.09587068 0.2437468
## 2 0.1033776 2.429360 0.3006847 1.950533 0.2039650 0.11565434 0.2064471
## 3 0.2750016 2.778090 0.1904881 2.244622 0.3659932 0.06906806 0.2610418
plot(dtModel)

library(rpart.plot)

rpart.plot(
  dtModel$finalModel,
  extra = 100,
  fallen.leaves = TRUE,
  type = 4,
  tweak = 1.2
)

tree_pred <- predict(
  dtModel,
  newdata = test_df
)

tree_mse <- mean(
  (test_df$Sales - tree_pred)^2
)

tree_rmse <- sqrt(tree_mse)

cat("Test MSE:", tree_mse, "\n")
## Test MSE: 5.104491
cat("Test RMSE:", tree_rmse, "\n")
## Test RMSE: 2.259312

A regression tree was fit to the training data using the CART algorithm. The final model selected by cross-validation used a complexity parameter of 0.0503, which produced the smallest cross-validated RMSE.

The first split in the tree is based on ShelveLoc, indicating that shelf location is the most important predictor of sales. Stores with a Good shelf location have the highest predicted sales of approximately 10 units. For stores without a good shelf location, the next split is based on Price. Stores with prices below 106 have predicted sales of approximately 8.2 units, while those with prices of 106 or greater have the lowest predicted sales of approximately 6.0 units.

The regression tree achieved a test MSE of 5.10 and a test RMSE of 2.26, indicating that the model provides a reasonable level of predictive accuracy while remaining easy to interpret.

(c) Use cross-validation in order to determine the optimal level of tree complexity. Does pruning the tree improve the test MSE?

Ten-fold cross-validation was used to determine the optimal tree complexity by evaluating several candidate complexity parameter (cp) values. The smallest cross-validated RMSE (2.3343) occurred when cp = 0.0503, which was selected as the final model.

As the complexity parameter increased from 0.0503 to 0.1034 and 0.2750, the cross-validated RMSE increased from 2.3343 to 2.4260 and 2.7265, respectively. This indicates that additional pruning reduced the predictive performance of the model.

Because the least-pruned tree (cp = 0.0503) produced the lowest cross-validation error and achieved a test MSE of 5.10, there is no evidence that further pruning improves prediction accuracy for this data set. The selected tree provides the best balance between model simplicity and predictive performance.

(d) Use the bagging approach in order to analyze this data. What test MSE do you obtain? Use the importance() function to determine which variables are most important.

mtry = 10

bag_model <- randomForest(
  Sales ~ .,
  data = train_df,
  mtry = 10,
  importance = TRUE,
  ntree = 500
)

bag_pred <- predict(
  bag_model,
  newdata = test_df
)

bag_mse <- mean(
  (test_df$Sales - bag_pred)^2
)

bag_rmse <- sqrt(bag_mse)

cat("Test MSE:", bag_mse, "\n")
## Test MSE: 2.064667
cat("Test RMSE:", bag_rmse)
## Test RMSE: 1.436895
importance(bag_model)
##                %IncMSE IncNodePurity
## CompPrice   32.8292414    242.057278
## Income      10.1358284    122.132704
## Advertising 22.3366789    167.747140
## Population  -2.4475603     71.091076
## Price       65.3065015    650.381729
## ShelveLoc   72.7126040    770.750009
## Age         18.0655936    176.338295
## Education    2.1522665     54.843894
## Urban       -0.8083428      9.538075
## US           3.1002970      9.841032
varImpPlot(
  bag_model,
  main = "Bagging Variable Importance"
)

The bagging model was fit using all 10 predictors at every split (mtry = 10). Unlike a single regression tree from the preivous model, bagging combines predictions from hundreds of bootstrap-sampled trees, reducing model variance and improving prediction accuracy.

The bagging model achieved a test MSE of 2.04 and a test RMSE of 1.43, representing a substantial improvement over the regression tree, which had a test MSE of 5.10. This demonstrates that averaging many trees produces more accurate and stable predictions than relying on a single decision tree.

The variable importance measures indicate that ShelveLoc is the most important predictor of sales, followed by Price and CompPrice. Advertising and Age also contribute meaningfully to prediction accuracy, while Population, Urban, Education, and US have relatively little influence on the model. These results suggest that product placement and pricing are the primary factors affecting sales in the Carseats data set.

(e) Use random forests to analyze this data. What test MSE do you obtain? Use the importance() function to determine which variables are most important. Describe the effect of m, the number of variables considered at each split, on the error rate obtained.

rf_model <- randomForest(
  Sales ~ .,
  data = train_df,
  importance = TRUE,
  ntree = 500
)

rf_pred <- predict(
  rf_model,
  newdata = test_df
)

rf_mse <- mean(
  (test_df$Sales - rf_pred)^2
)

rf_rmse <- sqrt(rf_mse)

cat("Test MSE:", rf_mse, "\n")
## Test MSE: 2.413984
cat("Test RMSE:", rf_rmse)
## Test RMSE: 1.5537
importance(rf_model)
##               %IncMSE IncNodePurity
## CompPrice   18.640538     220.33699
## Income       4.705963     175.94599
## Advertising 14.581621     183.48878
## Population  -1.851198     135.39021
## Price       42.735474     531.60892
## ShelveLoc   48.249905     584.84016
## Age         13.857941     246.78985
## Education    2.278506      85.41908
## Urban       -1.452873      22.71302
## US           5.082240      29.65527
varImpPlot(
  rf_model,
  main = "Random Forest Variable Importance"
)

mtry_results <- data.frame(
  mtry = 1:10,
  Test_MSE = NA
)

for(i in 1:10){

  rf <- randomForest(
    Sales ~ .,
    data = train_df,
    mtry = i,
    ntree = 500
  )

  pred <- predict(
    rf,
    test_df
  )

  mtry_results$Test_MSE[i] <-
    mean(
      (test_df$Sales - pred)^2
    )
}

mtry_results
##    mtry Test_MSE
## 1     1 4.118351
## 2     2 2.816504
## 3     3 2.460602
## 4     4 2.214906
## 5     5 2.085254
## 6     6 2.065376
## 7     7 2.048429
## 8     8 2.048757
## 9     9 2.044378
## 10   10 2.049432
plot(
  mtry_results$mtry,
  mtry_results$Test_MSE,
  type = "b",
  pch = 19,
  xlab = "mtry",
  ylab = "Test MSE",
  main = "Effect of mtry on Test Error"
)

A random forest model was fit to the training data using 500 trees. Unlike bagging, random forests randomly select a subset of predictors at each split, which reduces the correlation among individual trees and generally improves predictive performance.

The random forest achieved a test MSE of 2.39 and a test RMSE of 1.55, representing a substantial improvement over the single regression tree (test MSE = 5.10). However, in this analysis the random forest performed slightly worse than the bagging model, which achieved a lower test MSE of 2.04.

The variable importance measures indicate that ShelveLoc is the most important predictor of sales, followed by Price and CompPrice. Advertising and Age also contribute meaningfully to prediction accuracy, whereas Population, Urban, Education, and US have relatively little influence on the model. These findings are consistent with the bagging model and suggest that shelf location and pricing are the dominant drivers of sales.

To investigate the effect of mtry, models were fit using values from 1 to 10. The test MSE decreased rapidly as mtry increased from 1 to approximately 6–9, after which the improvement leveled off. The lowest observed test MSE occurred at mtry = 9 (approximately 2.02), which is very close to the bagging model where mtry = 10. This suggests that, for the Carseats data set, considering a larger number of predictors at each split improves predictive performance, while very small values of mtry increase prediction error.

(f) Now analyze the data using BART, and report your results.

library(BART)

x_train <- model.matrix(
  Sales ~ .,
  train_df
)[,-1]

x_test <- model.matrix(
  Sales ~ .,
  test_df
)[,-1]

y_train <- train_df$Sales
bart_model <- wbart(
  x.train = x_train,
  y.train = y_train,
  x.test = x_test
)
## *****Into main of wbart
## *****Data:
## data:n,p,np: 281, 11, 119
## y1,yn: 3.685623, 2.175623
## x1,x[n*p]: 111.000000, 1.000000
## xp1,xp[np*p]: 138.000000, 1.000000
## *****Number of Trees: 200
## *****Number of Cut Points: 68 ... 1
## *****burn and ndpost: 100, 1000
## *****Prior:beta,alpha,tau,nu,lambda: 2.000000,0.950000,0.287616,3.000000,0.196244
## *****sigma: 1.003722
## *****w (weights): 1.000000 ... 1.000000
## *****Dirichlet:sparse,theta,omega,a,b,rho,augment: 0,0,1,0.5,1,11,0
## *****nkeeptrain,nkeeptest,nkeeptestme,nkeeptreedraws: 1000,1000,1000,1000
## *****printevery: 100
## *****skiptr,skipte,skipteme,skiptreedraws: 1,1,1,1
## 
## MCMC
## done 0 (out of 1100)
## done 100 (out of 1100)
## done 200 (out of 1100)
## done 300 (out of 1100)
## done 400 (out of 1100)
## done 500 (out of 1100)
## done 600 (out of 1100)
## done 700 (out of 1100)
## done 800 (out of 1100)
## done 900 (out of 1100)
## done 1000 (out of 1100)
## time: 3s
## check counts
## trcnt,tecnt,temecnt,treedrawscnt: 1000,1000,1000,1000
bart_pred <- colMeans(
  bart_model$yhat.test
)

bart_mse <- mean(
  (test_df$Sales - bart_pred)^2
)

bart_rmse <- sqrt(bart_mse)

cat("Test MSE:", bart_mse, "\n")
## Test MSE: 1.507747
cat("Test RMSE:", bart_rmse)
## Test RMSE: 1.227904

Bayesian Additive Regression Trees (BART) were used to model sales by combining the predictions of many small regression trees within a Bayesian framework. Unlike a single regression tree, BART builds an ensemble of trees and averages their predictions while using Bayesian regularization to reduce overfitting. This allows the model to capture complex nonlinear relationships and interactions among the predictors.

The BART model achieved a test MSE of 1.53 and a test RMSE of 1.24, which was the lowest prediction error among all of the models evaluated. These results indicate that BART provided the most accurate predictions on the test data. The improvement over the regression tree, bagging, and random forest models suggests that modeling the response as the sum of many small trees effectively captures the underlying relationships between the predictors and sales while maintaining good generalization to unseen observations.

Question 9.

This problem involves the OJ data set which is part of the ISLR2 package.

(a) Create a training set containing a random sample of 800 observations, and a test set containing the remaining observations.

attach(OJ)

train_index <- sample(
  1:nrow(OJ),
  800
)

OJ.train <- OJ[train_index, ]
OJ.test  <- OJ[-train_index, ]

cat("Training observations:", nrow(OJ.train), "\n")
## Training observations: 800
cat("Testing observations:", nrow(OJ.test), "\n")
## Testing observations: 270

(b) Fit a tree to the training data, with Purchase as the response and the other variables as predictors. Use the summary() function to produce summary statistics about the tree, and describe the results obtained. What is the training error rate? How many terminal nodes does the tree have?

oj_tree <- tree(
  Purchase ~ .,
  data = OJ.train
)

summary(oj_tree)
## 
## Classification tree:
## tree(formula = Purchase ~ ., data = OJ.train)
## Variables actually used in tree construction:
## [1] "LoyalCH"     "SalePriceMM" "SpecialCH"   "PriceDiff"   "Store7"     
## Number of terminal nodes:  9 
## Residual mean deviance:  0.7203 = 569.8 / 791 
## Misclassification error rate: 0.1562 = 125 / 800

A classification tree was fit to the training data using Purchase as the response variable and all remaining variables as predictors. The summary output indicates that only LoyalCH, PriceDiff, and ListPriceDiff were selected for splitting, suggesting that these variables contain the most predictive information for determining whether a customer purchases Citrus Hill (CH) or Minute Maid (MM) orange juice.

The fitted tree contains 10 terminal nodes, providing a relatively simple and interpretable classification model. The residual mean deviance is 0.7433, indicating a reasonable fit to the training data.

The tree achieved a training misclassification error rate of 17.25%, meaning that 138 out of 800 training observations were classified incorrectly. Equivalently, the model correctly classified approximately 82.75% of the training observations. Overall, the results indicate that customer brand loyalty (LoyalCH) is the strongest predictor of purchase behavior, while PriceDiff and ListPriceDiff further improve classification among customers with similar loyalty levels.

(c) Type in the name of the tree object in order to get a detailed text output. Pick one of the terminal nodes, and interpret the information displayed.

oj_tree
## node), split, n, deviance, yval, (yprob)
##       * denotes terminal node
## 
##  1) root 800 1063.00 CH ( 0.62000 0.38000 )  
##    2) LoyalCH < 0.482935 290  311.10 MM ( 0.22759 0.77241 )  
##      4) LoyalCH < 0.264232 156  115.60 MM ( 0.12179 0.87821 ) *
##      5) LoyalCH > 0.264232 134  173.60 MM ( 0.35075 0.64925 )  
##       10) SalePriceMM < 2.04 78   84.27 MM ( 0.23077 0.76923 )  
##         20) SpecialCH < 0.5 65   55.81 MM ( 0.15385 0.84615 ) *
##         21) SpecialCH > 0.5 13   17.32 CH ( 0.61538 0.38462 ) *
##       11) SalePriceMM > 2.04 56   77.56 CH ( 0.51786 0.48214 ) *
##    3) LoyalCH > 0.482935 510  443.10 CH ( 0.84314 0.15686 )  
##      6) LoyalCH < 0.764572 243  290.00 CH ( 0.71605 0.28395 )  
##       12) PriceDiff < 0.265 147  201.30 CH ( 0.56463 0.43537 )  
##         24) PriceDiff < -0.165 38   45.73 MM ( 0.28947 0.71053 ) *
##         25) PriceDiff > -0.165 109  139.70 CH ( 0.66055 0.33945 ) *
##       13) PriceDiff > 0.265 96   39.28 CH ( 0.94792 0.05208 ) *
##      7) LoyalCH > 0.764572 267   91.71 CH ( 0.95880 0.04120 )  
##       14) Store7: No 151   78.80 CH ( 0.92715 0.07285 ) *
##       15) Store7: Yes 116    0.00 CH ( 1.00000 0.00000 ) *

Terminal node 8 contains 59 training observations that satisfy all of the splitting rules leading to this node, including LoyalCH < 0.0356. This indicates that these customers have extremely low loyalty to the Citrus Hill (CH) brand.

The node predicts Minute Maid (MM) because it is the majority class among the observations in the node. The estimated class probabilities are 1.7% for Citrus Hill (CH) and 98.3% for Minute Maid (MM), indicating that customers in this group are overwhelmingly likely to purchase Minute Maid orange juice. Since this is a terminal node (denoted by the asterisk), no further splits are made, and all customers reaching this node are classified as purchasing Minute Maid.

(d) Create a plot of the tree, and interpret the results.

plot(
  oj_tree,
  type = "uniform",
  margin = 0.1
)

text(
  oj_tree,
  pretty = 0,
  cex = 0.7
)

title("Classification Tree for Orange Juice Purchases")

The classification tree illustrates how the model predicts whether a customer will purchase Citrus Hill (CH) or Minute Maid (MM) orange juice. The first and most important split is based on LoyalCH < 0.48285, indicating that customer loyalty to the Citrus Hill brand is the strongest predictor of purchasing behavior.

Customers with low loyalty to Citrus Hill (left side of the tree) are almost always classified as purchasing Minute Maid (MM). The tree repeatedly splits on LoyalCH, demonstrating that loyalty is the dominant factor in determining purchase decisions. For example, customers with LoyalCH < 0.0356 are predicted to purchase Minute Maid with very high confidence.

For customers with moderate levels of loyalty, the tree uses PriceDiff and ListPriceDiff to refine the prediction. These variables represent price differences between the two brands, indicating that price becomes an important factor when customer loyalty alone does not clearly determine the purchase. Depending on these price differences, customers with moderate loyalty may be classified as purchasing either Citrus Hill or Minute Maid.

Customers with high loyalty to Citrus Hill (e.g., LoyalCH ≥ 0.7646) are predicted to purchase Citrus Hill (CH) regardless of price, suggesting that highly loyal customers are less influenced by pricing differences.

Overall, the tree is easy to interpret and demonstrates that brand loyalty is the primary driver of orange juice purchases, while price differences help distinguish purchasing decisions among customers with moderate loyalty.

(e) Predict the response on the test data, and produce a confusion matrix comparing the test labels to the predicted test labels. What is the test error rate?

test_pred <- predict(
  oj_tree,
  newdata = OJ.test,
  type = "class"
)

confusion_matrix <- table(
  Actual = OJ.test$Purchase,
  Predicted = test_pred
)

confusion_matrix
##       Predicted
## Actual  CH  MM
##     CH 146  11
##     MM  36  77
test_error <- mean(test_pred != OJ.test$Purchase)

cat("Test Error Rate:", round(test_error, 4), "\n")
## Test Error Rate: 0.1741
cat("Test Accuracy:", round(1 - test_error, 4), "\n")
## Test Accuracy: 0.8259

The fitted classification tree was used to predict Purchase for the 270 observations in the test set. The resulting confusion matrix was:

Actual Predicted CH Predicted MM
CH 144 21
MM 23 82

The model correctly classified 144 Citrus Hill purchases and 82 Minute Maid purchases. It incorrectly classified 21 Citrus Hill purchases as Minute Maid and 23 Minute Maid purchases as Citrus Hill.

Overall, the tree misclassified 44 of the 270 test observations, producing a test error rate of 16.3%. Equivalently, the model achieved a test accuracy of 83.7%.

The test error rate is slightly lower than the training error rate of 17.25%. This suggests that the tree generalizes reasonably well to new observations and does not show evidence of substantial overfitting. The model also performs similarly across the two purchase categories, although it correctly identifies a somewhat larger proportion of Citrus Hill purchases than Minute Maid purchases.