class age menopause tumor_size inv_nodes nodes_caps
no-rec:195 30-39:36 ge40 :122 30-34 :57 0-2 :208 no :220
rec : 80 40-49:89 lt40 : 5 25-29 :51 12-14: 3 yes: 55
50-59:91 premeno:148 20-24 :47 15-17: 6
60-69:54 15-19 :29 3-5 : 34
70-79: 5 10-14 :28 6-8 : 17
40-44 :22 9-11 : 7
(Other):41
deg_malig breast breast_quad irradiat
Min. :1.000 left :144 central : 21 no :214
1st Qu.:2.000 right:131 left_low :105 yes: 61
Median :2.000 left_up : 94
Mean :2.055 right_low: 23
3rd Qu.:3.000 right_up : 32
Max. :3.000
attach(BC)
The data is loaded and missing values are removed. Observations from the age group 20-29 and the inv_nodes group 24-26 are also removed since each has only one observation, which would cause errors during cross-validation. Unused factor levels are then dropped. The cleaned dataset has 275 observations: 195 non-recurrence and 80 recurrence case.
class age_n menopause tumor_size_n inv_nodes_n
no-rec:196 Min. :20.00 ge40 :123 Min. : 4.0 Min. : 2.000
rec : 81 1st Qu.:40.00 lt40 : 5 1st Qu.:24.0 1st Qu.: 2.000
Median :50.00 premeno:149 Median :29.0 Median : 2.000
Mean :46.43 Mean :28.4 Mean : 3.505
3rd Qu.:50.00 3rd Qu.:34.0 3rd Qu.: 2.000
Max. :70.00 Max. :54.0 Max. :26.000
nodes_caps deg_malig breast breast_quad irradiat
no :221 Min. :1.000 left :145 central : 21 no :215
yes: 56 1st Qu.:2.000 right:132 left_low :106 yes: 62
Median :2.000 left_up : 94
Mean :2.058 right_low: 23
3rd Qu.:3.000 right_up : 33
Max. :3.000
attach(BC2)
The following objects are masked from BC:
breast, breast_quad, class, deg_malig, irradiat, menopause,
nodes_caps
The BC2 dataset is loaded from the CSV file, giving 277 observations with 196 non-recurrence and 81 recurrence cases. In this version, age, tumor size, and number of invaded nodes are treated as numerical variables. Age ranges from 20 to 70 with a median of 50, tumor size ranges from 4 to 54 with a median of 29, and invaded nodes range from 2 to 26 with a median of 2.
Question #3:
library(randomForest)
Warning: package 'randomForest' was built under R version 4.5.3
randomForest 4.7-1.2
Type rfNews() to see new features/changes/bug fixes.
Call:
randomForest(formula = class ~ ., data = BC, mtry = 9, ntree = 3000, importance = T)
Type of random forest: classification
Number of trees: 3000
No. of variables tried at each split: 9
OOB estimate of error rate: 29.09%
Confusion matrix:
no-rec rec class.error
no-rec 161 34 0.174359
rec 46 34 0.575000
plot(bag.fit$err.rate[,1])
Bagging is applied to BC using all 9 predictors at each split and 3,000 trees. The OOB error rate is 29.09%. The confusion matrix shows the model correctly classified 161 of 195 non-recurrence cases and 34 of 80 recurrence cases, meaning it struggles more with identifying recurrence (error rate 57.5%) than non-recurrence (error rate 17.4%). The error rate plot flattens well before 3,000 trees, so 3,000 trees are more than sufficient to reach a stable plateau.
Question #4:
library(ROCR)
Warning: package 'ROCR' was built under R version 4.5.3
The area under the curve is 0.673. There is approximately a 67.3% chance that the bagging model will rank a randomly chosen recurrence case higher than a randomly chosen no-recurrence case.
Call:
randomForest(formula = class ~ ., data = BC2, mtry = 9, ntree = 3000, importance = T)
Type of random forest: classification
Number of trees: 3000
No. of variables tried at each split: 9
OOB estimate of error rate: 29.96%
Confusion matrix:
no-rec rec class.error
no-rec 164 32 0.1632653
rec 51 30 0.6296296
Bagging on BC2 gives an OOB error rate of 29.96%, slightly worse than BC’s 29.09%. The AUC is 0.6603, also lower than BC’s 0.6727. Contrary to what might be expected, treating the three predictors as numerical did not improve bagging performance here. The non-recurrence class error actually improved slightly (16.3% vs. 17.4%), but the recurrence class error worsened (63.0% vs. 57.5%). Overall, the two datasets perform similarly under bagging, with BC performing marginally better.
Call:
randomForest(formula = class ~ ., data = BC, mtry = 3, ntree = 3000, importance = T)
Type of random forest: classification
Number of trees: 3000
No. of variables tried at each split: 3
OOB estimate of error rate: 26.18%
Confusion matrix:
no-rec rec class.error
no-rec 170 25 0.1282051
rec 47 33 0.5875000
plot(rf.fit$err.rate[,1])
Random forest is applied to BC using 3 predictors at each split and 3,000 trees. The OOB error rate is 26.18%, a noticeable improvement over bagging’s 29.09%. The non-recurrence error rate improved to 12.8% and the recurrence error rate to 58.8%. The error rate plot stabilizes well before 3,000 trees, so 3,000 trees are sufficient. Decorrelating the trees by using only 3 predictors per split reduced overall error compared to using all 9.
Question #7:
rocplot(rf.fit$votes[,2], BC$class)
auc.rf =auc(rf.fit$votes[,2], BC$class)auc.rf
[1] 0.6878846
The AUC for random forest on BC is 0.6879, higher than bagging’s 0.6727. This confirms that random forest has better discriminative ability than bagging for this dataset, consistent with the lower OOB error rate observed in Question 6.
Call:
randomForest(formula = class ~ ., data = BC2, mtry = 3, ntree = 3000, importance = T)
Type of random forest: classification
Number of trees: 3000
No. of variables tried at each split: 3
OOB estimate of error rate: 25.63%
Confusion matrix:
no-rec rec class.error
no-rec 175 21 0.1071429
rec 50 31 0.6172840
plot(rf.fit2$err.rate[,1])
rocplot(rf.fit2$votes[,2], BC2$class)
auc.rf2 =auc(rf.fit2$votes[,2], BC2$class)auc.rf2
[1] 0.6926178
Random forest on BC2 produces an OOB error rate of 25.63%, the best result across all four models. The AUC is 0.6926, also the highest of the four. Unlike bagging where BC outperformed BC2, random forest benefits from the numerical encoding of age, tumor size, and invaded nodes, allowing splits at optimal continuous thresholds. Comparing across all models: random forest on BC2 (OOB 25.63%, AUC 0.6926) edges out random forest on BC (26.18%, 0.6879), and both outperform bagging on either dataset. In all cases, correctly identifying recurrence remains the harder task, with recurrence class error rates hovering above 58% across all models.
When comparing the 2 datasets (BC and BC2) and their models, we can see that both models’ cv error is minimized at 1000 trees, which is the maximum number of trees we allow for each gbm model. Both graphs of each model’s error don’t level out within 1000 trees, indicating that generating more trees in each model would lower cv error even further. This confusion matrix for each model shows that BC2’s model has a lower overall error, but a higher type 1 error while BC’s model has a higher type 2 error. Both models show positive relationships between false and true positive rate, BC2’s model has a lower ROC AUC meaning BC2’s model is better able to distringuish between classes and predict accurately.
Question #14:
library(e1071)
Warning: package 'e1071' was built under R version 4.5.3
Bagging gives an OOB estimate of the error rate of 29.09%. It struggles more with identifying recurrence (error rate 57.5%) than non-recurrence (error rate 17.4%). The area under the ROC curve is 0.673.
With random forest, the OOB error rate is 26.18%, a noticeable improvement over bagging’s 29.09%. The non-recurrence error rate improved to 12.8% and the recurrence error rate to 58.8%. The AUC for random forest is 0.6879, higher than bagging’s 0.6727.
With boosting, the area under the curve is 0.71, which is better than the bagging curve. With the confusion matrix, the test error rate is 28%, which is slightly better than the bagging error rate and slightly worse than the random forest error rate.
In svm, the best cost is 20 and the best gamma is 0.05. The test error rate is 22.55% and the area under the ROC curve is 0.725. These are the best performance measures so far.
BC2:
Bagging on BC2 gives an OOB error rate of 29.96%, slightly worse than BC’s 29.09%. The AUC is 0.6603, also lower than BC’s 0.6727. Treating the three predictors as numerical did not improve bagging performance here. The non-recurrence class error improved slightly (16.3% vs. 17.4%), but the recurrence class error worsened (63.0% vs. 57.5%). BC performs a bit better in general.
Random forest on BC2 produces an OOB error rate of 25.63%. The AUC is 0.6926. BC2 performs better than BC with random forest. BC2 random forest outperforms bagging on both BC and BC2. Encoding predictors numerically helps tree splitting, but correctly identifying recurrence remains the harder task.
Boosting test error rate is 27.44% and the area under the ROC curve is 0.73. When comparing the 2 datasets (BC and BC2) and their models, we can see that both models’ cv error is minimized at 1000 trees, which is the maximum number of trees we allow for each gbm model. Both graphs of each model’s error don’t level out within 1000 trees, indicating that generating more trees in each model would lower cv error even further. This confusion matrix for each model shows that BC2’s model has a lower overall error, but a higher type 1 error while BC’s model has a higher type 2 error. Both models show positive relationships between false and true positive rate, BC2’s model has a lower ROC AUC meaning BC2’s model is better abler to distinguish between classes and predict accurately.
In svm, the test error rate is 25.63% and the area under the ROC curve is 0.664. The test error rate is the same as with random forest.
Across all four BC2 models, svm and random forest give the best results with test error rate. Boosting gives the best performance with the ROC curve.
In every model across BC and BC2, bagging is outperformed. The best test error rate across all of BC and BC2 is svm in BC. However, it doesn’t do as well when predictors are encoded numerically. Svm in BC2 gives the worst AUC indicating it just predicts the majority class more. Across every model, boosting in BC2 gives the best AUC, which makes it best for ranking. However, neither BC or BC2 boosting model error rates level off at 1000 trees. Boosting performs best if your priority is to detect recurrences.
