Section A

The response is holiday and the predictors are: season, workingday, casual, and registered. Please use svm() function with radial kernel and gamma = 10 and cost = 100.

svm.model <- svm(holiday ~ season + workingday + casual + registered,
                 data = trainData,
                 kernel = "radial",
                 gamma = 10,
                 cost = 100)
summary(svm.model)

## 
## Call:
## svm(formula = holiday ~ season + workingday + casual + registered, 
##     data = trainData, kernel = "radial", gamma = 10, cost = 100)
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  100 
## 
## Number of Support Vectors:  633
## 
##  ( 428 205 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  Non-Holiday Holiday

The SVM model uses a radial kernel for a binary classification task (Holiday vs. Non-Holiday), with a gamma of 10 and cost of 100. These parameters increase model complexity, potentially improving fit but also raising the risk of overfitting.

The model relies on 633 support vectors (428 from one class, 205 from the other), indicating that the data may not be easily separable. This could also reflect some class imbalance.

Section B

Perform a grid search to find the best model with potential cost: 1, 10, 50, 100 and potential gamma: 1, 3, and 5 and using radial kernel and training dataset.

tuned.results <- tune(svm, holiday ~ season + workingday + casual + registered, 
                     data = trainData, kernel = "radial", 
                     ranges = list(cost = c(1, 10, 50, 100), 
                                   gamma = c(1, 3, 5)
                     ))
summary(tuned.results)

## 
## Parameter tuning of 'svm':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  cost gamma
##   100     1
## 
## - best performance: 0.02797245 
## 
## - Detailed performance results:
##    cost gamma      error  dispersion
## 1     1     1 0.02879909 0.003578686
## 2    10     1 0.02866154 0.003164643
## 3    50     1 0.02824812 0.003627397
## 4   100     1 0.02797245 0.004158739
## 5     1     3 0.02921269 0.003096545
## 6    10     3 0.02893702 0.003947184
## 7    50     3 0.03003971 0.004246814
## 8   100     3 0.03003952 0.004041429
## 9     1     5 0.02866154 0.003230569
## 10   10     5 0.03045293 0.004517214
## 11   50     5 0.03059048 0.002954565
## 12  100     5 0.03169260 0.002893269

Models with gamma = 1 consistently outperformed those with higher gamma values, regardless of the cost setting. Increasing the cost from 1 to 100 led to steadily lower error rates, with the best performance achieved at cost = 100 and gamma = 1. In contrast, higher gamma values caused the model to overfit, resulting in poorer generalization.

Despite a slightly higher error variance, the best model remains stable and reliable. It strikes a solid balance between accuracy and generalization, making it the ideal configuration. It’s recommended to train the final SVM using these parameters and validate it on a test set to ensure robust performance.

Section C

Print out the model results. What’s the best model parameters?

best_tuned.model <- tuned.results$best.model
summary(best_tuned.model)

## 
## Call:
## best.tune(METHOD = svm, train.x = holiday ~ season + workingday + 
##     casual + registered, data = trainData, ranges = list(cost = c(1, 
##     10, 50, 100), gamma = c(1, 3, 5)), kernel = "radial")
## 
## 
## Parameters:
##    SVM-Type:  C-classification 
##  SVM-Kernel:  radial 
##        cost:  100 
## 
## Number of Support Vectors:  645
## 
##  ( 440 205 )
## 
## 
## Number of Classes:  2 
## 
## Levels: 
##  Non-Holiday Holiday

The SVM model was designed to predict holidays using features like season, workingday, casual, and registered, employing a radial basis function (RBF) kernel to handle non-linear patterns. Through grid search tuning over various cost and gamma values, the optimal parameters were found to be cost = 100 and gamma = 1, resulting in the lowest cross-validation error.

The final model uses 645 support vectors (440 for Non-Holiday, 205 for Holiday), indicating a complex decision boundary and slight class imbalance. This imbalance may affect model predictions, favoring the majority class.

Section D

Forecast holiday using the test dataset and the best model found in c).

predictions <- predict(best_tuned.model, newdata = testData)
summary(predictions)

## Non-Holiday     Holiday 
##        3620           9

This summary suggests a strong imbalance in predicted outcomes, with most predictions being “Non-Holiday”.

Section E

Get the true observations of holiday in the test dataset.

#observations <- testData$holiday
actuals <- Bike.df[-trainIdx, "holiday"]
summary(actuals)

## Non-Holiday     Holiday 
##        3532          97

The summary suggests that most records (3532 out of 3629) are from “Non-Holiday” periods, while only a small subset of records (97) correspond to Holidays. This imbalance in the data might influence modeling results.

Section F

Compute the test error by constructing the confusion matrix. Is it a good model?

conf.matrix <- confusionMatrix(predictions, actuals)
print(conf.matrix)

## Confusion Matrix and Statistics
## 
##              Reference
## Prediction    Non-Holiday Holiday
##   Non-Holiday        3530      90
##   Holiday               2       7
##                                          
##                Accuracy : 0.9746         
##                  95% CI : (0.969, 0.9795)
##     No Information Rate : 0.9733         
##     P-Value [Acc > NIR] : 0.3262         
##                                          
##                   Kappa : 0.1281         
##                                          
##  Mcnemar's Test P-Value : <2e-16         
##                                          
##             Sensitivity : 0.99943        
##             Specificity : 0.07216        
##          Pos Pred Value : 0.97514        
##          Neg Pred Value : 0.77778        
##              Prevalence : 0.97327        
##          Detection Rate : 0.97272        
##    Detection Prevalence : 0.99752        
##       Balanced Accuracy : 0.53580        
##                                          
##        'Positive' Class : Non-Holiday    
## 

Test Error = 1 – Accuracy = 1 – 0.9746 = 0.0254

The model demonstrates high overall accuracy (97.46%) and excellent sensitivity for identifying Non-Holidays, but it performs poorly in detecting Holidays due to significant class imbalance. With low specificity (7.2%), a poor Kappa score (0.1281), and low balanced accuracy (53.58%), the model is heavily biased toward the majority class. Despite appearing effective at first glance, it fails to reliably identify the minority class (Holidays), making it unsuitable if accurate Holiday prediction is important.