Homework 6
Blanca Kishi
04/12/2026
Starter code for German credit scoring
Refer to http://archive.ics.uci.edu/ml/datasets/Statlog+(German+Credit+Data))
for variable description. The response variable is Class
and all others are predictors.
Only run the following code once to install the package
caret. The German credit scoring data in
provided in that package.
install.packages('caret')Task1: Data Preparation
1. Load the caret package and the GermanCredit dataset. (10pts)
library(caret) #this package contains the german data with its numeric format## Loading required package: ggplot2## Loading required package: latticelibrary(e1071)##
## Attaching package: 'e1071'## The following object is masked from 'package:ggplot2':
##
## elementdata(GermanCredit)
GermanCredit$Class <- as.numeric(GermanCredit$Class == "Good") # use this code to convert `Class` into True or False (equivalent to 1 or 0)
GermanCredit$Class <- as.factor(GermanCredit$Class) #make sure `Class` is a factor as SVM require a factor response，now 1 is good and 0 is bad.
str(GermanCredit)## 'data.frame': 1000 obs. of 62 variables:
## $ Duration : int 6 48 12 42 24 36 24 36 12 30 ...
## $ Amount : int 1169 5951 2096 7882 4870 9055 2835 6948 3059 5234 ...
## $ InstallmentRatePercentage : int 4 2 2 2 3 2 3 2 2 4 ...
## $ ResidenceDuration : int 4 2 3 4 4 4 4 2 4 2 ...
## $ Age : int 67 22 49 45 53 35 53 35 61 28 ...
## $ NumberExistingCredits : int 2 1 1 1 2 1 1 1 1 2 ...
## $ NumberPeopleMaintenance : int 1 1 2 2 2 2 1 1 1 1 ...
## $ Telephone : num 0 1 1 1 1 0 1 0 1 1 ...
## $ ForeignWorker : num 1 1 1 1 1 1 1 1 1 1 ...
## $ Class : Factor w/ 2 levels "0","1": 2 1 2 2 1 2 2 2 2 1 ...
## $ CheckingAccountStatus.lt.0 : num 1 0 0 1 1 0 0 0 0 0 ...
## $ CheckingAccountStatus.0.to.200 : num 0 1 0 0 0 0 0 1 0 1 ...
## $ CheckingAccountStatus.gt.200 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CheckingAccountStatus.none : num 0 0 1 0 0 1 1 0 1 0 ...
## $ CreditHistory.NoCredit.AllPaid : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.ThisBank.AllPaid : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.PaidDuly : num 0 1 0 1 0 1 1 1 1 0 ...
## $ CreditHistory.Delay : num 0 0 0 0 1 0 0 0 0 0 ...
## $ CreditHistory.Critical : num 1 0 1 0 0 0 0 0 0 1 ...
## $ Purpose.NewCar : num 0 0 0 0 1 0 0 0 0 1 ...
## $ Purpose.UsedCar : num 0 0 0 0 0 0 0 1 0 0 ...
## $ Purpose.Furniture.Equipment : num 0 0 0 1 0 0 1 0 0 0 ...
## $ Purpose.Radio.Television : num 1 1 0 0 0 0 0 0 1 0 ...
## $ Purpose.DomesticAppliance : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Repairs : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Education : num 0 0 1 0 0 1 0 0 0 0 ...
## $ Purpose.Vacation : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Retraining : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Business : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Other : num 0 0 0 0 0 0 0 0 0 0 ...
## $ SavingsAccountBonds.lt.100 : num 0 1 1 1 1 0 0 1 0 1 ...
## $ SavingsAccountBonds.100.to.500 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ SavingsAccountBonds.500.to.1000 : num 0 0 0 0 0 0 1 0 0 0 ...
## $ SavingsAccountBonds.gt.1000 : num 0 0 0 0 0 0 0 0 1 0 ...
## $ SavingsAccountBonds.Unknown : num 1 0 0 0 0 1 0 0 0 0 ...
## $ EmploymentDuration.lt.1 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ EmploymentDuration.1.to.4 : num 0 1 0 0 1 1 0 1 0 0 ...
## $ EmploymentDuration.4.to.7 : num 0 0 1 1 0 0 0 0 1 0 ...
## $ EmploymentDuration.gt.7 : num 1 0 0 0 0 0 1 0 0 0 ...
## $ EmploymentDuration.Unemployed : num 0 0 0 0 0 0 0 0 0 1 ...
## $ Personal.Male.Divorced.Seperated : num 0 0 0 0 0 0 0 0 1 0 ...
## $ Personal.Female.NotSingle : num 0 1 0 0 0 0 0 0 0 0 ...
## $ Personal.Male.Single : num 1 0 1 1 1 1 1 1 0 0 ...
## $ Personal.Male.Married.Widowed : num 0 0 0 0 0 0 0 0 0 1 ...
## $ Personal.Female.Single : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherDebtorsGuarantors.None : num 1 1 1 0 1 1 1 1 1 1 ...
## $ OtherDebtorsGuarantors.CoApplicant : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherDebtorsGuarantors.Guarantor : num 0 0 0 1 0 0 0 0 0 0 ...
## $ Property.RealEstate : num 1 1 1 0 0 0 0 0 1 0 ...
## $ Property.Insurance : num 0 0 0 1 0 0 1 0 0 0 ...
## $ Property.CarOther : num 0 0 0 0 0 0 0 1 0 1 ...
## $ Property.Unknown : num 0 0 0 0 1 1 0 0 0 0 ...
## $ OtherInstallmentPlans.Bank : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherInstallmentPlans.Stores : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherInstallmentPlans.None : num 1 1 1 1 1 1 1 1 1 1 ...
## $ Housing.Rent : num 0 0 0 0 0 0 0 1 0 0 ...
## $ Housing.Own : num 1 1 1 0 0 0 1 0 1 1 ...
## $ Housing.ForFree : num 0 0 0 1 1 1 0 0 0 0 ...
## $ Job.UnemployedUnskilled : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Job.UnskilledResident : num 0 0 1 0 0 1 0 0 1 0 ...
## $ Job.SkilledEmployee : num 1 1 0 1 1 0 1 0 0 0 ...
## $ Job.Management.SelfEmp.HighlyQualified: num 0 0 0 0 0 0 0 1 0 1 ...# This is the code that drop variables that provide no information in the data
# Just run it
GermanCredit = GermanCredit[,-c(14,19,27,30,35,40,44,45,48,52,55,58,62)]2. Explore the dataset to understand its structure. It’s okay to use same code from last homework. (5pts)
# Explore structure and summary
str(GermanCredit)## 'data.frame': 1000 obs. of 49 variables:
## $ Duration : int 6 48 12 42 24 36 24 36 12 30 ...
## $ Amount : int 1169 5951 2096 7882 4870 9055 2835 6948 3059 5234 ...
## $ InstallmentRatePercentage : int 4 2 2 2 3 2 3 2 2 4 ...
## $ ResidenceDuration : int 4 2 3 4 4 4 4 2 4 2 ...
## $ Age : int 67 22 49 45 53 35 53 35 61 28 ...
## $ NumberExistingCredits : int 2 1 1 1 2 1 1 1 1 2 ...
## $ NumberPeopleMaintenance : int 1 1 2 2 2 2 1 1 1 1 ...
## $ Telephone : num 0 1 1 1 1 0 1 0 1 1 ...
## $ ForeignWorker : num 1 1 1 1 1 1 1 1 1 1 ...
## $ Class : Factor w/ 2 levels "0","1": 2 1 2 2 1 2 2 2 2 1 ...
## $ CheckingAccountStatus.lt.0 : num 1 0 0 1 1 0 0 0 0 0 ...
## $ CheckingAccountStatus.0.to.200 : num 0 1 0 0 0 0 0 1 0 1 ...
## $ CheckingAccountStatus.gt.200 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.NoCredit.AllPaid : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.ThisBank.AllPaid : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.PaidDuly : num 0 1 0 1 0 1 1 1 1 0 ...
## $ CreditHistory.Delay : num 0 0 0 0 1 0 0 0 0 0 ...
## $ Purpose.NewCar : num 0 0 0 0 1 0 0 0 0 1 ...
## $ Purpose.UsedCar : num 0 0 0 0 0 0 0 1 0 0 ...
## $ Purpose.Furniture.Equipment : num 0 0 0 1 0 0 1 0 0 0 ...
## $ Purpose.Radio.Television : num 1 1 0 0 0 0 0 0 1 0 ...
## $ Purpose.DomesticAppliance : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Repairs : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Education : num 0 0 1 0 0 1 0 0 0 0 ...
## $ Purpose.Retraining : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Business : num 0 0 0 0 0 0 0 0 0 0 ...
## $ SavingsAccountBonds.lt.100 : num 0 1 1 1 1 0 0 1 0 1 ...
## $ SavingsAccountBonds.100.to.500 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ SavingsAccountBonds.500.to.1000 : num 0 0 0 0 0 0 1 0 0 0 ...
## $ SavingsAccountBonds.gt.1000 : num 0 0 0 0 0 0 0 0 1 0 ...
## $ EmploymentDuration.lt.1 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ EmploymentDuration.1.to.4 : num 0 1 0 0 1 1 0 1 0 0 ...
## $ EmploymentDuration.4.to.7 : num 0 0 1 1 0 0 0 0 1 0 ...
## $ EmploymentDuration.gt.7 : num 1 0 0 0 0 0 1 0 0 0 ...
## $ Personal.Male.Divorced.Seperated : num 0 0 0 0 0 0 0 0 1 0 ...
## $ Personal.Female.NotSingle : num 0 1 0 0 0 0 0 0 0 0 ...
## $ Personal.Male.Single : num 1 0 1 1 1 1 1 1 0 0 ...
## $ OtherDebtorsGuarantors.None : num 1 1 1 0 1 1 1 1 1 1 ...
## $ OtherDebtorsGuarantors.CoApplicant: num 0 0 0 0 0 0 0 0 0 0 ...
## $ Property.RealEstate : num 1 1 1 0 0 0 0 0 1 0 ...
## $ Property.Insurance : num 0 0 0 1 0 0 1 0 0 0 ...
## $ Property.CarOther : num 0 0 0 0 0 0 0 1 0 1 ...
## $ OtherInstallmentPlans.Bank : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherInstallmentPlans.Stores : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Housing.Rent : num 0 0 0 0 0 0 0 1 0 0 ...
## $ Housing.Own : num 1 1 1 0 0 0 1 0 1 1 ...
## $ Job.UnemployedUnskilled : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Job.UnskilledResident : num 0 0 1 0 0 1 0 0 1 0 ...
## $ Job.SkilledEmployee : num 1 1 0 1 1 0 1 0 0 0 ...summary(GermanCredit)## Duration Amount InstallmentRatePercentage ResidenceDuration
## Min. : 4.0 Min. : 250 Min. :1.000 Min. :1.000
## 1st Qu.:12.0 1st Qu.: 1366 1st Qu.:2.000 1st Qu.:2.000
## Median :18.0 Median : 2320 Median :3.000 Median :3.000
## Mean :20.9 Mean : 3271 Mean :2.973 Mean :2.845
## 3rd Qu.:24.0 3rd Qu.: 3972 3rd Qu.:4.000 3rd Qu.:4.000
## Max. :72.0 Max. :18424 Max. :4.000 Max. :4.000
## Age NumberExistingCredits NumberPeopleMaintenance Telephone
## Min. :19.00 Min. :1.000 Min. :1.000 Min. :0.000
## 1st Qu.:27.00 1st Qu.:1.000 1st Qu.:1.000 1st Qu.:0.000
## Median :33.00 Median :1.000 Median :1.000 Median :1.000
## Mean :35.55 Mean :1.407 Mean :1.155 Mean :0.596
## 3rd Qu.:42.00 3rd Qu.:2.000 3rd Qu.:1.000 3rd Qu.:1.000
## Max. :75.00 Max. :4.000 Max. :2.000 Max. :1.000
## ForeignWorker Class CheckingAccountStatus.lt.0
## Min. :0.000 0:300 Min. :0.000
## 1st Qu.:1.000 1:700 1st Qu.:0.000
## Median :1.000 Median :0.000
## Mean :0.963 Mean :0.274
## 3rd Qu.:1.000 3rd Qu.:1.000
## Max. :1.000 Max. :1.000
## CheckingAccountStatus.0.to.200 CheckingAccountStatus.gt.200
## Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000
## Mean :0.269 Mean :0.063
## 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000
## CreditHistory.NoCredit.AllPaid CreditHistory.ThisBank.AllPaid
## Min. :0.00 Min. :0.000
## 1st Qu.:0.00 1st Qu.:0.000
## Median :0.00 Median :0.000
## Mean :0.04 Mean :0.049
## 3rd Qu.:0.00 3rd Qu.:0.000
## Max. :1.00 Max. :1.000
## CreditHistory.PaidDuly CreditHistory.Delay Purpose.NewCar Purpose.UsedCar
## Min. :0.00 Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.00 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :1.00 Median :0.000 Median :0.000 Median :0.000
## Mean :0.53 Mean :0.088 Mean :0.234 Mean :0.103
## 3rd Qu.:1.00 3rd Qu.:0.000 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.00 Max. :1.000 Max. :1.000 Max. :1.000
## Purpose.Furniture.Equipment Purpose.Radio.Television Purpose.DomesticAppliance
## Min. :0.000 Min. :0.00 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.00 1st Qu.:0.000
## Median :0.000 Median :0.00 Median :0.000
## Mean :0.181 Mean :0.28 Mean :0.012
## 3rd Qu.:0.000 3rd Qu.:1.00 3rd Qu.:0.000
## Max. :1.000 Max. :1.00 Max. :1.000
## Purpose.Repairs Purpose.Education Purpose.Retraining Purpose.Business
## Min. :0.000 Min. :0.00 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.00 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.00 Median :0.000 Median :0.000
## Mean :0.022 Mean :0.05 Mean :0.009 Mean :0.097
## 3rd Qu.:0.000 3rd Qu.:0.00 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.00 Max. :1.000 Max. :1.000
## SavingsAccountBonds.lt.100 SavingsAccountBonds.100.to.500
## Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000
## Median :1.000 Median :0.000
## Mean :0.603 Mean :0.103
## 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000
## SavingsAccountBonds.500.to.1000 SavingsAccountBonds.gt.1000
## Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000
## Mean :0.063 Mean :0.048
## 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000
## EmploymentDuration.lt.1 EmploymentDuration.1.to.4 EmploymentDuration.4.to.7
## Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000 Median :0.000
## Mean :0.172 Mean :0.339 Mean :0.174
## 3rd Qu.:0.000 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000 Max. :1.000
## EmploymentDuration.gt.7 Personal.Male.Divorced.Seperated
## Min. :0.000 Min. :0.00
## 1st Qu.:0.000 1st Qu.:0.00
## Median :0.000 Median :0.00
## Mean :0.253 Mean :0.05
## 3rd Qu.:1.000 3rd Qu.:0.00
## Max. :1.000 Max. :1.00
## Personal.Female.NotSingle Personal.Male.Single OtherDebtorsGuarantors.None
## Min. :0.00 Min. :0.000 Min. :0.000
## 1st Qu.:0.00 1st Qu.:0.000 1st Qu.:1.000
## Median :0.00 Median :1.000 Median :1.000
## Mean :0.31 Mean :0.548 Mean :0.907
## 3rd Qu.:1.00 3rd Qu.:1.000 3rd Qu.:1.000
## Max. :1.00 Max. :1.000 Max. :1.000
## OtherDebtorsGuarantors.CoApplicant Property.RealEstate Property.Insurance
## Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000 Median :0.000
## Mean :0.041 Mean :0.282 Mean :0.232
## 3rd Qu.:0.000 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000 Max. :1.000
## Property.CarOther OtherInstallmentPlans.Bank OtherInstallmentPlans.Stores
## Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000 Median :0.000
## Mean :0.332 Mean :0.139 Mean :0.047
## 3rd Qu.:1.000 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000 Max. :1.000
## Housing.Rent Housing.Own Job.UnemployedUnskilled Job.UnskilledResident
## Min. :0.000 Min. :0.000 Min. :0.000 Min. :0.0
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.0
## Median :0.000 Median :1.000 Median :0.000 Median :0.0
## Mean :0.179 Mean :0.713 Mean :0.022 Mean :0.2
## 3rd Qu.:0.000 3rd Qu.:1.000 3rd Qu.:0.000 3rd Qu.:0.0
## Max. :1.000 Max. :1.000 Max. :1.000 Max. :1.0
## Job.SkilledEmployee
## Min. :0.00
## 1st Qu.:0.00
## Median :1.00
## Mean :0.63
## 3rd Qu.:1.00
## Max. :1.00table(GermanCredit$Class) ##
## 0 1
## 300 700prop.table(table(GermanCredit$Class)) ##
## 0 1
## 0.3 0.7Your observation: The GermanCredit dataset contains 1000 observations and 41 predictors after dropping the uninformative columns. The response variable Class is a factor with two levels, showing a clear class imbalance 70% Good vs 30% Bad. Most predictors are numeric, including many dummy variables created from the original categorical features. There are no missing values.
3. Split the dataset into training and test set
with 80-20 split. Please use the random seed as 2024 for
reproducibility. (5pts)
set.seed(2024)
trainIndex <- createDataPartition(GermanCredit$Class, p = 0.8, list = FALSE)
train <- GermanCredit[trainIndex, ]
test <- GermanCredit[-trainIndex, ]
# Check sizes and class balance
dim(train)## [1] 800 49dim(test)## [1] 200 49prop.table(table(train$Class))##
## 0 1
## 0.3 0.7prop.table(table(test$Class))##
## 0 1
## 0.3 0.7Your observation: The training set has 800 observations and the test set has 200 observations. The class distribution is well preserved in both sets approximately 70% Good and 30% Bad, which ensures that the model is trained and evaluated on representative data.
Task 2: SVM without weighted class cost (30pts)
1. Fit a SVM model using the training set with linear kernel. Please use all variables, but make sure the variable types are right. If running on old laptop, could take some time! (10pts)
library(e1071)
set.seed(2024)
svm_linear <- svm(Class ~ ., data = train,
kernel = "linear",
scale = TRUE)Your observation: The model converged with issues so I added the package e1071. Because of the class imbalance, the model tends to predict the majority class “Good” more frequently.
2. Use the training set to get prediected classes. (5pts)
pred_train_class <- predict(svm_linear, train)Your observation:
3. Obtain confusion matrix and MR on training set. (5pts)
cm_train <- confusionMatrix(pred_train_class, train$Class, positive = "1")
print(cm_train)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 148 68
## 1 92 492
##
## Accuracy : 0.8
## 95% CI : (0.7706, 0.8272)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 9.547e-11
##
## Kappa : 0.5098
##
## Mcnemar's Test P-Value : 0.06902
##
## Sensitivity : 0.8786
## Specificity : 0.6167
## Pos Pred Value : 0.8425
## Neg Pred Value : 0.6852
## Prevalence : 0.7000
## Detection Rate : 0.6150
## Detection Prevalence : 0.7300
## Balanced Accuracy : 0.7476
##
## 'Positive' Class : 1
## MR_train <- 1 - cm_train$overall["Accuracy"]
cat("Training MR:", round(MR_train, 4), "\n")## Training MR: 0.2Your observation: The unweighted linear SVM achieved high accuracy but showed bias toward the “Good” class. It correctly classified most Good cases but had a higher misclassification rate for Bad cases. The misclassification rate on training data was relatively low.
4. Use the testing set to get prediected classes. (5pts)
pred_test_class <- predict(svm_linear, test)Your observation:
5. Obtain confusion matrix and MR on testing set. (5pts)
cm_test <- confusionMatrix(pred_test_class, test$Class, positive = "1")
print(cm_test)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 31 24
## 1 29 116
##
## Accuracy : 0.735
## 95% CI : (0.6681, 0.7948)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 0.1579
##
## Kappa : 0.3537
##
## Mcnemar's Test P-Value : 0.5827
##
## Sensitivity : 0.8286
## Specificity : 0.5167
## Pos Pred Value : 0.8000
## Neg Pred Value : 0.5636
## Prevalence : 0.7000
## Detection Rate : 0.5800
## Detection Prevalence : 0.7250
## Balanced Accuracy : 0.6726
##
## 'Positive' Class : 1
## MR_test <- 1 - cm_test$overall["Accuracy"]
cat("Test MR:", round(MR_test, 4), "\n")## Test MR: 0.265Your observation: The test misclassification rate was higher than on the training set, indicating some degree of overfitting. Performance on the minority “Bad” class remained weaker without class weighting.
Task 3: SVM with weighted class cost, and probabilities enabled (35pts ,each 5pts)
1. Fit a SVM model using the training set with weight of 2
on “1” and weight of 1 on “0”. Please use all variables, but make sure
the variable types are right. Also, enable probability fitting with
probability = TRUE.
class_weights <- c("0" = 1, "1" = 2)
set.seed(2024)
svm_weighted <- svm(Class ~ ., data = train,
kernel = "linear",
class.weights = class_weights,
probability = TRUE,
scale = TRUE)Your observation: Giving higher weight to the “Good” class changed the decision boundary compared to the unweighted model, making the model more sensitive to the majority class.
2. Use the training set to get prediected probabilities and classes.
pred_train_class_w <- predict(svm_weighted, train)
pred_train_prob_w <- predict(svm_weighted, train, probability = TRUE)
prob_train_1 <- attr(pred_train_prob_w, "probabilities")[, "1"]Your observation:
3. Obtain confusion matrix and MR on training set (use predicted classes).
cm_train_w <- confusionMatrix(pred_train_class_w, train$Class, positive = "1")
print(cm_train_w)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 64 8
## 1 176 552
##
## Accuracy : 0.77
## 95% CI : (0.7392, 0.7987)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 5.783e-06
##
## Kappa : 0.3155
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Sensitivity : 0.9857
## Specificity : 0.2667
## Pos Pred Value : 0.7582
## Neg Pred Value : 0.8889
## Prevalence : 0.7000
## Detection Rate : 0.6900
## Detection Prevalence : 0.9100
## Balanced Accuracy : 0.6262
##
## 'Positive' Class : 1
## MR_train_w <- 1 - cm_train_w$overall["Accuracy"]
cat("Weighted Training MR:", round(MR_train_w, 4), "\n")## Weighted Training MR: 0.23Your observation: The weighted model generally improves performance on the higher-weighted class while slightly changing the balance for the “Bad” class. The training misclassification rate changed compared to the unweighted version.
4. Obtain ROC and AUC on training set (use predicted probabilities).
library(pROC)## Type 'citation("pROC")' for a citation.##
## Attaching package: 'pROC'## The following objects are masked from 'package:stats':
##
## cov, smooth, varroc_train <- roc(train$Class, prob_train_1, levels = c("0","1"), direction = "<")
plot(roc_train, main = "ROC - Weighted SVM (Training)")
auc_train <- auc(roc_train)
cat("Training AUC:", round(auc_train, 4), "\n")## Training AUC: 0.825Your observation: The ROC curve and AUC on the training set show good discriminative ability. The AUC value indicates that the model can reasonably distinguish between Good and Bad credit risks.
5. Use the testing set to get prediected probabilities and classes.
pred_test_class_w <- predict(svm_weighted, test)
pred_test_prob_w <- predict(svm_weighted, test, probability = TRUE)
prob_test_1 <- attr(pred_test_prob_w, "probabilities")[, "1"]Your observation:
6. Obtain confusion matrix and MR on testing set. (use predicted classes).
cm_test_w <- confusionMatrix(pred_test_class_w, test$Class, positive = "1")
print(cm_test_w)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 13 9
## 1 47 131
##
## Accuracy : 0.72
## 95% CI : (0.6523, 0.781)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 0.2972
##
## Kappa : 0.186
##
## Mcnemar's Test P-Value : 7.641e-07
##
## Sensitivity : 0.9357
## Specificity : 0.2167
## Pos Pred Value : 0.7360
## Neg Pred Value : 0.5909
## Prevalence : 0.7000
## Detection Rate : 0.6550
## Detection Prevalence : 0.8900
## Balanced Accuracy : 0.5762
##
## 'Positive' Class : 1
## MR_test_w <- 1 - cm_test_w$overall["Accuracy"]
cat("Weighted Test MR:", round(MR_test_w, 4), "\n")## Weighted Test MR: 0.28Your observation: The weighted SVM produced a different confusion matrix than the unweighted model. The test misclassification rate reflects the impact of the class weights.
7. Obtain ROC and AUC on testing set. (use predicted probabilities).
roc_test <- roc(test$Class, prob_test_1, levels = c("0","1"), direction = "<")
plot(roc_test, main = "ROC - Weighted SVM (Test)")
auc_test <- auc(roc_test)
cat("Test AUC:", round(auc_test, 4), "\n")## Test AUC: 0.7115Your observation: The test AUC was 0.7115. This value shows moderate discriminative power. While not extremely high, it demonstrates that the model can separate the two classes better than random guessing.
Task 4: Conclusion (15pts)
1. Summarize your findings and discuss what you observed from the above analysis. (5pts)
In this analysis, the linear SVM without class weights performed reasonably well on overall accuracy but was biased toward predicting the majority “Good” class due to the 70/30 imbalance. Introducing class weights altered the model’s behavior and changed the confusion matrices on both training and test sets. Enabling probabilities allowed us to compute AUC, with the test AUC reaching 0.7115. Overall, the weighted model provided a different sensitivity-specificity trade-off.
2. Please recall the results from last homework, how do you compare SVM to logistic regression? No coding is required for this question, just discuss. (10pts)
Compared to logistic regression from the previous homework, the linear SVM produced similar overall accuracy and test performance on this dataset. Both models struggled with the class imbalance when unweighted, tending to favor the majority “Good” class. Logistic regression provides easily interpretable coefficients that show the direction and strength of each predictor’s effect, while SVM focuses on maximizing the margin and does not naturally give coefficient interpretations. SVM with a linear kernel behaves similarly to logistic regression but can be more robust to outliers in some cases. With class weights, SVM directly penalizes misclassifications differently per class, which is straightforward. Logistic regression can also handle weights. In terms of AUC, both models usually give comparable results on this dataset. Overall, neither model dramatically outperformed the other here. Logistic regression may be preferred when interpretability is important, while SVM can capture more complex patterns if tuned properly.