Homework 6
Austin Phillips
11/09/2026
Starter code for German credit scoring
The response variable is Class. A value of 1 means Good
credit and 0 means Bad credit.
Install these packages once in the Console if you do not have them yet (do not run every time you knit):
install.packages("caret")
install.packages("e1071")
install.packages("pROC")Task 1: Data Preparation
1. Load the caret package and the GermanCredit dataset. (5pts)
library(caret)## Loading required package: ggplot2## Loading required package: latticelibrary(e1071)##
## Attaching package: 'e1071'## The following object is masked from 'package:ggplot2':
##
## elementlibrary(pROC)## Type 'citation("pROC")' for a citation.##
## Attaching package: 'pROC'## The following objects are masked from 'package:stats':
##
## cov, smooth, vardata(GermanCredit)
GermanCredit$Class <- as.numeric(GermanCredit$Class == "Good")
GermanCredit$Class <- as.factor(GermanCredit$Class)
str(GermanCredit)## 'data.frame': 1000 obs. of 62 variables:
## $ Duration : int 6 48 12 42 24 36 24 36 12 30 ...
## $ Amount : int 1169 5951 2096 7882 4870 9055 2835 6948 3059 5234 ...
## $ InstallmentRatePercentage : int 4 2 2 2 3 2 3 2 2 4 ...
## $ ResidenceDuration : int 4 2 3 4 4 4 4 2 4 2 ...
## $ Age : int 67 22 49 45 53 35 53 35 61 28 ...
## $ NumberExistingCredits : int 2 1 1 1 2 1 1 1 1 2 ...
## $ NumberPeopleMaintenance : int 1 1 2 2 2 2 1 1 1 1 ...
## $ Telephone : num 0 1 1 1 1 0 1 0 1 1 ...
## $ ForeignWorker : num 1 1 1 1 1 1 1 1 1 1 ...
## $ Class : Factor w/ 2 levels "0","1": 2 1 2 2 1 2 2 2 2 1 ...
## $ CheckingAccountStatus.lt.0 : num 1 0 0 1 1 0 0 0 0 0 ...
## $ CheckingAccountStatus.0.to.200 : num 0 1 0 0 0 0 0 1 0 1 ...
## $ CheckingAccountStatus.gt.200 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CheckingAccountStatus.none : num 0 0 1 0 0 1 1 0 1 0 ...
## $ CreditHistory.NoCredit.AllPaid : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.ThisBank.AllPaid : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CreditHistory.PaidDuly : num 0 1 0 1 0 1 1 1 1 0 ...
## $ CreditHistory.Delay : num 0 0 0 0 1 0 0 0 0 0 ...
## $ CreditHistory.Critical : num 1 0 1 0 0 0 0 0 0 1 ...
## $ Purpose.NewCar : num 0 0 0 0 1 0 0 0 0 1 ...
## $ Purpose.UsedCar : num 0 0 0 0 0 0 0 1 0 0 ...
## $ Purpose.Furniture.Equipment : num 0 0 0 1 0 0 1 0 0 0 ...
## $ Purpose.Radio.Television : num 1 1 0 0 0 0 0 0 1 0 ...
## $ Purpose.DomesticAppliance : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Repairs : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Education : num 0 0 1 0 0 1 0 0 0 0 ...
## $ Purpose.Vacation : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Retraining : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Business : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Purpose.Other : num 0 0 0 0 0 0 0 0 0 0 ...
## $ SavingsAccountBonds.lt.100 : num 0 1 1 1 1 0 0 1 0 1 ...
## $ SavingsAccountBonds.100.to.500 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ SavingsAccountBonds.500.to.1000 : num 0 0 0 0 0 0 1 0 0 0 ...
## $ SavingsAccountBonds.gt.1000 : num 0 0 0 0 0 0 0 0 1 0 ...
## $ SavingsAccountBonds.Unknown : num 1 0 0 0 0 1 0 0 0 0 ...
## $ EmploymentDuration.lt.1 : num 0 0 0 0 0 0 0 0 0 0 ...
## $ EmploymentDuration.1.to.4 : num 0 1 0 0 1 1 0 1 0 0 ...
## $ EmploymentDuration.4.to.7 : num 0 0 1 1 0 0 0 0 1 0 ...
## $ EmploymentDuration.gt.7 : num 1 0 0 0 0 0 1 0 0 0 ...
## $ EmploymentDuration.Unemployed : num 0 0 0 0 0 0 0 0 0 1 ...
## $ Personal.Male.Divorced.Seperated : num 0 0 0 0 0 0 0 0 1 0 ...
## $ Personal.Female.NotSingle : num 0 1 0 0 0 0 0 0 0 0 ...
## $ Personal.Male.Single : num 1 0 1 1 1 1 1 1 0 0 ...
## $ Personal.Male.Married.Widowed : num 0 0 0 0 0 0 0 0 0 1 ...
## $ Personal.Female.Single : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherDebtorsGuarantors.None : num 1 1 1 0 1 1 1 1 1 1 ...
## $ OtherDebtorsGuarantors.CoApplicant : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherDebtorsGuarantors.Guarantor : num 0 0 0 1 0 0 0 0 0 0 ...
## $ Property.RealEstate : num 1 1 1 0 0 0 0 0 1 0 ...
## $ Property.Insurance : num 0 0 0 1 0 0 1 0 0 0 ...
## $ Property.CarOther : num 0 0 0 0 0 0 0 1 0 1 ...
## $ Property.Unknown : num 0 0 0 0 1 1 0 0 0 0 ...
## $ OtherInstallmentPlans.Bank : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherInstallmentPlans.Stores : num 0 0 0 0 0 0 0 0 0 0 ...
## $ OtherInstallmentPlans.None : num 1 1 1 1 1 1 1 1 1 1 ...
## $ Housing.Rent : num 0 0 0 0 0 0 0 1 0 0 ...
## $ Housing.Own : num 1 1 1 0 0 0 1 0 1 1 ...
## $ Housing.ForFree : num 0 0 0 1 1 1 0 0 0 0 ...
## $ Job.UnemployedUnskilled : num 0 0 0 0 0 0 0 0 0 0 ...
## $ Job.UnskilledResident : num 0 0 1 0 0 1 0 0 1 0 ...
## $ Job.SkilledEmployee : num 1 1 0 1 1 0 1 0 0 0 ...
## $ Job.Management.SelfEmp.HighlyQualified: num 0 0 0 0 0 0 0 1 0 1 ...# Drop the variables listed in the homework.
GermanCredit <- GermanCredit[, -c(14,19,27,30,35,40,44,45,48,52,55,58,62)]2. Explore the dataset to understand its structure. (2.5pts)
dim(GermanCredit)## [1] 1000 49head(GermanCredit)## Duration Amount InstallmentRatePercentage ResidenceDuration Age
## 1 6 1169 4 4 67
## 2 48 5951 2 2 22
## 3 12 2096 2 3 49
## 4 42 7882 2 4 45
## 5 24 4870 3 4 53
## 6 36 9055 2 4 35
## NumberExistingCredits NumberPeopleMaintenance Telephone ForeignWorker Class
## 1 2 1 0 1 1
## 2 1 1 1 1 0
## 3 1 2 1 1 1
## 4 1 2 1 1 1
## 5 2 2 1 1 0
## 6 1 2 0 1 1
## CheckingAccountStatus.lt.0 CheckingAccountStatus.0.to.200
## 1 1 0
## 2 0 1
## 3 0 0
## 4 1 0
## 5 1 0
## 6 0 0
## CheckingAccountStatus.gt.200 CreditHistory.NoCredit.AllPaid
## 1 0 0
## 2 0 0
## 3 0 0
## 4 0 0
## 5 0 0
## 6 0 0
## CreditHistory.ThisBank.AllPaid CreditHistory.PaidDuly CreditHistory.Delay
## 1 0 0 0
## 2 0 1 0
## 3 0 0 0
## 4 0 1 0
## 5 0 0 1
## 6 0 1 0
## Purpose.NewCar Purpose.UsedCar Purpose.Furniture.Equipment
## 1 0 0 0
## 2 0 0 0
## 3 0 0 0
## 4 0 0 1
## 5 1 0 0
## 6 0 0 0
## Purpose.Radio.Television Purpose.DomesticAppliance Purpose.Repairs
## 1 1 0 0
## 2 1 0 0
## 3 0 0 0
## 4 0 0 0
## 5 0 0 0
## 6 0 0 0
## Purpose.Education Purpose.Retraining Purpose.Business
## 1 0 0 0
## 2 0 0 0
## 3 1 0 0
## 4 0 0 0
## 5 0 0 0
## 6 1 0 0
## SavingsAccountBonds.lt.100 SavingsAccountBonds.100.to.500
## 1 0 0
## 2 1 0
## 3 1 0
## 4 1 0
## 5 1 0
## 6 0 0
## SavingsAccountBonds.500.to.1000 SavingsAccountBonds.gt.1000
## 1 0 0
## 2 0 0
## 3 0 0
## 4 0 0
## 5 0 0
## 6 0 0
## EmploymentDuration.lt.1 EmploymentDuration.1.to.4 EmploymentDuration.4.to.7
## 1 0 0 0
## 2 0 1 0
## 3 0 0 1
## 4 0 0 1
## 5 0 1 0
## 6 0 1 0
## EmploymentDuration.gt.7 Personal.Male.Divorced.Seperated
## 1 1 0
## 2 0 0
## 3 0 0
## 4 0 0
## 5 0 0
## 6 0 0
## Personal.Female.NotSingle Personal.Male.Single OtherDebtorsGuarantors.None
## 1 0 1 1
## 2 1 0 1
## 3 0 1 1
## 4 0 1 0
## 5 0 1 1
## 6 0 1 1
## OtherDebtorsGuarantors.CoApplicant Property.RealEstate Property.Insurance
## 1 0 1 0
## 2 0 1 0
## 3 0 1 0
## 4 0 0 1
## 5 0 0 0
## 6 0 0 0
## Property.CarOther OtherInstallmentPlans.Bank OtherInstallmentPlans.Stores
## 1 0 0 0
## 2 0 0 0
## 3 0 0 0
## 4 0 0 0
## 5 0 0 0
## 6 0 0 0
## Housing.Rent Housing.Own Job.UnemployedUnskilled Job.UnskilledResident
## 1 0 1 0 0
## 2 0 1 0 0
## 3 0 1 0 1
## 4 0 0 0 0
## 5 0 0 0 0
## 6 0 0 0 1
## Job.SkilledEmployee
## 1 1
## 2 1
## 3 0
## 4 1
## 5 1
## 6 0summary(GermanCredit)## Duration Amount InstallmentRatePercentage ResidenceDuration
## Min. : 4.0 Min. : 250 Min. :1.000 Min. :1.000
## 1st Qu.:12.0 1st Qu.: 1366 1st Qu.:2.000 1st Qu.:2.000
## Median :18.0 Median : 2320 Median :3.000 Median :3.000
## Mean :20.9 Mean : 3271 Mean :2.973 Mean :2.845
## 3rd Qu.:24.0 3rd Qu.: 3972 3rd Qu.:4.000 3rd Qu.:4.000
## Max. :72.0 Max. :18424 Max. :4.000 Max. :4.000
## Age NumberExistingCredits NumberPeopleMaintenance Telephone
## Min. :19.00 Min. :1.000 Min. :1.000 Min. :0.000
## 1st Qu.:27.00 1st Qu.:1.000 1st Qu.:1.000 1st Qu.:0.000
## Median :33.00 Median :1.000 Median :1.000 Median :1.000
## Mean :35.55 Mean :1.407 Mean :1.155 Mean :0.596
## 3rd Qu.:42.00 3rd Qu.:2.000 3rd Qu.:1.000 3rd Qu.:1.000
## Max. :75.00 Max. :4.000 Max. :2.000 Max. :1.000
## ForeignWorker Class CheckingAccountStatus.lt.0
## Min. :0.000 0:300 Min. :0.000
## 1st Qu.:1.000 1:700 1st Qu.:0.000
## Median :1.000 Median :0.000
## Mean :0.963 Mean :0.274
## 3rd Qu.:1.000 3rd Qu.:1.000
## Max. :1.000 Max. :1.000
## CheckingAccountStatus.0.to.200 CheckingAccountStatus.gt.200
## Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000
## Mean :0.269 Mean :0.063
## 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000
## CreditHistory.NoCredit.AllPaid CreditHistory.ThisBank.AllPaid
## Min. :0.00 Min. :0.000
## 1st Qu.:0.00 1st Qu.:0.000
## Median :0.00 Median :0.000
## Mean :0.04 Mean :0.049
## 3rd Qu.:0.00 3rd Qu.:0.000
## Max. :1.00 Max. :1.000
## CreditHistory.PaidDuly CreditHistory.Delay Purpose.NewCar Purpose.UsedCar
## Min. :0.00 Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.00 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :1.00 Median :0.000 Median :0.000 Median :0.000
## Mean :0.53 Mean :0.088 Mean :0.234 Mean :0.103
## 3rd Qu.:1.00 3rd Qu.:0.000 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.00 Max. :1.000 Max. :1.000 Max. :1.000
## Purpose.Furniture.Equipment Purpose.Radio.Television Purpose.DomesticAppliance
## Min. :0.000 Min. :0.00 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.00 1st Qu.:0.000
## Median :0.000 Median :0.00 Median :0.000
## Mean :0.181 Mean :0.28 Mean :0.012
## 3rd Qu.:0.000 3rd Qu.:1.00 3rd Qu.:0.000
## Max. :1.000 Max. :1.00 Max. :1.000
## Purpose.Repairs Purpose.Education Purpose.Retraining Purpose.Business
## Min. :0.000 Min. :0.00 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.00 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.00 Median :0.000 Median :0.000
## Mean :0.022 Mean :0.05 Mean :0.009 Mean :0.097
## 3rd Qu.:0.000 3rd Qu.:0.00 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.00 Max. :1.000 Max. :1.000
## SavingsAccountBonds.lt.100 SavingsAccountBonds.100.to.500
## Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000
## Median :1.000 Median :0.000
## Mean :0.603 Mean :0.103
## 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000
## SavingsAccountBonds.500.to.1000 SavingsAccountBonds.gt.1000
## Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000
## Mean :0.063 Mean :0.048
## 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000
## EmploymentDuration.lt.1 EmploymentDuration.1.to.4 EmploymentDuration.4.to.7
## Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000 Median :0.000
## Mean :0.172 Mean :0.339 Mean :0.174
## 3rd Qu.:0.000 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000 Max. :1.000
## EmploymentDuration.gt.7 Personal.Male.Divorced.Seperated
## Min. :0.000 Min. :0.00
## 1st Qu.:0.000 1st Qu.:0.00
## Median :0.000 Median :0.00
## Mean :0.253 Mean :0.05
## 3rd Qu.:1.000 3rd Qu.:0.00
## Max. :1.000 Max. :1.00
## Personal.Female.NotSingle Personal.Male.Single OtherDebtorsGuarantors.None
## Min. :0.00 Min. :0.000 Min. :0.000
## 1st Qu.:0.00 1st Qu.:0.000 1st Qu.:1.000
## Median :0.00 Median :1.000 Median :1.000
## Mean :0.31 Mean :0.548 Mean :0.907
## 3rd Qu.:1.00 3rd Qu.:1.000 3rd Qu.:1.000
## Max. :1.00 Max. :1.000 Max. :1.000
## OtherDebtorsGuarantors.CoApplicant Property.RealEstate Property.Insurance
## Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000 Median :0.000
## Mean :0.041 Mean :0.282 Mean :0.232
## 3rd Qu.:0.000 3rd Qu.:1.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000 Max. :1.000
## Property.CarOther OtherInstallmentPlans.Bank OtherInstallmentPlans.Stores
## Min. :0.000 Min. :0.000 Min. :0.000
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000
## Median :0.000 Median :0.000 Median :0.000
## Mean :0.332 Mean :0.139 Mean :0.047
## 3rd Qu.:1.000 3rd Qu.:0.000 3rd Qu.:0.000
## Max. :1.000 Max. :1.000 Max. :1.000
## Housing.Rent Housing.Own Job.UnemployedUnskilled Job.UnskilledResident
## Min. :0.000 Min. :0.000 Min. :0.000 Min. :0.0
## 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.000 1st Qu.:0.0
## Median :0.000 Median :1.000 Median :0.000 Median :0.0
## Mean :0.179 Mean :0.713 Mean :0.022 Mean :0.2
## 3rd Qu.:0.000 3rd Qu.:1.000 3rd Qu.:0.000 3rd Qu.:0.0
## Max. :1.000 Max. :1.000 Max. :1.000 Max. :1.0
## Job.SkilledEmployee
## Min. :0.00
## 1st Qu.:0.00
## Median :1.00
## Mean :0.63
## 3rd Qu.:1.00
## Max. :1.00table(GermanCredit$Class)##
## 0 1
## 300 700Your observation: The data has 1,000 observations.
Class is a factor with two groups: 0 (Bad) and 1 (Good).
The table shows how many customers are in each group. After dropping the
listed columns, I can use the remaining variables to predict credit
class.
3. Split the dataset into training and test sets (80-20), seed 2024. (2.5pts)
set.seed(2024)
train_rows <- createDataPartition(GermanCredit$Class, p = 0.80, list = FALSE)
train <- GermanCredit[train_rows, ]
test <- GermanCredit[-train_rows, ]
nrow(train)## [1] 800nrow(test)## [1] 200table(train$Class)##
## 0 1
## 240 560table(test$Class)##
## 0 1
## 60 140Your observation: About 80% of the data is used to train the model, and 20% is saved to test how well it predicts new observations.
Task 2: SVM without weighted class cost (15pts)
1. Fit a linear SVM model using the training set. (5pts)
# Class is already a factor, which is needed for classification.
svm1 <- svm(Class ~ ., data = train, kernel = "linear")
svm1##
## Call:
## svm(formula = Class ~ ., data = train, kernel = "linear")
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 1
##
## Number of Support Vectors: 403Your observation: I fitted a linear SVM using all remaining predictors. This model does not give either class extra weight.
2. Predict classes on the training set. (2.5pts)
train_pred1 <- predict(svm1, train)
head(train_pred1)## 1 2 3 4 5 6
## 1 0 1 1 0 1
## Levels: 0 1Your observation: The predictions are either 0 (Bad) or 1 (Good).
3. Confusion matrix and misclassification rate (MR) on training set. (2.5pts)
confusionMatrix(train_pred1, train$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 148 68
## 1 92 492
##
## Accuracy : 0.8
## 95% CI : (0.7706, 0.8272)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 9.547e-11
##
## Kappa : 0.5098
##
## Mcnemar's Test P-Value : 0.06902
##
## Sensitivity : 0.6167
## Specificity : 0.8786
## Pos Pred Value : 0.6852
## Neg Pred Value : 0.8425
## Prevalence : 0.3000
## Detection Rate : 0.1850
## Detection Prevalence : 0.2700
## Balanced Accuracy : 0.7476
##
## 'Positive' Class : 0
## train_mr1 <- mean(train_pred1 != train$Class)
train_mr1## [1] 0.2Your observation: The confusion matrix shows correct and incorrect predictions. The training misclassification rate is 0.2, or 20%.
4. Predict classes on the testing set. (2.5pts)
test_pred1 <- predict(svm1, test)
head(test_pred1)## 17 21 24 28 33 46
## 1 1 1 1 1 1
## Levels: 0 1Your observation: I used the fitted model to predict the credit classes in the test set.
5. Confusion matrix and MR on testing set. (2.5pts)
confusionMatrix(test_pred1, test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 31 24
## 1 29 116
##
## Accuracy : 0.735
## 95% CI : (0.6681, 0.7948)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 0.1579
##
## Kappa : 0.3537
##
## Mcnemar's Test P-Value : 0.5827
##
## Sensitivity : 0.5167
## Specificity : 0.8286
## Pos Pred Value : 0.5636
## Neg Pred Value : 0.8000
## Prevalence : 0.3000
## Detection Rate : 0.1550
## Detection Prevalence : 0.2750
## Balanced Accuracy : 0.6726
##
## 'Positive' Class : 0
## test_mr1 <- mean(test_pred1 != test$Class)
test_mr1## [1] 0.265Your observation: The testing misclassification rate is 0.265, or 26.5%. This tells me how often the model is wrong on data it did not train on.
Task 3: SVM with weighted class cost and probabilities (17.5pts)
1. Fit a linear SVM with class weights and probabilities. (2.5pts)
svm2 <- svm(Class ~ ., data = train, kernel = "linear",
class.weights = c("0" = 1, "1" = 2),
probability = TRUE)
svm2##
## Call:
## svm(formula = Class ~ ., data = train, kernel = "linear", class.weights = c(`0` = 1,
## `1` = 2), probability = TRUE)
##
##
## Parameters:
## SVM-Type: C-classification
## SVM-Kernel: linear
## cost: 1
##
## Number of Support Vectors: 378Your observation: This model gives class 1 (Good) twice the weight of class 0 (Bad). I also enabled predicted probabilities.
2. Predict probabilities and classes on training set. (2.5pts)
train_pred2 <- predict(svm2, train, probability = TRUE)
train_probs2 <- attr(train_pred2, "probabilities")[, "1"]
head(train_pred2)## 1 2 3 4 5 6
## 1 0 1 1 0 1
## Levels: 0 1head(train_probs2)## 1 2 3 4 5 6
## 0.8407723 0.4059899 0.9206866 0.7775040 0.2748831 0.7380708Your observation: I obtained both predicted classes and probabilities for class 1 (Good).
3. Confusion matrix and MR on training set. (2.5pts)
confusionMatrix(train_pred2, train$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 101 24
## 1 139 536
##
## Accuracy : 0.7962
## 95% CI : (0.7666, 0.8237)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 4.762e-10
##
## Kappa : 0.4379
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Sensitivity : 0.4208
## Specificity : 0.9571
## Pos Pred Value : 0.8080
## Neg Pred Value : 0.7941
## Prevalence : 0.3000
## Detection Rate : 0.1263
## Detection Prevalence : 0.1562
## Balanced Accuracy : 0.6890
##
## 'Positive' Class : 0
## train_mr2 <- mean(train_pred2 != train$Class)
train_mr2## [1] 0.20375Your observation: The weighted model’s training misclassification rate is 0.204, or 20.4%.
4. ROC and AUC on training set. (2.5pts)
train_roc <- roc(train$Class, train_probs2,
levels = c("0", "1"), direction = "<", quiet = TRUE)
plot(train_roc, main = "Training ROC Curve")
auc(train_roc)## Area under the curve: 0.825Your observation: The training AUC is 0.825. A larger AUC (closer to 1) means the model is better at distinguishing Good from Bad credit.
5. Predict probabilities and classes on testing set. (2.5pts)
test_pred2 <- predict(svm2, test, probability = TRUE)
test_probs2 <- attr(test_pred2, "probabilities")[, "1"]
head(test_pred2)## 17 21 24 28 33 46
## 1 1 1 1 1 1
## Levels: 0 1head(test_probs2)## 17 21 24 28 33 46
## 0.8571835 0.8560695 0.8688477 0.8136918 0.7015288 0.7625487Your observation: I obtained the weighted model’s predictions and probabilities on the test set.
6. Confusion matrix and MR on testing set. (2.5pts)
confusionMatrix(test_pred2, test$Class)## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 21 14
## 1 39 126
##
## Accuracy : 0.735
## 95% CI : (0.6681, 0.7948)
## No Information Rate : 0.7
## P-Value [Acc > NIR] : 0.1578863
##
## Kappa : 0.2838
##
## Mcnemar's Test P-Value : 0.0009784
##
## Sensitivity : 0.3500
## Specificity : 0.9000
## Pos Pred Value : 0.6000
## Neg Pred Value : 0.7636
## Prevalence : 0.3000
## Detection Rate : 0.1050
## Detection Prevalence : 0.1750
## Balanced Accuracy : 0.6250
##
## 'Positive' Class : 0
## test_mr2 <- mean(test_pred2 != test$Class)
test_mr2## [1] 0.265Your observation: The weighted model’s testing misclassification rate is 0.265, or 26.5%.
7. ROC and AUC on testing set. (2.5pts)
test_roc <- roc(test$Class, test_probs2,
levels = c("0", "1"), direction = "<", quiet = TRUE)
plot(test_roc, main = "Testing ROC Curve")
auc(test_roc)## Area under the curve: 0.7115Your observation: The testing AUC is 0.712. It shows how well the weighted model separates the two classes on new data.
Task 4: Conclusion (7.5pts)
1. Summarize your findings. (2.5pts)
The unweighted SVM had a training MR of 0.2 and a testing MR of 0.265. The weighted SVM had a training MR of 0.204 and a testing MR of 0.265. The weighted model had a testing AUC of 0.712.
Both SVM models had the same testing misclassification rate. Changing the class weights affects how the model treats mistakes for each class, so I would also look at the confusion matrices rather than only the overall MR.
2. Compare SVM with logistic regression from last homework. (5pts)
In Homework 5, I used logistic regression to predict good or bad credit. I found that loan duration, credit history, and checking account status were important predictors. Changing the probability cutoff also affected how customers were classified.
In Homework 6, I used SVM, which separates customers into classes rather than directly modeling probabilities like logistic regression. Both models can predict credit risk, but SVM also allows different class weights. Overall, I would compare the test misclassification rates and AUC values to determine which model performed better, using the same training and testing data.
3. Optional: Try a radial kernel.
svm_radial <- svm(Class ~ ., data = train, kernel = "radial")
radial_pred <- predict(svm_radial, test)
confusionMatrix(radial_pred, test$Class)
mean(radial_pred != test$Class)