Cross Validation

Before we build the model, we should split the dataset into train and test data in order to perform model validation. We will split heart_clean dataset into 80% train and 20% test using sample() function and use set.seed() with the seed 417. Data train will be used for modelling and data test will be used to validate whether the model can be a good model for unseen data.

RNGkind(sample.kind = "Rounding") 

## Warning in RNGkind(sample.kind = "Rounding"): non-uniform 'Rounding' sampler
## used

set.seed(417)

index <- sample(nrow(heart_clean), nrow(heart_clean)*0.8) 
heart_train <- heart_clean[index,] 
heart_test <- heart_clean[-index,] 

Let’s take a look at the distribution of our target variable in train data using prop.table(table(data)) to make sure that the train data also have a balanced proportion of our target class.

# target class proportion for data train
prop.table(table(heart_train$target))

## 
##        no       yes 
## 0.4628099 0.5371901

Model Fitting

One of the techniques for selecting predictor variables is using stepwise regression algorithm. First, we will create a logistic regression model using glm() function to predict target using all variables. Then, we use the step() function with direction="backward" to perform stepwise regression.

model_log <- glm(target~., data = heart_train, family = "binomial")
model_step <- step(model_log, direction = "backward", trace = 0)
summary(model_step)

## 
## Call:
## glm(formula = target ~ sex + cp + trestbps + thalach + exang + 
##     slope + ca + thal, family = "binomial", data = heart_train)
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -2.93413  -0.27363   0.07124   0.37508   2.11624  
## 
## Coefficients:
##                    Estimate Std. Error z value Pr(>|z|)    
## (Intercept)        -0.55786    4.54776  -0.123 0.902371    
## sexfemale          -1.89609    0.64079  -2.959 0.003087 ** 
## cpatypical angina   1.25096    0.69705   1.795 0.072708 .  
## cpnon-anginal pain  2.40113    0.64668   3.713 0.000205 ***
## cpasymptomatic      2.12390    0.81513   2.606 0.009171 ** 
## trestbps           -0.02279    0.01233  -1.848 0.064534 .  
## thalach             0.02431    0.01324   1.836 0.066330 .  
## exangyes           -1.20845    0.53547  -2.257 0.024019 *  
## slopeflat          -0.52100    0.94624  -0.551 0.581908    
## slopedownsloping    1.58712    0.98648   1.609 0.107644    
## ca1                -2.91998    0.62354  -4.683 2.83e-06 ***
## ca2                -4.38372    0.94062  -4.660 3.15e-06 ***
## ca3                -2.60980    0.95458  -2.734 0.006257 ** 
## ca4                 1.66614    1.83606   0.907 0.364168    
## thal1               2.96444    3.95135   0.750 0.453113    
## thal2               2.10669    3.86836   0.545 0.586033    
## thal3               0.73313    3.86960   0.189 0.849733    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 334.14  on 241  degrees of freedom
## Residual deviance: 137.59  on 225  degrees of freedom
## AIC: 171.59
## 
## Number of Fisher Scoring iterations: 6

Then, we will try to predict the probability for variable target in heart_test. We will store it in the new column in heart_test dataset. After that, classify variable pred_log. If pred_log is more than 0.5, then it is classified as 1. If pred_log is below 0.5, then it is classified as 0.

# Probability prediction
heart_test$pred_log <- predict(model_step, newdata = heart_test, type = "response")
# Classifying the class
heart_test$pred_Lablog <- ifelse(heart_test$pred_log > 0.5, "yes", "no") %>% 
                          as.factor()

Model Evaluation

In this step, we will try to validate whether or not our model did an excellent job of predicting unseen data. We will make a confusion matrix from the logistic regression result based on the actual label from variable target in heart_test dataset and the predicted label from variable pred_Lablog in heart_test dataset and use the positive class as “1”.

There are some evaluating classifiers that can we use to interpret the model:

• Sensitivity/recall = the proportion of positives that are correctly identified as positive.
• Pos Pred Value/Precision = the proportion of correctly identified as positives from all classified as positive.
• Accuracy: the proportion of correctly identified cases from all cases.
• Specificity: the proportion of negatives that are correctly identified as negative.

# Model Evaluation
confusionMatrix(data = heart_test$pred_Lablog,
                reference = heart_test$target,
                positive = "yes")

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction no yes
##        no  19   4
##        yes  7  31
##                                           
##                Accuracy : 0.8197          
##                  95% CI : (0.7002, 0.9064)
##     No Information Rate : 0.5738          
##     P-Value [Acc > NIR] : 4.229e-05       
##                                           
##                   Kappa : 0.6258          
##                                           
##  Mcnemar's Test P-Value : 0.5465          
##                                           
##             Sensitivity : 0.8857          
##             Specificity : 0.7308          
##          Pos Pred Value : 0.8158          
##          Neg Pred Value : 0.8261          
##              Prevalence : 0.5738          
##          Detection Rate : 0.5082          
##    Detection Prevalence : 0.6230          
##       Balanced Accuracy : 0.8082          
##                                           
##        'Positive' Class : yes             
## 

Observe from the confusion matrix that:

• Out of 35 actual “yes”, we classified 31 of them correctly.
• Out of 26 actual “no”, we classified 19 of them correctly.
• Out of 61 cases of heart disease in our data test, we classified 50 of them correctly.
• The accuracy is 81.97%
• The sensitivity or recall is 88.57%
• The pos pred value or precision is 81.58%

Cross Validation

Now let’s try to explore the classification model using the k-Nearest Neighbor algorithm. In the k-Nearest Neighbor algorithm, we need to perform one more step of data pre-processing. For both our train and test set, drop the categorical variable from each data except our target variable. Separate the predictor and target from our train and test set.

# data train predictor
train_x <-  heart_train %>% 
            select_if(is.numeric)

# data train target
train_y <- heart_train %>% 
            select(target)

# data test predictor
test_x <-  heart_test %>% 
          select_if(is.numeric) %>% 
          select(-c(pred_log))

# data test target
test_y <- heart_test %>% 
          select(target)

Model Fitting

The distance calculation for KNN is heavily dependent upon the measurement scale of the input features. Any variable that has an extremely different range of values from the other variable can potentially cause problems for our classifier. So let’s apply normalization techniques to rescale the features to a standard range of values.

To normalize the features in train_x, we will use scale() function. Meanwhile, for the testing data, we will normalize each features using the attributes center and scale obtained from the train_x data.

# scaling
train_x <- scale(train_x)
test_x <- scale(test_x,
                center = attr(train_x, "scaled:center"), 
                scale = attr(train_x, "scaled:scale")) 

After normalizing our data, we need to find the right K to use for our kNN model. Method that we can use to choose the number of k is square root by number of row.

# number of k
sqrt(nrow(train_x))

## [1] 15.55635

Using k value we have calculated before, we will try to predict test_y using train_x and target variable from train_y dataset using knn() function.

# predict
heart_pred <- knn(train = train_x, 
                 test = test_x,
                 cl = train_y$target, 
                 k = 15)
head(heart_pred)

## [1] yes yes no  yes yes no 
## Levels: no yes

Model Evaluation

In this section, we will try to validate whether or not our model did an excellent job of predicting unseen data. We will try to make a confusion matrix from the KNN prediction result based on the actual label from target in test_y dataset and the predicted label heart_pred and use the positive class as “1”.

# Model Evaluation 
confusionMatrix(data = heart_pred,
                reference = test_y$target,
                positive = "yes")

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction no yes
##        no  15   7
##        yes 11  28
##                                           
##                Accuracy : 0.7049          
##                  95% CI : (0.5743, 0.8148)
##     No Information Rate : 0.5738          
##     P-Value [Acc > NIR] : 0.02452         
##                                           
##                   Kappa : 0.3845          
##                                           
##  Mcnemar's Test P-Value : 0.47950         
##                                           
##             Sensitivity : 0.8000          
##             Specificity : 0.5769          
##          Pos Pred Value : 0.7179          
##          Neg Pred Value : 0.6818          
##              Prevalence : 0.5738          
##          Detection Rate : 0.4590          
##    Detection Prevalence : 0.6393          
##       Balanced Accuracy : 0.6885          
##                                           
##        'Positive' Class : yes             
## 

Observe from the confusion matrix that:

• Out of 35 actual “yes”, we classified 28 of them correctly.
• Out of 26 actual “no”, we classified 15 of them correctly.
• Out of 61 cases of heart disease in our data test, we classified 43 of them correctly.
• The accuracy is 70.49%
• The sensitivity or recall is 80%
• The pos pred value or precision is 71.79%