# From the book Machine Learning with R by Brett Lantz
# Diagnosing breast cancer with the
# kNN algorithm
# Step 1 – collecting data
# We will utilize the "Breast Cancer Wisconsin Diagnostic" dataset from the UCI
# Machine Learning Repository, which is available at http://archive.ics.uci.edu/ml.
setwd("C:\\Users\\GATEWAY\\Documents\\machine_learning_withR\\Machine Learning with R, Second Edition_Code\\Chapter 03")
# Step 2 – exploring and preparing the data
# Let's explore the data and see if we can shine some light on the relationships. At the
# same time, we will prepare the data for use with the kNN learning method.
wbcd <- read.csv("~/machine_learning_withR/Machine Learning with R, Second Edition_Code/Chapter 03/wisc_bc_data.csv",
stringsAsFactors=FALSE)
str(wbcd)
## 'data.frame': 569 obs. of 32 variables:
## $ id : int 87139402 8910251 905520 868871 9012568 906539 925291 87880 862989 89827 ...
## $ diagnosis : chr "B" "B" "B" "B" ...
## $ radius_mean : num 12.3 10.6 11 11.3 15.2 ...
## $ texture_mean : num 12.4 18.9 16.8 13.4 13.2 ...
## $ perimeter_mean : num 78.8 69.3 70.9 73 97.7 ...
## $ area_mean : num 464 346 373 385 712 ...
## $ smoothness_mean : num 0.1028 0.0969 0.1077 0.1164 0.0796 ...
## $ compactness_mean : num 0.0698 0.1147 0.078 0.1136 0.0693 ...
## $ concavity_mean : num 0.0399 0.0639 0.0305 0.0464 0.0339 ...
## $ points_mean : num 0.037 0.0264 0.0248 0.048 0.0266 ...
## $ symmetry_mean : num 0.196 0.192 0.171 0.177 0.172 ...
## $ dimension_mean : num 0.0595 0.0649 0.0634 0.0607 0.0554 ...
## $ radius_se : num 0.236 0.451 0.197 0.338 0.178 ...
## $ texture_se : num 0.666 1.197 1.387 1.343 0.412 ...
## $ perimeter_se : num 1.67 3.43 1.34 1.85 1.34 ...
## $ area_se : num 17.4 27.1 13.5 26.3 17.7 ...
## $ smoothness_se : num 0.00805 0.00747 0.00516 0.01127 0.00501 ...
## $ compactness_se : num 0.0118 0.03581 0.00936 0.03498 0.01485 ...
## $ concavity_se : num 0.0168 0.0335 0.0106 0.0219 0.0155 ...
## $ points_se : num 0.01241 0.01365 0.00748 0.01965 0.00915 ...
## $ symmetry_se : num 0.0192 0.035 0.0172 0.0158 0.0165 ...
## $ dimension_se : num 0.00225 0.00332 0.0022 0.00344 0.00177 ...
## $ radius_worst : num 13.5 11.9 12.4 11.9 16.2 ...
## $ texture_worst : num 15.6 22.9 26.4 15.8 15.7 ...
## $ perimeter_worst : num 87 78.3 79.9 76.5 104.5 ...
## $ area_worst : num 549 425 471 434 819 ...
## $ smoothness_worst : num 0.139 0.121 0.137 0.137 0.113 ...
## $ compactness_worst: num 0.127 0.252 0.148 0.182 0.174 ...
## $ concavity_worst : num 0.1242 0.1916 0.1067 0.0867 0.1362 ...
## $ points_worst : num 0.0939 0.0793 0.0743 0.0861 0.0818 ...
## $ symmetry_worst : num 0.283 0.294 0.3 0.21 0.249 ...
## $ dimension_worst : num 0.0677 0.0759 0.0788 0.0678 0.0677 ...
# The first variable is an integer variable named id. As this is simply a unique
# identifier (ID) for each patient in the data, it does not provide useful information
# and we will need to exclude it from the model.
# Regardless of the machine learning method, ID variables
# should always be excluded. Neglecting to do so can lead to
# erroneous findings because the ID can be used to uniquely
# "predict" each example. Therefore, a model that includes an
# identifier will most likely suffer from overfitting, and is not
# likely to generalize well to other data.
# Let's drop the id feature altogether. As it is located in the first column, we can
# exclude it by making a copy of the wbcd data frame without column 1:
wbcd <-wbcd[-1]
# The next variable, diagnosis, is of particular interest, as it is the outcome we
# hope to predict. This feature indicates whether the example is from a benign or
# malignant mass. The table() output indicates that 357 masses are benign while
# 212 are malignant:
table(wbcd$diagnosis)
##
## B M
## 357 212
# Many R machine learning classifiers require that the target feature is coded as a factor,
# so we will need to recode the diagnosis variable. We will also take this opportunity
# to give the B and M values more informative labels using the labels parameter:
wbcd$diagnosis <- factor(wbcd$diagnosis,levels = c("B","M"),labels = c("Benign","Malignant"))
summary(wbcd$diagnosis)
## Benign Malignant
## 357 212
# Now, when we look at the prop.table() output, we notice that the values have
# been labeled Benign and Malignant, with 62.7 percent and 37.3 percent of the
# masses, respectively:
round(prop.table(table(wbcd$diagnosis))*100,digits = 1)
##
## Benign Malignant
## 62.7 37.3
summary(wbcd[c("radius_mean","texture_mean","area_mean")])
## radius_mean texture_mean area_mean
## Min. : 6.981 Min. : 9.71 Min. : 143.5
## 1st Qu.:11.700 1st Qu.:16.17 1st Qu.: 420.3
## Median :13.370 Median :18.84 Median : 551.1
## Mean :14.127 Mean :19.29 Mean : 654.9
## 3rd Qu.:15.780 3rd Qu.:21.80 3rd Qu.: 782.7
## Max. :28.110 Max. :39.28 Max. :2501.0
# Looking at the features side-by-side, do you notice anything problematic about the
# values? Recall that the distance calculation for kNN is heavily dependent upon the
# measurement scale of the input features. As smoothness_mean ranges from 0.05 to
# 0.16, while area_mean ranges from 143.5 to 2501.0, the impact of area is going to be
# much larger than smoothness in the distance calculation. This could potentially cause
# problems for our classifier, so let's apply normalization to rescale the features to a
# standard range of values.
# Transformation – normalizing numeric data
normalize <- function(x){
return((x-min(x))/(max(x)-min(x)))
}
normalize(c(1,2,3,4,5))
## [1] 0.00 0.25 0.50 0.75 1.00
normalize(c(100,500,1000,10000))
## [1] 0.00000000 0.04040404 0.09090909 1.00000000
# We can now apply the normalize() function to the numeric features in our data
# frame. Rather than normalizing each of the 30 numeric variables individually, we
# will use one of R's functions to automate the process.
# The lapply() function of R takes a list and applies a function to each element of the
# list. As a data frame is a list of equal-length vectors, we can use lapply() to apply
# normalize() to each feature in the data frame. The final step is to convert the list
# returned by lapply() to a data frame using the as.data.frame() function. The full
# process looks like this:
wbcd_n <- as.data.frame(lapply(wbcd[2:31], normalize))
summary(wbcd$area_mean)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 143.5 420.3 551.1 654.9 782.7 2501.0
summary(wbcd_n$area_mean)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.1174 0.1729 0.2169 0.2711 1.0000
# Data preparation – creating training and
# test datasets
# Although all 569 biopsies are labeled with a benign or malignant status, it is not
# very interesting to predict what we already know. Additionally, any performance
# measures we obtain during training may be misleading, as we do not know the
# extent to which cases has been overfitted, or how well it will generalize to unseen
# cases. A more interesting question is how well our learner performs on a dataset
# of unlabeled data. If we had access to a laboratory, we could apply our learner to
# measurements taken from the next 100 masses of unknown cancer status and see
# how well the machine learner's predictions compare to diagnoses obtained using
# conventional methods.
# In the absence of such data, we can simulate this scenario by dividing our data into
# two portions: a training dataset that will be used to build the kNN model and a test
# dataset that will be used to estimate the predictive accuracy of the model. We will
# use the first 469 records for the training dataset and the remaining 100 to simulate
# new patients.
wbcd_train <- wbcd_n[1:469,]
wbcd_test <- wbcd_n[470:569,]
# If the previous code is confusing, remember that data is extracted from data frames
# using the [row, column] syntax. A blank value for the row or column value
# indicates that all rows or columns should be included. Hence, the first line of code
# takes rows 1 to 469 and all columns, and the second line takes 100 rows from 470 to
# 569 and all columns.
# When constructing training and test datasets, it is important
# that each dataset is a representative subset of the full set of
# data. In the case that we just saw, the records were already
# sorted in a random order, so we could simply extract 100
# consecutive records to create a test dataset. This would not be
# an appropriate method if the data was ordered in a non-random
# pattern such as chronologically, or in groups of similar values.
# In these cases, random sampling methods would be needed
# When we constructed our training and test data, we excluded the target variable,
# diagnosis. For training the kNN model, we will need to store these class labels in
# factor vectors, divided to the training and test datasets:
wbcd_train_labels <- wbcd[1:469,1]#This code takes the diagnosis factor in column 1 of
# the wbcd data frame and creates
wbcd_test_labels <- wbcd[470:569,1]#the vectors, wbcd_train_labels and wbcd_test_labels. We will use these in the
#next steps of training and evaluating our classifier.
# Step 3 – training a model on the data
# Equipped with our training data and labels vector, we are now ready to classify our
# unknown records. For the kNN algorithm, the training phase actually involves no
# model building—the process of training a lazy learner like kNN simply involves
# storing the input data in a structured format.
# To classify our test instances, we will use a kNN implementation from the class
# package, which provides a set of basic R functions for classification. If this package is
# not already installed on your system, you can install it by typing:
library(class)
# The knn() function in the class package provides a standard, classic
# implementation of the kNN algorithm. For each instance in the test data, the
# function will identify the k-nearest neighbors, using Euclidean distance, where k is
# a user-specified number. The test instance is classified by taking a "vote" among the
# k-Nearest Neighbors—specifically, this involves assigning the class of the majority of
# the k neighbors. A tie vote is broken at random.
# There are several other kNN functions in other R
# packages, providing more sophisticated or more efficient
# implementations. If you run into limits with knn(), take a
# look at the Comprehensive R Archive Network (CRAN) to
# see what else is out there. With that said, you may be surprised
# how well the basic knn() function works out of the box.
wbcd_test_pred <- knn(train = wbcd_train,test = wbcd_test,
cl= wbcd_train_labels,k=21)
# Step 4 – evaluating model performance
# The next step of the process is to evaluate how well the predicted classes in the
# wbcd_test_pred vector match up with the known values in the wbcd_test_labels
# vector. To do this, we can use the CrossTable() function in the gmodels
# package
library(gmodels)
CrossTable(x=wbcd_test_labels,y=wbcd_test_pred,
prop.chisq = FALSE)
##
##
## Cell Contents
## |-------------------------|
## | N |
## | N / Row Total |
## | N / Col Total |
## | N / Table Total |
## |-------------------------|
##
##
## Total Observations in Table: 100
##
##
## | wbcd_test_pred
## wbcd_test_labels | Benign | Malignant | Row Total |
## -----------------|-----------|-----------|-----------|
## Benign | 61 | 0 | 61 |
## | 1.000 | 0.000 | 0.610 |
## | 0.968 | 0.000 | |
## | 0.610 | 0.000 | |
## -----------------|-----------|-----------|-----------|
## Malignant | 2 | 37 | 39 |
## | 0.051 | 0.949 | 0.390 |
## | 0.032 | 1.000 | |
## | 0.020 | 0.370 | |
## -----------------|-----------|-----------|-----------|
## Column Total | 63 | 37 | 100 |
## | 0.630 | 0.370 | |
## -----------------|-----------|-----------|-----------|
##
##
# Step 5 – improving model performance
# Transformation – z-score standardization
# To standardize a vector, we can use R's built in scale() function, which by default
# rescales values using the z-score standardization. The scale() function offers the
# additional benefit that it can be applied directly to a data frame, so we can avoid use
# of the lapply() function.
wbcd_z <- as.data.frame(scale(wbcd[-1]))
# To confirm that the transformation was applied correctly, we can look at the
# summary statistics:
summary(wbcd_z$area_mean)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## -1.4530 -0.6666 -0.2949 0.0000 0.3632 5.2460
# The mean of a z-score standardized variable should always be zero, and the range
# should be fairly compact. A z-score greater than 3 or less than -3 indicates an
# extremely rare value. The previous summary seems reasonable.
# As we had done before, we need to divide the data into training and test sets, then
# classify the test instances using the knn() function. We'll then compare the predicted
# labels to the actual labels using CrossTable():
wbcd_train <- wbcd_z[1:469,]
wbcd_test <- wbcd_z[470:569,]
wbcd_train_labels <- wbcd[1:469,1]
wbcd_test_label <- wbcd[470:569,1]
wbcd_test_pred <- knn(train = wbcd_train,test = wbcd_test,
cl=wbcd_train_labels,k=21)
CrossTable(x=wbcd_test_label,y=wbcd_test_pred,
prop.chisq = FALSE)
##
##
## Cell Contents
## |-------------------------|
## | N |
## | N / Row Total |
## | N / Col Total |
## | N / Table Total |
## |-------------------------|
##
##
## Total Observations in Table: 100
##
##
## | wbcd_test_pred
## wbcd_test_label | Benign | Malignant | Row Total |
## ----------------|-----------|-----------|-----------|
## Benign | 61 | 0 | 61 |
## | 1.000 | 0.000 | 0.610 |
## | 0.924 | 0.000 | |
## | 0.610 | 0.000 | |
## ----------------|-----------|-----------|-----------|
## Malignant | 5 | 34 | 39 |
## | 0.128 | 0.872 | 0.390 |
## | 0.076 | 1.000 | |
## | 0.050 | 0.340 | |
## ----------------|-----------|-----------|-----------|
## Column Total | 66 | 34 | 100 |
## | 0.660 | 0.340 | |
## ----------------|-----------|-----------|-----------|
##
##
# Unfortunately, in the following table, the results of our new transformation show a
# slight decline in accuracy. The instances where we had correctly classified 98 percent
# of examples previously, we classified only 95 percent correctly this time. Making
# matters worse, we did no better at classifying the dangerous false negatives.
# Summary
# In this chapter, we learned about classification using k-nearest neighbors. Unlike
# many classification algorithms, kNN does not do any learning. It simply stores the
# training data verbatim. Unlabeled test examples are then matched to the most similar
# records in the training set using a distance function, and the unlabeled example is
# assigned the label of its neighbors.