Practical Machine Learning Project
Omid
2021-01-16
Introduction
Using devices such as Jawbone Up, Nike FuelBand, and Fitbit, it is now possible to collect a large amount of data about personal activities relatively inexpensively. These type of devices are part of the quantified self movement, a group of enthusiasts who take measurements about themselves regularly to improve their health, to find patterns in their behavior, or because they are tech geeks. One thing that people regularly do is quantify how much of a particular activity they do, but they rarely quantify how well they do it.
In this project, we will use data from accelerometers on the belt, forearm, arm, and dumbell of 6 participants. They were asked to perform barbell lifts correctly and incorrectly in 5 different ways.
The data consist of a Training data and a Test data (to be used to validate the selected model). The goal of the project is to predict the manner in which they did the exercise. This is the classe variable in the training set. You may use any of the other variables to predict with.
Library & Data Loading and Processing
We first load necessary libraries and read the training and testing data.
options(warn = -1)
library(caret);library(rpart);library(rpart.plot); library(corrplot)## Loading required package: lattice## Loading required package: ggplot2## corrplot 0.84 loadedlibrary(RColorBrewer); library(rattle); library(randomForest); library(gbm)## Loading required package: tibble## Loading required package: bitops## Rattle: A free graphical interface for data science with R.
## Version 5.4.0 Copyright (c) 2006-2020 Togaware Pty Ltd.
## Type 'rattle()' to shake, rattle, and roll your data.## randomForest 4.6-14## Type rfNews() to see new features/changes/bug fixes.##
## Attaching package: 'randomForest'## The following object is masked from 'package:rattle':
##
## importance## The following object is masked from 'package:ggplot2':
##
## margin## Loaded gbm 2.1.8train_url <- "https://d396qusza40orc.cloudfront.net/predmachlearn/pml-training.csv"
test_url <- "https://d396qusza40orc.cloudfront.net/predmachlearn/pml-testing.csv"
train_set<- read.csv(train_url, header = T, na.strings = c("", "NA"))
test_set <- read.csv(test_url, header = T, na.strings = c("", "NA"))
dim(train_set); dim(test_set)## [1] 19622 160## [1] 20 160Data Cleaning
We also remove the variables with missing values. The resulting tarining and test data consist of 19622 and 20 cases, respectively and 60 variables.
train1<- train_set[, colSums(is.na(train_set)) == 0]
test1 <- test_set[, colSums(is.na(test_set)) == 0]
dim(train1); dim(test1)## [1] 19622 60## [1] 20 60Additionally, we exclude the first 7 variables, as they do not have much impact on the outcome. That is we consider 53 variables.
train <- train1[, -c(1:7)]
test <- test1[, -c(1:7)]
dim(train); dim(test)## [1] 19622 53## [1] 20 53Data Splitting
We now divide the (training) data into 70% training set and 30% testing set. We will use the Note that later we will use the resulting prediction model on the 20 cases of the testing data.
set.seed(1234)
inTrain <- createDataPartition(train$classe, p = 0.7, list = FALSE)
trainSet<- train[inTrain, ]
testSet <- train[-inTrain, ]
dim(trainSet); dim(testSet)Correlation Cutoff
We also obtain for highly correlated variables with a cut off equal to 0.75 as follows.
corMat <- cor(trainSet[, -53])
highlyCorr = findCorrelation(corMat, cutoff=0.75)
names(trainSet)[highlyCorr]## [1] "accel_belt_z" "roll_belt" "accel_belt_y"
## [4] "total_accel_belt" "accel_dumbbell_z" "accel_belt_x"
## [7] "pitch_belt" "magnet_dumbbell_x" "accel_dumbbell_y"
## [10] "magnet_dumbbell_y" "accel_dumbbell_x" "accel_arm_x"
## [13] "accel_arm_z" "magnet_arm_y" "magnet_belt_z"
## [16] "accel_forearm_y" "gyros_forearm_y" "gyros_dumbbell_x"
## [19] "gyros_dumbbell_z" "gyros_arm_x"Model Building
We use the following algorithms to predict the outcome. * Classification trees * Random forests * Generalized Boosted Model
Prediction with Classification Trees
We first fit the model, and then we create the classification tree.
set.seed(12345)
DecTreeMod <- rpart(classe ~ ., data=trainSet, method="class")
fancyRpartPlot(DecTreeMod)
We then validate the model using the testSet and assess its performance.
PredTreeMod <- predict(DecTreeMod, testSet, type = "class")
cmtree <- confusionMatrix(PredTreeMod, testSet$classe)
cmtree## Confusion Matrix and Statistics
##
## Reference
## Prediction A B C D E
## A 1522 167 12 49 13
## B 58 706 100 79 96
## C 47 109 819 148 139
## D 25 94 67 609 52
## E 22 63 28 79 782
##
## Overall Statistics
##
## Accuracy : 0.7541
## 95% CI : (0.7429, 0.7651)
## No Information Rate : 0.2845
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.6885
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Statistics by Class:
##
## Class: A Class: B Class: C Class: D Class: E
## Sensitivity 0.9092 0.6198 0.7982 0.6317 0.7227
## Specificity 0.9428 0.9298 0.9088 0.9516 0.9600
## Pos Pred Value 0.8633 0.6795 0.6490 0.7190 0.8029
## Neg Pred Value 0.9631 0.9106 0.9552 0.9295 0.9389
## Prevalence 0.2845 0.1935 0.1743 0.1638 0.1839
## Detection Rate 0.2586 0.1200 0.1392 0.1035 0.1329
## Detection Prevalence 0.2996 0.1766 0.2144 0.1439 0.1655
## Balanced Accuracy 0.9260 0.7748 0.8535 0.7917 0.8414We see that the accuracy rate of the model is 0.75; that is the the out of sample error is almost 0.25 which is significant.
Prediction with Random Forest
For this algorithm, we first fit the model while using cross-validation for resampling.
RFcontrol <- trainControl(method="cv", number=3, verboseIter=FALSE)
RFMod <- train(classe ~ ., data=trainSet, method="rf", trControl=RFcontrol)
RFMod$finalModel##
## Call:
## randomForest(x = x, y = y, mtry = param$mtry)
## Type of random forest: classification
## Number of trees: 500
## No. of variables tried at each split: 27
##
## OOB estimate of error rate: 0.7%
## Confusion matrix:
## A B C D E class.error
## A 3902 3 0 0 1 0.001024066
## B 19 2634 5 0 0 0.009029345
## C 0 17 2369 10 0 0.011268781
## D 0 1 26 2224 1 0.012433393
## E 0 2 5 6 2512 0.005148515We then validate the model using the test set to find out how well it performs by looking at its accuracy.
PredRF <- predict(RFMod, newdata=testSet)
RFcm <- confusionMatrix(PredRF, testSet$classe)
RFcm## Confusion Matrix and Statistics
##
## Reference
## Prediction A B C D E
## A 1673 5 0 0 0
## B 0 1130 12 0 0
## C 1 4 1011 9 0
## D 0 0 3 954 0
## E 0 0 0 1 1082
##
## Overall Statistics
##
## Accuracy : 0.9941
## 95% CI : (0.9917, 0.9959)
## No Information Rate : 0.2845
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.9925
##
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: A Class: B Class: C Class: D Class: E
## Sensitivity 0.9994 0.9921 0.9854 0.9896 1.0000
## Specificity 0.9988 0.9975 0.9971 0.9994 0.9998
## Pos Pred Value 0.9970 0.9895 0.9863 0.9969 0.9991
## Neg Pred Value 0.9998 0.9981 0.9969 0.9980 1.0000
## Prevalence 0.2845 0.1935 0.1743 0.1638 0.1839
## Detection Rate 0.2843 0.1920 0.1718 0.1621 0.1839
## Detection Prevalence 0.2851 0.1941 0.1742 0.1626 0.1840
## Balanced Accuracy 0.9991 0.9948 0.9912 0.9945 0.9999The accuracy rate using random forest is 0.995; That is the out of sample error is 0.005. Note that this might be due to overfitting.
Prediction with Generalized Boosted Model (GBM)
Finally, we use GBM and 5-fold cross-validation for resampling.
set.seed(12345)
GBMcontrol <- trainControl(method = "repeatedcv", number = 5, repeats = 1)
GBMmod <- train(classe ~ ., data=trainSet, method = "gbm", trControl = GBMcontrol, verbose = FALSE)
GBMmod$finalModel## A gradient boosted model with multinomial loss function.
## 150 iterations were performed.
## There were 52 predictors of which 52 had non-zero influence.print(GBMmod)## Stochastic Gradient Boosting
##
## 13737 samples
## 52 predictor
## 5 classes: 'A', 'B', 'C', 'D', 'E'
##
## No pre-processing
## Resampling: Cross-Validated (5 fold, repeated 1 times)
## Summary of sample sizes: 10990, 10990, 10989, 10991, 10988
## Resampling results across tuning parameters:
##
## interaction.depth n.trees Accuracy Kappa
## 1 50 0.7521285 0.6858434
## 1 100 0.8227397 0.7756753
## 1 150 0.8521496 0.8129547
## 2 50 0.8563724 0.8180344
## 2 100 0.9059465 0.8809760
## 2 150 0.9302623 0.9117412
## 3 50 0.8969931 0.8695557
## 3 100 0.9398712 0.9238994
## 3 150 0.9593802 0.9486037
##
## Tuning parameter 'shrinkage' was held constant at a value of 0.1
##
## Tuning parameter 'n.minobsinnode' was held constant at a value of 10
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were n.trees = 150, interaction.depth =
## 3, shrinkage = 0.1 and n.minobsinnode = 10.Similarly, we validate and assess the perfomance of the GBM model.
PredGBM <- predict(GBMmod, newdata=testSet)
GBMcm <- confusionMatrix(PredGBM, testSet$classe)
GBMcm## Confusion Matrix and Statistics
##
## Reference
## Prediction A B C D E
## A 1650 26 1 2 1
## B 17 1091 35 3 12
## C 5 20 979 26 11
## D 1 2 11 928 10
## E 1 0 0 5 1048
##
## Overall Statistics
##
## Accuracy : 0.9679
## 95% CI : (0.9631, 0.9722)
## No Information Rate : 0.2845
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.9594
##
## Mcnemar's Test P-Value : 1.749e-05
##
## Statistics by Class:
##
## Class: A Class: B Class: C Class: D Class: E
## Sensitivity 0.9857 0.9579 0.9542 0.9627 0.9686
## Specificity 0.9929 0.9859 0.9872 0.9951 0.9988
## Pos Pred Value 0.9821 0.9421 0.9404 0.9748 0.9943
## Neg Pred Value 0.9943 0.9898 0.9903 0.9927 0.9930
## Prevalence 0.2845 0.1935 0.1743 0.1638 0.1839
## Detection Rate 0.2804 0.1854 0.1664 0.1577 0.1781
## Detection Prevalence 0.2855 0.1968 0.1769 0.1618 0.1791
## Balanced Accuracy 0.9893 0.9719 0.9707 0.9789 0.9837The accuracy rate using random forest is 0.97 which means the out of sample error is 0.03.
Select the Best Model
Comparing the out of sample error of the three models, we select the Random Forest model for prediction of 20 cases of the testing data.
FinalPred <- predict(RFMod, newdata=test)
FinalPred## [1] B A B A A E D B A A B C B A E E A B B B
## Levels: A B C D E