Final Project Report - Practical Machine Learning
By Sue Lynn
Background Introduction
Using devices such as Jawbone Up, Nike FuelBand, and Fitbit it is now possible to collect a large amount of data about personal activity relatively inexpensively. These type of devices are part of the quantified self movement - a group of enthusiasts who take measurements about themselves regularly to improve their health, to find patterns in their behavior, or because they are tech geeks. One thing that people regularly do is quantify how much of a particular activity they do, but they rarely quantify how well they do it. In this project, your goal will be to use data from accelerometers on the belt, forearm, arm, and dumbell of 6 participants. They were asked to perform barbell lifts correctly and incorrectly in 5 different ways. More information is available from the website here: http://groupware.les.inf.puc-rio.br/har (see the section on the Weight Lifting Exercise Dataset).
The goal of your project is to predict the manner in which they did the exercise
In order to reproduce the same results, you need a certain set of packages, as well as setting a pseudo random seed equal to the one I used. *Note:To install, for instance, the caret package in R, run this command: install.packages(“caret”)
The following Libraries were used for this project, which you should install - if not done yet - and load on your working environment.
library(caret)## Warning: package 'caret' was built under R version 3.2.3## Loading required package: lattice
## Loading required package: ggplot2## Warning: package 'ggplot2' was built under R version 3.2.3library(randomForest)## Warning: package 'randomForest' was built under R version 3.2.3## randomForest 4.6-12
## Type rfNews() to see new features/changes/bug fixes.
##
## Attaching package: 'randomForest'
##
## The following object is masked from 'package:ggplot2':
##
## marginlibrary(ggthemes)## Warning: package 'ggthemes' was built under R version 3.2.3library(gridExtra)## Warning: package 'gridExtra' was built under R version 3.2.3library(ggplot2)
library(grid)Getting The Data
Downloading the training data set and the testing the data set into the hard drive.
train = read.csv("C:\\Users\\sue\\Documents\\R\\pml-training.csv",header=TRUE)
train_used = train[,c(8:11,37:49,60:68,84:86,102,113:124,140,151:160)]
testing = read.csv("C:\\Users\\sue\\Documents\\R\\pml-testing.csv",header=TRUE)
test_used = testing[,c(8:11,37:49,60:68,84:86,102,113:124,140,151:160)]The raw dataset contained 19622 rows of data, with 160 variables. The clearning of the data was done by removing the many variables that contained a large missing data (usually with only one row of data), so these were removed from the dataset. In addition, variables not concerning the movement sensors were also removed. This resulted in a dataset of 53 variables.
dim(train)## [1] 19622 160dim(train_used)## [1] 19622 53Partioning The Training Set Into Two
The dataset was partitioned into training and testing datasets, with 60% of the original data going to the training set and 40% to the testing set. The model was built with the training dataset, then tested on the testing dataset. The following code performs this procedure:
train_part = createDataPartition(train_used$classe, p = 0.6, list = FALSE)
myTraining = train_used[train_part, ]
myTesting = train_used[-train_part, ]
dim(myTraining); dim(myTesting)## [1] 11776 53## [1] 7846 53The Model
Many methods of classification were attempted, including niave Bayes, multinomial logistic regression, and decision trees. It was determined that the Random Forest method produced the best results. In addition, principal component analysis was attempted however this greatly reduced the prediction accuracy.
Cross validation was not used, as, according to the creators of the Random Forest algorithm: “In random forests, there is no need for cross-validation or a separate test set to get an unbiased estimate of the test set error.” - Leo Breiman and Adele Cutler
The R code is shown below, as is the confusion matrix. The OOB error rate in the training and the confusion matrix is shown below. For informational purposes a plot of the error rate versus number of trees is also shown.
set.seed(1777)
random_forest=randomForest(classe~.,data=myTraining,ntree=500,importance=TRUE)
random_forest##
## Call:
## randomForest(formula = classe ~ ., data = myTraining, ntree = 500, importance = TRUE)
## Type of random forest: classification
## Number of trees: 500
## No. of variables tried at each split: 7
##
## OOB estimate of error rate: 0.74%
## Confusion matrix:
## A B C D E class.error
## A 3344 3 0 0 1 0.001194743
## B 20 2250 9 0 0 0.012724879
## C 0 17 2035 2 0 0.009250243
## D 0 0 24 1903 3 0.013989637
## E 0 1 2 5 2157 0.003695150plot(random_forest,main="Random Forest: Error Rate vs Number of Trees")
Variable Importance
It may be of interest to know which variables were most ‘important’ in the building of the model. This can be seen by plotting the mean decrease in accuracy and the mean decrease in the gini coefficient per variable. In short, The more the accuracy of the random forest decreases due to the exclusion (or permutation) of a single variable, the more important that variable is deemed to be. The mean decrease in Gini coefficient is a measure of how each variable contributes to the homogeneity of the nodes and leaves in the resulting random forest. (from https://dinsdalelab.sdsu.edu/metag.stats/code/randomforest.html)
imp=importance(random_forest)
impL=imp[,c(6,7)]
imp.ma=as.matrix(impL)
imp.df=data.frame(imp.ma)
write.csv(imp.df, "imp.df.csv", row.names=TRUE)
imp.df.csv=read.csv("imp.df.csv",header=TRUE)
colnames(imp.df.csv)=c("Variable","MeanDecreaseAccuracy","MeanDecreaseGini")
imp.sort = imp.df.csv[order(-imp.df.csv$MeanDecreaseAccuracy),]
imp.sort = transform(imp.df.csv,
Variable = reorder(Variable, MeanDecreaseAccuracy))
VIP=ggplot(data=imp.sort, aes(x=Variable, y=MeanDecreaseAccuracy)) +
ylab("Mean Decrease Accuracy")+xlab("")+
geom_bar(stat="identity",fill="skyblue",alpha=.8,width=.75)+
coord_flip()+theme_few()
imp.sort.Gini <- transform(imp.df.csv,
Variable = reorder(Variable, MeanDecreaseGini))
VIP.Gini=ggplot(data=imp.sort.Gini, aes(x=Variable, y=MeanDecreaseGini)) +
ylab("Mean Decrease Gini")+xlab("")+
geom_bar(stat="identity",fill="skyblue",alpha=.8,width=.75)+
coord_flip()+theme_few()
VarImpPlot=arrangeGrob(VIP, VIP.Gini,ncol=2)
grid.draw(VarImpPlot)
predictionTesting = predict(random_forest, newdata= myTesting)
confusionMatrix(predictionTesting, myTesting$classe)## Confusion Matrix and Statistics
##
## Reference
## Prediction A B C D E
## A 2232 10 0 0 0
## B 0 1506 8 0 0
## C 0 2 1359 14 1
## D 0 0 1 1270 3
## E 0 0 0 2 1438
##
## Overall Statistics
##
## Accuracy : 0.9948
## 95% CI : (0.9929, 0.9962)
## No Information Rate : 0.2845
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.9934
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: A Class: B Class: C Class: D Class: E
## Sensitivity 1.0000 0.9921 0.9934 0.9876 0.9972
## Specificity 0.9982 0.9987 0.9974 0.9994 0.9997
## Pos Pred Value 0.9955 0.9947 0.9876 0.9969 0.9986
## Neg Pred Value 1.0000 0.9981 0.9986 0.9976 0.9994
## Prevalence 0.2845 0.1935 0.1744 0.1639 0.1838
## Detection Rate 0.2845 0.1919 0.1732 0.1619 0.1833
## Detection Prevalence 0.2858 0.1930 0.1754 0.1624 0.1835
## Balanced Accuracy 0.9991 0.9954 0.9954 0.9935 0.9985Model Applied to Testing Dataset
predictionTesting = predict(random_forest, newdata= myTesting)
confusionMatrix(predictionTesting, myTesting$classe)## Confusion Matrix and Statistics
##
## Reference
## Prediction A B C D E
## A 2232 10 0 0 0
## B 0 1506 8 0 0
## C 0 2 1360 14 1
## D 0 0 0 1270 3
## E 0 0 0 2 1438
##
## Overall Statistics
##
## Accuracy : 0.9949
## 95% CI : (0.9931, 0.9964)
## No Information Rate : 0.2845
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.9936
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: A Class: B Class: C Class: D Class: E
## Sensitivity 1.0000 0.9921 0.9942 0.9876 0.9972
## Specificity 0.9982 0.9987 0.9974 0.9995 0.9997
## Pos Pred Value 0.9955 0.9947 0.9877 0.9976 0.9986
## Neg Pred Value 1.0000 0.9981 0.9988 0.9976 0.9994
## Prevalence 0.2845 0.1935 0.1744 0.1639 0.1838
## Detection Rate 0.2845 0.1919 0.1733 0.1619 0.1833
## Detection Prevalence 0.2858 0.1930 0.1755 0.1622 0.1835
## Balanced Accuracy 0.9991 0.9954 0.9958 0.9936 0.9985