R Markdown

Summary

The goal is to predict how a barbell lift was performed (classe: A-E) from accelerometer data on the belt, forearm, arm and dumbbell of 6 participants. A random forest reached about 99.5% accuracy on a held-out validation set, so I expect an out-of-sample error of about 0.5-1%. Data source: Velloso, E., Bulling, A., Gellersen, H., Ugulino, W., Fuks, H. (2013), Qualitative Activity Recognition of Weight Lifting Exercises, PUC-Rio HAR group (http://web.archive.org/web/20161224072740/http:/groupware.les.inf.puc-rio.br/har).

Data cleaning

url_tr <- "https://d396qusza40orc.cloudfront.net/predmachlearn/pml-training.csv"
url_te <- "https://d396qusza40orc.cloudfront.net/predmachlearn/pml-testing.csv"
train_raw <- read.csv(url_tr, na.strings = c("NA", "", "#DIV/0!"))
test_raw  <- read.csv(url_te, na.strings = c("NA", "", "#DIV/0!"))

keep <- colnames(train_raw)[colSums(is.na(train_raw)) / nrow(train_raw) < 0.9]
train_clean <- train_raw[, keep][, -(1:7)]
train_clean$classe <- factor(train_clean$classe)
dim(train_clean)
## [1] 19622    53

Columns that were more than 90% missing were removed, along with the first seven identifier/timestamp columns (row index, user name, timestamps, windows), which would not generalise to new people. This leaves 52 sensor predictors.

Model and cross-validation

I split the training data 70/30 into a training set and a validation set. A random forest was chosen because it handles many correlated numeric predictors, needs no scaling, and is robust to outliers. caret::train() repeatedly failed in my sandbox environment, so I fitted randomForest directly (100 trees) and did the cross-validation manually.

idx <- createDataPartition(train_clean$classe, p = 0.7, list = FALSE)
trn <- train_clean[idx, ]
val <- train_clean[-idx, ]
fit <- randomForest(classe ~ ., data = trn, ntree = 100)

5-fold cross-validation on the training set (50 trees per fold):

folds <- createFolds(trn$classe, k = 5)
acc <- sapply(folds, function(f) {
  m <- randomForest(classe ~ ., data = trn[-f, ], ntree = 50)
  mean(predict(m, trn[f, ]) == trn$classe[f])
})
round(acc, 4); round(1 - mean(acc), 4)
##  Fold1  Fold2  Fold3  Fold4  Fold5 
## 0.9924 0.9913 0.9894 0.9924 0.9913
## [1] 0.0087

Out-of-sample error

cm <- confusionMatrix(predict(fit, val), val$classe)
cm$overall[c("Accuracy", "Kappa")]
##  Accuracy     Kappa 
## 0.9947324 0.9933367

Validation accuracy was 0.9949 (95% CI 0.9927-0.9966), an estimated out-of-sample error of about 0.5%. The 5-fold CV error was about 0.9%, slightly higher because each fold model used fewer trees and less data. The validation set was never used for training or tuning, so its error is an unbiased estimate. Caveat: participants appear in both training and validation sets, so error on entirely new people may be somewhat higher.

Variable importance

varImpPlot(fit, n.var = 15, main = "Top 15 predictors")

Test predictions

predict(fit, test_raw)
##  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 
##  B  A  B  A  A  E  D  B  A  A  B  C  B  A  E  E  A  B  B  B 
## Levels: A B C D E