Background

Using devices such as Jawbone Up, Nike FuelBand, and Fitbit, it is now possible to collect data on personal activity relatively inexpensively. In this project, data from accelerometers on the belt, forearm, arm, and dumbbell of 6 participants is used to predict how well they performed barbell lifts (the classe variable: A = correct form, B–E = four common mistakes).

Data Loading and Cleaning

train_raw <- read.csv("pml-training.csv", na.strings = c("NA", "", "#DIV/0!"))
test_raw  <- read.csv("pml-testing.csv",  na.strings = c("NA", "", "#DIV/0!"))
dim(train_raw)
## [1] 19622   160

The raw data has 160 columns. Two groups are removed before modeling:

  1. Identifier / bookkeeping columns — X, user_name, the raw and converted timestamps, new_window, num_window. These describe who and when, not how the movement was performed, so keeping them would let the model “cheat” (e.g. by subject or by time order) instead of learning generalizable movement patterns.
  2. Columns that are >90% NA — summary statistics (kurtosis_*, skewness_*, avg_*, stddev_*, var_*, min_*, max_*, amplitude_*) that are only populated on the rare new_window == "yes" rows. With ~98% missingness they carry almost no usable signal and would otherwise force heavy imputation.
id_cols <- c("X", "user_name", "raw_timestamp_part_1", "raw_timestamp_part_2",
             "cvtd_timestamp", "new_window", "num_window")

na_frac <- sapply(train_raw, function(x) mean(is.na(x)))
mostly_na <- names(na_frac[na_frac > 0.9])

drop_cols <- unique(c(id_cols, mostly_na))
feature_cols <- setdiff(names(train_raw), c(drop_cols, "classe"))
length(feature_cols)  # 52 predictors remain
## [1] 52
train_clean <- train_raw[, c(feature_cols, "classe")]
train_clean$classe <- as.factor(train_clean$classe)
test_clean  <- test_raw[, feature_cols]

This leaves 52 numeric sensor features (roll/pitch/yaw, accelerometer, gyroscope, and magnetometer readings for the belt, arm, forearm, and dumbbell) with no missing values.

Cross-Validation Strategy

The 19,622-row training set is split 70/30 into a training and a held-out validation set. Within the 70% training portion, 5-fold cross-validation is used during model fitting (via caret::trainControl) to tune and get a stable internal accuracy estimate. The held-out 30% is used purely to get an honest, independent estimate of out-of-sample accuracy before touching the 20 official test cases.

inTrain <- createDataPartition(train_clean$classe, p = 0.7, list = FALSE)
training   <- train_clean[inTrain, ]
validation <- train_clean[-inTrain, ]
dim(training); dim(validation)
## [1] 13737    53
## [1] 5885   53

Model Building

A random forest is used: it handles the mix of correlated sensor features well, needs little preprocessing, and typically performs strongly on this type of multi-class movement-classification problem.

ctrl <- trainControl(method = "cv", number = 5, allowParallel = TRUE)

rf_model <- train(classe ~ ., data = training,
                   method = "rf",
                   trControl = ctrl,
                   ntree = 300,
                   importance = TRUE)
rf_model
## Random Forest 
## 
## 13737 samples
##    52 predictor
##     5 classes: 'A', 'B', 'C', 'D', 'E' 
## 
## No pre-processing
## Resampling: Cross-Validated (5 fold) 
## Summary of sample sizes: 10990, 10990, 10990, 10989, 10989 
## Resampling results across tuning parameters:
## 
##   mtry  Accuracy   Kappa    
##    2    0.9906823  0.9882120
##   27    0.9910460  0.9886725
##   52    0.9841303  0.9799211
## 
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was mtry = 27.

Model Evaluation

val_pred <- predict(rf_model, validation)
cm <- confusionMatrix(val_pred, validation$classe)
cm
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    A    B    C    D    E
##          A 1672    0    0    0    0
##          B    1 1136    4    0    0
##          C    1    3 1020    9    0
##          D    0    0    2  954    5
##          E    0    0    0    1 1077
## 
## Overall Statistics
##                                           
##                Accuracy : 0.9956          
##                  95% CI : (0.9935, 0.9971)
##     No Information Rate : 0.2845          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.9944          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: A Class: B Class: C Class: D Class: E
## Sensitivity            0.9988   0.9974   0.9942   0.9896   0.9954
## Specificity            1.0000   0.9989   0.9973   0.9986   0.9998
## Pos Pred Value         1.0000   0.9956   0.9874   0.9927   0.9991
## Neg Pred Value         0.9995   0.9994   0.9988   0.9980   0.9990
## Prevalence             0.2845   0.1935   0.1743   0.1638   0.1839
## Detection Rate         0.2841   0.1930   0.1733   0.1621   0.1830
## Detection Prevalence   0.2841   0.1939   0.1755   0.1633   0.1832
## Balanced Accuracy      0.9994   0.9982   0.9957   0.9941   0.9976

Expected out-of-sample error is estimated from the held-out validation set (data the model never saw during training or CV tuning):

oos_error <- 1 - cm$overall["Accuracy"]
oos_error
##    Accuracy 
## 0.004418012

The optimal tuning parameter, chosen by 5-fold CV accuracy among the candidates tried, was mtry =27`. The 5-fold cross-validation accuracy was 99.1%, while the independent held-out validation accuracy was 99.56%. Both indicate very high predictive performance, with the held-out validation estimate corresponding to an estimated out-of-sample error of approximately 0.44%. These numbers are computed directly from the fitted objects above, so they will always match whatever this document actually knits to.

varImpPlot(rf_model$finalModel, n.var = 15, main = "Top 15 Predictors")

roll_belt, yaw_belt, pitch_forearm, and the dumbbell magnetometer readings are the strongest predictors — sensible, since belt orientation and forearm/dumbbell motion differ most between correct lifts and the specific mistakes each class represents.

cm_df <- as.data.frame(cm$table)
ggplot2::ggplot(cm_df, ggplot2::aes(Prediction, Reference, fill = Freq)) +
  ggplot2::geom_tile() +
  ggplot2::geom_text(ggplot2::aes(label = Freq), color = "white") +
  ggplot2::scale_fill_gradient(low = "steelblue", high = "darkblue") +
  ggplot2::labs(title = "Confusion Matrix (Validation Set)") +
  ggplot2::theme_minimal()

Why This Approach

Predicting the 20 Test Cases

# Refit on the FULL training set (more data -> better final model) for
# the 20 unlabeled test cases used in the quiz.
final_model <- train(classe ~ ., data = train_clean,
                      method = "rf",
                      trControl = trainControl(method = "cv", number = 5),
                      ntree = 300)

final_predictions <- predict(final_model, test_clean)
data.frame(problem_id = test_raw$problem_id, prediction = final_predictions)
##    problem_id prediction
## 1           1          B
## 2           2          A
## 3           3          B
## 4           4          A
## 5           5          A
## 6           6          E
## 7           7          D
## 8           8          B
## 9           9          A
## 10         10          A
## 11         11          B
## 12         12          C
## 13         13          B
## 14         14          A
## 15         15          E
## 16         16          E
## 17         17          A
## 18         18          B
## 19         19          B
## 20         20          B
# Generates one .txt file per problem_id, in the format expected by the
# Course Project Prediction Quiz submission script.
pml_write_files <- function(x) {
  n <- length(x)
  for (i in 1:n) {
    filename <- paste0("problem_id_", i, ".txt")
    write.table(x[i], file = filename, quote = FALSE,
                row.names = FALSE, col.names = FALSE)
  }
}
pml_write_files(as.character(final_predictions))