Question 1

For this quiz we will be using several R packages. R package versions change over time, the right answers have been checked using the following versions of the packages.
AppliedPredictiveModeling: v1.1.6
caret: v6.0.47 used:6.0.76
ElemStatLearn: v2012.04-0 used: 2015.6.26
pgmm: v1.1 used: 1.2
rpart: v4.1.8 used: 4.1.10
gbm: v2.1 used:2.1.3
lubridate: v1.3.3 used: 1.6.0
forecast: v5.6 used:8.0
e1071: v1.6.4 used:1.6-8
If you aren’t using these versions of the packages, your answers may not exactly match the right answer, but hopefully should be close.
Load the vowel.train and vowel.test data sets:
library(ElemStatLearn)
library(caret)
data(vowel.train)
data(vowel.test)
Set the variable y to be a factor variable in both the training and test set.
vowel.train$y<-as.factor(vowel.train$y)
vowel.test$y<-as.factor(vowel.test$y)
Then set the seed to 33833.
set.seed(33833)
Fit (1) a random forest predictor relating the factor variable y to the remaining variables and
model_rf<-train(y ~., data=vowel.train, method="rf")
pred_rf<-predict(model_rf,vowel.test)
(2) a boosted predictor using the “gbm” method.
model_gbm<-train(y ~., data=vowel.train, method="gbm")
pred_gbm<-predict(model_gbm,vowel.test)
Fit these both with the train() command in the caret package.
What are the accuracies for the two approaches on the test data set? What is the accuracy among the test set samples where the two methods agree?
confusionMatrix(pred_rf, vowel.test$y)$overall['Accuracy']
##  Accuracy 
## 0.6147186
confusionMatrix(pred_gbm, vowel.test$y)$overall['Accuracy']
##  Accuracy 
## 0.5367965
DataBoth<-(pred_rf==pred_gbm)

confusionMatrix(pred_rf[DataBoth], vowel.test$y[DataBoth])$overall['Accuracy']
##  Accuracy 
## 0.6656051
  • RF Accuracy = 0.6082 realy 0.61
  • GBM Accuracy = 0.5152 realy 0.54

  • Agreement Accuracy = 0.6361 realy 0.66

  • RF Accuracy = 0.6082
  • GBM Accuracy = 0.5152

  • Agreement Accuracy = 0.5325

  • RF Accuracy = 0.3233
  • GBM Accuracy = 0.8371

  • Agreement Accuracy = 0.9983

  • RF Accuracy = 0.9987
  • GBM Accuracy = 0.5152

  • Agreement Accuracy = 0.9985

Question 2

Load the Alzheimer’s data using the following commands
library(caret)
library(gbm)
set.seed(3433)
library(AppliedPredictiveModeling)
data(AlzheimerDisease)
adData = data.frame(diagnosis,predictors)
inTrain = createDataPartition(adData$diagnosis, p = 3/4)[[1]]
training = adData[ inTrain,]
testing = adData[-inTrain,]
Set the seed to 62433 and predict diagnosis with all the other variables using a random forest (“rf”), boosted trees (“gbm”) and linear discriminant analysis (“lda”) model. Stack the predictions together using random forests (“rf”).
set.seed(62433)
mod_rf<-train(diagnosis ~., data=training, method="rf")  
mod_gbm<-train(diagnosis ~., data=training, method="gbm")  
mod_lda<-train(diagnosis ~., data=training, method="lda")  
## Warning in lda.default(x, grouping, ...): variables are collinear

## Warning in lda.default(x, grouping, ...): variables are collinear
pred_rf<-predict(mod_rf,testing)  
pred_gbm<-predict(mod_gbm,testing)  
pred_lda<-predict(mod_lda,testing)
predDF<-data.frame(pred_rf, pred_gbm, pred_lda, diagnosis=testing$diagnosis)
combModFit<-train(diagnosis ~., method="rf", data=predDF)  
combPred<-predict(combModFit, predDF)
The Accuracy obtaines are the following:
confusionMatrix(pred_rf,testing$diagnosis)$overall['Accuracy']  
##  Accuracy 
## 0.7682927
confusionMatrix(pred_gbm,testing$diagnosis)$overall['Accuracy']  
##  Accuracy 
## 0.7926829
confusionMatrix(pred_lda,testing$diagnosis)$overall['Accuracy']  
##  Accuracy 
## 0.7682927
confusionMatrix(combPred,testing$diagnosis)$overall['Accuracy']
## Accuracy 
## 0.804878
What is the resulting accuracy on the test set? Is it better or worse than each of the individual predictions?
  • Stacked Accuracy: 0.76 is better than random forests and boosting, but not lda.
  • Stacked Accuracy: 0.93 is better than all three other methods
  • Stacked Accuracy: 0.88 is better than all three other methods
  • Stacked Accuracy: 0.80 is better than random forests and lda and the same as boosting.

Question 3

Load the concrete data with the commands:
set.seed(3523)
library(AppliedPredictiveModeling)
data(concrete)
inTrain = createDataPartition(concrete$CompressiveStrength, p = 3/4)[[1]]
training = concrete[ inTrain,]
testing = concrete[-inTrain,]
Set the seed to 233 and fit a lasso model to predict Compressive Strength.
set.seed(233)  
mod_lasso<-train(CompressiveStrength ~., data=training, method="lasso")
library(elasticnet)
plot.enet(mod_lasso$finalModel, xvar="penalty", use.color=TRUE)

Which variable is the last coefficient to be set to zero as the penalty increases? (Hint: it may be useful to look up ?plot.enet).
  • BlastFurnaceSlag
  • Cement
  • FineAggregate
  • CoarseAggregate

Question 4

Load the data on the number of visitors to the instructors blog from here:
Using the commands:
library(lubridate) # For year() function below
dat = read.csv("gaData.csv")
training = dat[year(dat$date) < 2012,]
testing = dat[(year(dat$date)) > 2011,]
tstrain = ts(training$visitsTumblr)
Fit a model using the bats() function in the forecast package to the training time series. Then forecast this model for the remaining time points.
library(forecast)
mod_ts<-bats(tstrain)

#  level -> confidence level for prediction intervals
#  h -> number of periods for forecasting
#  fcast receives the Model that is the first [1] occurence of the list of 10 returned form forecast

fcast<-forecast(mod_ts, level=95, h=dim(testing) [1])

plot(fcast)

sum(fcast$lower < testing$visitsTumblr & testing$visitsTumblr < fcast$upper) / dim(testing)[1]
## [1] 0.9617021
For how many of the testing points is the true value within the 95% prediction interval bounds?
  • 94%
  • 96%
  • 93%
  • 95%

Question 5

Load the concrete data with the commands:
set.seed(3523)
library(AppliedPredictiveModeling)
library(caret)
library(forecast)
library(e1071)  

data(concrete)  

inTrain = createDataPartition(concrete$CompressiveStrength, p = 3/4)[[1]]
training = concrete[ inTrain,]
testing = concrete[-inTrain,]
Set the seed to 325 and fit a support vector machine using the e1071 package to predict Compressive Strength using the default settings. Predict on the testing set.
set.seed(325)

mod_svm<-svm(CompressiveStrength ~.,data=training)  

pred_svm<-predict(mod_svm, testing)  

##library forecast provides accuracy for an object of class"forecast" or a numerical **vector** containing forecasts

accuracy(pred_svm, testing$CompressiveStrength)
##                 ME     RMSE      MAE       MPE     MAPE
## Test set 0.1682863 6.715009 5.120835 -7.102348 19.27739
What is the RMSE?
  • 6.72 realy 6.71
  • 6.93
  • 11543.39
  • 107.44
Thanks for reading!