Practical Machine Learning Quiz 4
Sherifat Akintunde-Shitu
27 November 2018
Week four Quiz
Question 1.
Load the vowel.train and vowel.test data sets:
library(ElemStatLearn)
data(vowel.train)
data(vowel.test)Set the variable y to be a factor variable in both the training and test set. Then set the seed to 33833. Fit (1) a random forest predictor relating the factor variable y to the remaining variables and (2) a boosted predictor using the “gbm” method. Fit these both with the c.
What are the accuracies for the two approaches on the test data set? What is the accuracy among the test set samples where the two methods agree?
#Set the variable y to be a factor variable in both the training and test set. Then set the seed to 33833.
vowel.train$y <- factor(vowel.train$y)
vowel.test$y <- factor(vowel.test$y)
set.seed(33833)
##USing (2) a boosted predictor using the "gbm" method.
#(1) a random forest predictor relating the factor variable y to the remaining variables
vfit1<-train(y~.,vowel.train, method="rf")
#(2) a boosted predictor using the "gbm" method.
vfit2 <- train(y ~ ., vowel.train, method = "gbm", verbose = FALSE)
#compute accuracy on test data fot fit 1: random forest
predfit1 <- predict(vfit1, newdata = vowel.test)
confusionMatrix(predfit1, vowel.test$y) ## Confusion Matrix and Statistics
##
## Reference
## Prediction 1 2 3 4 5 6 7 8 9 10 11
## 1 30 0 0 0 0 0 0 0 0 4 0
## 2 12 24 3 0 0 0 2 0 2 15 1
## 3 0 14 34 3 0 1 0 0 0 0 2
## 4 0 0 1 29 1 0 0 0 0 0 2
## 5 0 0 0 0 18 7 9 0 0 0 0
## 6 0 0 2 9 19 25 2 0 0 0 7
## 7 0 0 0 0 3 0 27 7 5 0 3
## 8 0 0 0 0 0 0 0 29 6 1 0
## 9 0 4 0 0 0 0 1 6 23 3 13
## 10 0 0 0 0 0 0 1 0 2 19 0
## 11 0 0 2 1 1 9 0 0 4 0 14
##
## Overall Statistics
##
## Accuracy : 0.5887
## 95% CI : (0.5423, 0.634)
## No Information Rate : 0.0909
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.5476
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity 0.71429 0.57143 0.80952 0.69048 0.42857 0.59524
## Specificity 0.99048 0.91667 0.95238 0.99048 0.96190 0.90714
## Pos Pred Value 0.88235 0.40678 0.62963 0.87879 0.52941 0.39062
## Neg Pred Value 0.97196 0.95533 0.98039 0.96970 0.94393 0.95729
## Prevalence 0.09091 0.09091 0.09091 0.09091 0.09091 0.09091
## Detection Rate 0.06494 0.05195 0.07359 0.06277 0.03896 0.05411
## Detection Prevalence 0.07359 0.12771 0.11688 0.07143 0.07359 0.13853
## Balanced Accuracy 0.85238 0.74405 0.88095 0.84048 0.69524 0.75119
## Class: 7 Class: 8 Class: 9 Class: 10 Class: 11
## Sensitivity 0.64286 0.69048 0.54762 0.45238 0.33333
## Specificity 0.95714 0.98333 0.93571 0.99286 0.95952
## Pos Pred Value 0.60000 0.80556 0.46000 0.86364 0.45161
## Neg Pred Value 0.96403 0.96948 0.95388 0.94773 0.93503
## Prevalence 0.09091 0.09091 0.09091 0.09091 0.09091
## Detection Rate 0.05844 0.06277 0.04978 0.04113 0.03030
## Detection Prevalence 0.09740 0.07792 0.10823 0.04762 0.06710
## Balanced Accuracy 0.80000 0.83690 0.74167 0.72262 0.64643#compute accuracy on test data fot fit 2: gbm
predfit2 <- predict(vfit2, newdata = vowel.test)
confusionMatrix(predfit2, vowel.test$y)## Confusion Matrix and Statistics
##
## Reference
## Prediction 1 2 3 4 5 6 7 8 9 10 11
## 1 29 1 0 0 0 0 0 0 0 2 0
## 2 9 18 1 0 0 0 1 0 0 13 0
## 3 2 11 9 4 0 0 0 0 0 0 0
## 4 0 1 7 20 3 0 0 0 0 0 0
## 5 0 0 0 1 17 5 3 0 0 0 0
## 6 0 3 16 16 15 29 0 0 1 0 15
## 7 0 0 1 0 4 1 36 8 6 0 11
## 8 0 0 0 0 0 0 2 29 8 2 0
## 9 0 4 0 0 0 0 0 5 27 6 12
## 10 2 0 0 0 0 0 0 0 0 19 0
## 11 0 4 8 1 3 7 0 0 0 0 4
##
## Overall Statistics
##
## Accuracy : 0.513
## 95% CI : (0.4664, 0.5594)
## No Information Rate : 0.0909
## P-Value [Acc > NIR] : < 2.2e-16
##
## Kappa : 0.4643
## Mcnemar's Test P-Value : NA
##
## Statistics by Class:
##
## Class: 1 Class: 2 Class: 3 Class: 4 Class: 5 Class: 6
## Sensitivity 0.69048 0.42857 0.21429 0.47619 0.40476 0.69048
## Specificity 0.99286 0.94286 0.95952 0.97381 0.97857 0.84286
## Pos Pred Value 0.90625 0.42857 0.34615 0.64516 0.65385 0.30526
## Neg Pred Value 0.96977 0.94286 0.92431 0.94896 0.94266 0.96458
## Prevalence 0.09091 0.09091 0.09091 0.09091 0.09091 0.09091
## Detection Rate 0.06277 0.03896 0.01948 0.04329 0.03680 0.06277
## Detection Prevalence 0.06926 0.09091 0.05628 0.06710 0.05628 0.20563
## Balanced Accuracy 0.84167 0.68571 0.58690 0.72500 0.69167 0.76667
## Class: 7 Class: 8 Class: 9 Class: 10 Class: 11
## Sensitivity 0.85714 0.69048 0.64286 0.45238 0.095238
## Specificity 0.92619 0.97143 0.93571 0.99524 0.945238
## Pos Pred Value 0.53731 0.70732 0.50000 0.90476 0.148148
## Neg Pred Value 0.98481 0.96912 0.96324 0.94785 0.912644
## Prevalence 0.09091 0.09091 0.09091 0.09091 0.090909
## Detection Rate 0.07792 0.06277 0.05844 0.04113 0.008658
## Detection Prevalence 0.14502 0.08874 0.11688 0.04545 0.058442
## Balanced Accuracy 0.89167 0.83095 0.78929 0.72381 0.520238#find accuracy where the two models agree
fitagreed <- (predfit1 == predfit2)
confusionMatrix(vowel.test$y[fitagreed], predfit1[fitagreed])$overall['Accuracy']## Accuracy
## 0.621118Fit1(RF) Accuracy = 0.6082 Fit2(GBM) Accuracy = 0.5152 Agreement Accuracy = 0.6361
Question 2.
Load the Alzheimer’s data using the following commands
library(caret)
library(gbm)## Loaded gbm 2.1.4set.seed(3433)
library(AppliedPredictiveModeling)
data(AlzheimerDisease)
adData = data.frame(diagnosis,predictors)
inTrain = createDataPartition(adData$diagnosis, p = 3/4)[[1]]
training = adData[ inTrain,]
testing = adData[-inTrain,]Set the seed to 62433 and predict diagnosis with all the other variables using a random forest (“rf”), boosted trees (“gbm”) and linear discriminant analysis (“lda”) model. Stack the predictions together using random forests (“rf”). What is the resulting accuracy on the test set? Is it better or worse than each of the individual predictions?
#Set the seed to 62433
set.seed(62433)
#predict diagnosis with all the other variables using a random forest ("rf"),
ADfit1 = train(diagnosis~.,training, method="rf")
fit1pred <- predict(ADfit1, testing)
confusionMatrix(fit1pred, testing$diagnosis)$overall['Accuracy']## Accuracy
## 0.7926829#predict diagnosis with all the other variables using boosted trees ("gbm")
ADfit2 = train(diagnosis~.,training, method="gbm", verbose=FALSE)
fit2pred <- predict(ADfit2, testing)
confusionMatrix(fit1pred, testing$diagnosis)$overall['Accuracy']## Accuracy
## 0.7926829#predict diagnosis with all the other variables using linear discriminant analysis ("lda") mode
ADfit3 = train(diagnosis~.,training, method="lda")
fit3pred <- predict(ADfit3, testing)
confusionMatrix(fit3pred, testing$diagnosis)$overall['Accuracy']## Accuracy
## 0.7682927#combine prediction models
combine <- data.frame(fit1pred, fit2pred, fit3pred, diagnosis = testing$diagnosis)
stackfit <- train(diagnosis ~ ., combine, method = "rf")## note: only 2 unique complexity parameters in default grid. Truncating the grid to 2 .stackfitpred <- predict(stackfit, testing)
confusionMatrix(testing$diagnosis, stackfitpred)$overall['Accuracy']## Accuracy
## 0.804878- Stacked Accuracy: 0.80 is better than all three other methods
- Stacked Accuracy: 0.80 is better than random forests and lda and the same as boosting.
- Stacked Accuracy: 0.80 is worse than all the other methods.
- Stacked Accuracy: 0.88 is better than all three other methods
Question 3.
Load the concrete data with the commands:
set.seed(3523)
data(concrete)
inTrain = createDataPartition(concrete$CompressiveStrength, p = 3/4)[[1]]
training = concrete[ inTrain,]
testing = concrete[-inTrain,]Set the seed to 233 and fit a lasso model to predict Compressive Strength. Which variable is the last coefficient to be set to zero as the penalty increases? (Hint: it may be useful to look up ?plot.enet).
#Set the seed to 233
set.seed(233)
#fit a lasso model to predict Compressive Strength.
confit<- train(CompressiveStrength ~ ., training, method = "lasso")
library(lars); library(elasticnet)## Loaded lars 1.2?plot.enet## starting httpd help server ...## doneplot.enet(confit$finalModel, xvar="penalty", use.color=TRUE)
- CoarseAggregate
- Age
- Cement
- Water
Question 4.
Load the data on the number of visitors to the instructors blog from here:
[https://d396qusza40orc.cloudfront.net/predmachlearn/gaData.csv]https://d396qusza40orc.cloudfront.net/predmachlearn/gaData.csv
Using the commands:
library(lubridate) # For year() function below##
## Attaching package: 'lubridate'## The following object is masked from 'package:reshape':
##
## stamp## The following object is masked from 'package:base':
##
## date#load data
url <- "https://d396qusza40orc.cloudfront.net/predmachlearn/gaData.csv"
download.file(url, destfile = "gaData.csv")
dat = read.csv("gaData.csv")
#create test and training sets
training = dat[year(dat$date) < 2012,]
testing = dat[(year(dat$date)) > 2011,]
tstrain = ts(training$visitsTumblr)Fit a model using the bats() function in the forecast package to the training time series. Then forecast this model for the remaining time points. For how many of the testing points is the true value within the 95% prediction interval bounds?
library(forecast)
gaFit <- bats(tstrain)
predGaFit <- forecast(gaFit, level = 95, nrow(testing))
fslower95 <- predGaFit$lower
fsupper95 <- predGaFit$upper
table ((testing$visitsTumblr>fslower95) & (testing$visitsTumblr<fsupper95))##
## FALSE TRUE
## 9 226226/nrow(testing)## [1] 0.9617021- 100%
- 96%
- 94%
- 92%
Question 5.
Load the concrete data with the commands:
set.seed(3523)
library(AppliedPredictiveModeling)
data(concrete)
inTrain = createDataPartition(concrete$CompressiveStrength, p = 3/4)[[1]]
training = concrete[ inTrain,]
testing = concrete[-inTrain,]Set the seed to 325 and fit a support vector machine using the e1071 package to predict Compressive Strength using the default settings. Predict on the testing set. What is the RMSE?
#Set the seed to 325
set.seed(325)
#fit a support vector machine using the e1071 package to predict Compressive Strength using the default settings.
library(e1071)##
## Attaching package: 'e1071'## The following object is masked from 'package:Hmisc':
##
## imputefit <- svm(CompressiveStrength ~ ., training)
#Predict on the testing set.
predsvm <- predict(fit, testing)
#What is the RMSE?
error = predsvm - testing$CompressiveStrength
mse <- sqrt(mean(error^2))
mse## [1] 6.715009- 6.93
- 11543.39
- 6.72
- 107.44