AS4-2 Letter Recognition

1.

1.1 Predicting B or not B

letters = read.csv("data/letters_ABPR.csv")
letters$isB = as.factor(letters$letter == "B")
library(caTools)
set.seed(1000)
spl = sample.split(letters$isB,SplitRatio = 0.5)
Train = subset(letters,spl=="TRUE")
Test = subset(letters,spl=="FALSE")
table(Test$isB)

FALSE  TRUE 
 1175   383 

1175/nrow(Test)

[1] 0.7542

1.2 Predicting B or not B

What is the accuracy of the CART model on the test set? (Use type=“class” when making predictions on the test set.)

library(rpart)
CARTb = rpart(isB ~ . - letter, data=Train, method="class")
Pred = predict(CARTb , Test ,type = "class")
table(Test$isB,Pred)

       Pred
        FALSE TRUE
  FALSE  1118   57
  TRUE     43  340

(1118+340)/nrow(Test)

[1] 0.9358

1.3 Predicting B or not B

Now, build a random forest model to predict whether the letter is a B or not (the isB variable) using the training set. You should use all of the other variables as independent variables, except letter (since it helped us define what we are trying to predict!). Use the default settings for ntree and nodesize (don’t include these arguments at all). Right before building the model, set the seed to 1000. (NOTE: You might get a slightly different answer on this problem, even if you set the random seed. This has to do with your operating system and the implementation of the random forest algorithm.)

What is the accuracy of the model on the test set?

library(randomForest)
set.seed(1000)
randomForestb = randomForest(isB ~ . - letter, data=Train)
Pred2 = predict(randomForestb,Test)
table(Test$isB,Pred2)

       Pred2
        FALSE TRUE
  FALSE  1163   12
  TRUE      9  374

(1163+374)/nrow(Test)

[1] 0.9865

2.

2.1 Predicting the letters A, B, P, R

In a multiclass classification problem, a simple baseline model is to predict the most frequent class of all of the options.

What is the baseline accuracy on the testing set?

letters$letter = as.factor( letters$letter )
set.seed(2000)
spl = sample.split(letters$letter,SplitRatio = 0.5)
Train2 = subset(letters,spl=="TRUE")
Test2 = subset(letters,spl=="FALSE")
table(Test2$letter)

  A   B   P   R 
395 383 401 379 

401/nrow(Test2)

[1] 0.2574

2.2 Predicting the letters A, B, P, R

What is the test set accuracy of your CART model? Use the argument type=“class” when making predictions.

CART = rpart(letter ~ . - isB, data=Train, method="class")
Pred3 = predict(CART,Test2,type="class")
table(Test2$letter,Pred3)

   Pred3
      A   B   P   R
  A 349   1   7  38
  B   9 304   8  62
  P   3  28 359  11
  R   8  26   0 345

(349+304+359+345)/nrow(Test2)

[1] 0.871

2.3 Predicting the letters A, B, P, R

Now build a random forest model on the training data, using the same independent variables as in the previous problem – again, don’t forget to remove the isB variable. Just use the default parameter values for ntree and nodesize (you don’t need to include these arguments at all). Set the seed to 1000 right before building your model. (Remember that you might get a slightly different result even if you set the random seed.)

What is the test set accuracy of your random forest model?

library(randomForest)
set.seed(1000)
randomForest = randomForest(letter ~ . - isB,data=Train)
Pred4 = predict(randomForest,Test2,type="class")
table(Test2$letter,Pred4)

   Pred4
      A   B   P   R
  A 392   0   1   2
  B   0 378   0   5
  P   0   2 398   1
  R   0   6   0 373

(392+378+398+373)/nrow(Test2)

[1] 0.9891