AS4-2 Letter Recognition

Section-1

§ 1.1 What is the accuracy of this baseline method on the test set?

set.seed(1000)
letters = letters_ABPR
letters$isB = as.factor(letters$letter == "B")
split = sample.split(letters$isB, SplitRatio = 0.5)
train = subset(letters, split == TRUE)
test = subset(letters, split == FALSE)
table(test$isB)

FALSE  TRUE 
 1175   383 

1175/(1175+383)

[1] 0.7542

§ 1.2 What is the accuracy of the CART model on the test set?

CARTb = rpart(isB ~ . - letter, data=train, method="class")
predb = predict(CARTb, newdata = test, type = "class")
table(test$isB, predb)

       predb
        FALSE TRUE
  FALSE  1118   57
  TRUE     43  340

(1118+340)/nrow(test)

[1] 0.9358

§ 1.3 What is the accuracy of the model on the test set?

set.seed(1000)
forest1 = randomForest(isB ~ . - letter, data=train)
predbforest = predict(forest1, newdata = test)
table(test$isB, predbforest)

       predbforest
        FALSE TRUE
  FALSE  1163   12
  TRUE      9  374

(1163+374)/nrow(test)

[1] 0.9865

Section-2

§ 2.1 What is the baseline accuracy on the testing set?

set.seed(2000)
letters$letter = as.factor( letters$letter)
split = sample.split(letters$letter, SplitRatio = 0.5)
train = subset(letters, split == TRUE)
test = subset(letters, split == FALSE)
table(test$letter)

  A   B   P   R 
395 383 401 379 

401/nrow(test)

[1] 0.2574

§ 2.2 What is the test set accuracy of your CART model?

CART1 = rpart(letter ~ . - isB, data=train, method="class")
pred1 = predict(CART1, newdata = test, type = "class")
table(test$letter, pred1)

   pred1
      A   B   P   R
  A 348   4   0  43
  B   8 318  12  45
  P   2  21 363  15
  R  10  24   5 340

(348+318+363+340)/nrow(test)

[1] 0.8787

§ 2.3 What is the test set accuracy of your random forest model?

set.seed(1000)
forest2 = randomForest(letter ~ . - isB, data=train)
predbforest1 = predict(forest2, newdata = test)
table(test$letter, predbforest1)

   predbforest1
      A   B   P   R
  A 391   0   3   1
  B   0 380   1   2
  P   0   6 394   1
  R   3  14   0 362

(391+380+394+362)/nrow(test)

[1] 0.9801