R Notebook

Problem 1.1 - Predicting B or not B

Before building models, let’s consider a baseline method that always predicts the most frequent outcome, which is “not B”. What is the accuracy of this baseline method on the test set?

library(caTools)
letters = read.csv("data/letters_ABPR.csv")
letters$isB = as.factor(letters$letter == "B")
set.seed(1000)
spl = sample.split(letters$isB, SplitRatio = 0.5)
train = subset(letters, spl == TRUE)
test = subset(letters, spl == FALSE)
table(train$isB) 

FALSE  TRUE 
 1175   383 

table(test$isB)

FALSE  TRUE 
 1175   383 

1175/(1175+383)

[1] 0.754172

Problem 1.2 - Predicting B or not B

What is the accuracy of the CART model on the test set? (Use type=“class” when making predictions on the test set.)

CARTb = rpart(isB ~ . - letter, data=train, method="class")
predictions = predict(CARTb, newdata=test, type="class")
table(test$isB, predictions)

       predictions
        FALSE TRUE
  FALSE  1118   57
  TRUE     43  340

(1118+340)/nrow(test)

[1] 0.9358151

Problem 1.3 - Predicting B or Not B

What is the accuracy of the model on the test set?

(1163+375)/nrow(test)

[1] 0.987163

Problem 2.1 - Predicting the letters A, B, P, R

What is the baseline accuracy on the testing set?

set.seed(2000)

Warning message:
In strsplit(code, "\n", fixed = TRUE) :
  input string 1 is invalid in this locale

spl = sample.split(letters$letter, SplitRatio = 0.5)
train2 = subset(letters, spl == TRUE)
test2 = subset(letters, spl == FALSE)
table(train2$letter)

  A   B   P   R 
394 383 402 379 

table(test2$letter) 

  A   B   P   R 
395 383 401 379 

401/nrow(test)

[1] 0.2573813

###以預測有最多的observations的P為主，這樣Accuracy才會是最高，蘇然是50-50分，但A、P是奇數，所以數字會不同。"

Problem 2.2 - Predicting the letters A, B, P, R

CARTletter = rpart(letter ~ . - isB, data=train2, method="class")

Warning message:
In strsplit(code, "\n", fixed = TRUE) :
  input string 1 is invalid in this locale

predictLetter = predict(CARTletter, newdata=test2, type="class")
table(test2$letter, predictLetter)

   predictLetter
      A   B   P   R
  A 348   4   0  43
  B   8 318  12  45
  P   2  21 363  15
  R  10  24   5 340

(348+318+363+340)/nrow(test)

[1] 0.8786906

###可以用nrow(test)去除的原因是，他代表data frame所有觀察值的數量，另外，table裡面的數字，348、318，代表的是數量。"

Problem 2.3 - Predicting the letters A, B, P, R

What is the test set accuracy of your random forest model?

set.seed(1000)
RFletter = randomForest(letter ~ . - isB, data=train2)
predictLetter = predict(RFletter, newdata=test2)
table(test2$letter, predictLetter)

   predictLetter
      A   B   P   R
  A 391   0   3   1
  B   0 380   1   2
  P   0   6 394   1
  R   3  14   0 362

(391+380+394+362)/nrow(test2)

[1] 0.9801027