AS4-2 字母辨認 (temp.)

主要議題：依圖形屬性辨認不同字型的英文字母

學習重點：

• 多類別決策樹模型
• 隨機森林的預測能力
• 決策樹 vs 隨機森林

rm(list=ls(all=T))
options(digits=4, scipen=12)
library(dplyr)
library(rpart)
library(rpart.plot)
library(caret)
library(randomForest)
library(caTools)
source('DPP.R')

---

1. 兩類別模型

1.1 兩類別-底線模型

Let’s warm up by attempting to predict just whether a letter is B or not. To begin, load the file letters_ABPR.csv into R, and call it letters. Then, create a new variable isB in the dataframe, which takes the value “TRUE” if the observation corresponds to the letter B, and “FALSE” if it does not. You can do this by typing the following command into your R console:

letters\(isB = as.factor(letters\)letter == “B”)

Now split the data set into a training and testing set, putting 50% of the data in the training set. Set the seed to 1000 before making the split. The first argument to sample.split should be the dependent variable “letters$isB”. Remember that TRUE values from sample.split should go in the training set.

Before building models, let’s consider a baseline method that always predicts the most frequent outcome, which is “not B”. What is the accuracy of this baseline method on the test set?

D =  read.csv('data/letters_ABPR.csv')
D$isB = factor(D$letter=="B")
set.seed(1000)
spl = sample.split(D$isB, SplitRatio = 0.5)
TR = subset(D, spl)
TS = subset(D, !spl)
1 - mean(TS$isB == "TRUE")

[1] 0.7542

1.2 兩類別-決策樹模型

Now build a classification tree to predict whether a letter is a B or not, using the training set to build your model. Remember to remove the variable “letter” out of the model, as this is related to what we are trying to predict! To just remove one variable, you can either write out the other variables, or remember what we did in the Billboards problem in Week 3, and use the following notation:

m1 = rpart(isB ~ . - letter, TR, method='class')
prp(m1)

We are just using the default parameters in our CART model, so we don’t need to add the minbucket or cp arguments at all. We also added the argument method=“class” since this is a classification problem.

What is the accuracy of the CART model on the test set? (Use type=“class” when making predictions on the test set.)

pred = predict(m1, TS, type="class")
table(TS$isB, pred) %>% {sum(diag(.))/sum(.)}

[1] 0.9358

1.3 兩類別-隨機森林模型

Now, build a random forest model to predict whether the letter is a B or not (the isB variable) using the training set. You should use all of the other variables as independent variables, except letter (since it helped us define what we are trying to predict!). Use the default settings for ntree and nodesize (don’t include these arguments at all). Right before building the model, set the seed to 1000. (NOTE: You might get a slightly different answer on this problem, even if you set the random seed. This has to do with your operating system and the implementation of the random forest algorithm.)

What is the accuracy of the model on the test set?

set.seed(1000)
m2 = randomForest(isB ~ . - letter, TR)       # default settings
pred = predict(m2, TS, 'class')
table(TS$isB, pred) %>% {sum(diag(.))/sum(.)} # 

[1] 0.9865

---

2. 多類別模型

2.1 多類別-底線模型

Let us now move on to the problem that we were originally interested in, which is to predict whether or not a letter is one of the four letters A, B, P or R.

As we saw in the D2Hawkeye lecture, building a multiclass classification CART model in R is no harder than building the models for binary classification problems. Fortunately, building a random forest model is just as easy.

The variable in our data frame which we will be trying to predict is “letter”. Start by converting letter in the original data set (letters) to a factor by running the following command in R:

D$letter = factor(D$letter)

Now, generate new training and testing sets of the letters data frame using letters$letter as the first input to the sample.split function. Before splitting, set your seed to 2000. Again put 50% of the data in the training set. (Why do we need to split the data again? Remember that sample.split balances the outcome variable in the training and testing sets. With a new outcome variable, we want to re-generate our split.)

set.seed(1000)
spl = sample.split(D$letter, SplitRatio = 0.5)
TR = subset(D, spl)
TS = subset(D, !spl)

In a multiclass classification problem, a simple baseline model is to predict the most frequent class of all of the options.

What is the baseline accuracy on the testing set?

table(TS$letter) %>% prop.table %>% sort

     R      B      A      P 
0.2433 0.2458 0.2535 0.2574 

2.2 多類別-決策樹模型

Now build a classification tree to predict “letter”, using the training set to build your model. You should use all of the other variables as independent variables, except “isB”, since it is related to what we are trying to predict! Just use the default parameters in your CART model. Add the argument method=“class” since this is a classification problem. Even though we have multiple classes here, nothing changes in how we build the model from the binary case.

What is the test set accuracy of your CART model? Use the argument type=“class” when making predictions.

(HINT: When you are computing the test set accuracy using the confusion matrix, you want to add everything on the main diagonal and divide by the total number of observations in the test set, which can be computed with nrow(test), where test is the name of your test set).

m3 = rpart(letter ~ . - isB, TR, method='class')
prp(m3)

rpart.plot(m3)

pred = predict(m3, TS, type="class")
table(TS$letter, pred) %>% {sum(diag(.))/sum(.)}

[1] 0.878

2.3 多類別-隨機森林模型

set.seed(1000)
m4 = randomForest(letter ~ . - isB, TR)       
pred = predict(m4, TS, 'class')
table(TS$letter, pred) %>% {sum(diag(.))/sum(.)} 

[1] 0.982

---