Cross Valid. & Model Selection, Ta-Feng

• 本章重點: 介紹何謂交叉驗證(cross validation)
• 將交叉驗證之結果應用在我們的預測性模型上

交叉驗證與參數調校流程

Fig-1: Supervised Learning Process

Fig-2: CV, Model Sel. & Parameter Tuning

---

Preparing Data

Libraries

Sys.setlocale("LC_ALL","C")

[1] "C"

library(dplyr)
library(ggplot2)
library(caTools)
library(Matrix)
library(rpart)
library(rpart.plot)
library(caret)
library(doParallel)

Loading and Spliting

rm(list=ls(all=TRUE))
load("data/tf2.rdata")
A$buy = factor(ifelse(A$buy, "yes", "no"))  # comply to the rule of caret
TR = A[spl, c(2:9,11)]
TS = A[!spl, c(2:9,11)]

Turn on Parallel Processing

clust = makeCluster(detectCores())
registerDoParallel(clust); getDoParWorkers()

[1] 4

• 開啟平行運算，將電腦的每一個CPU都叫出來工作，以免執行交叉驗證的等待時間過長
• 可以看到自己的電腦有幾顆CPU

決策樹之交叉驗證

CV Control for Classification

ctrl = trainControl(
  method="repeatedcv", number=10,    # 10-fold, Repeated CV
  savePredictions = "final", classProbs=TRUE,
  summaryFunction=twoClassSummary)

• 設定交叉驗證要將原本的資料切成幾塊(執行幾次)

CV: rpart(), Classification Tree

ctrl$repeats = 2
t0 = Sys.time(); set.seed(2)
cv.rpart = train(
  buy ~ ., data=TR, method="rpart", 
  trControl=ctrl, metric="ROC",
  tuneGrid = expand.grid(cp = seq(0.0002,0.001,0.0001) ) )
Sys.time() - t0

Time difference of 1.117289 mins

plot(cv.rpart)

cv.rpart$results 

• 如同影片所說，複雜度越高不一定越“準”，因此透過參數調校，找出最適合的複雜度和參數組合

Classification Tree, Final Model

rpart1 = rpart(buy ~ ., TR, method="class", cp=0.0005)
predict(rpart1, TS, type="prob")[,2] %>% 
  colAUC(TS$buy)

                [,1]
no vs. yes 0.7401771

---

CV: glm(), General Linear Model(邏輯式回歸)

ctrl$repeats = 2
t0 = Sys.time(); set.seed(2)
cv.glm = train(
  buy ~ ., data=TR, method="glm", 
  trControl=ctrl, metric="ROC")
Sys.time() - t0

Time difference of 22.44225 secs

cv.glm$results

glm(), Final Model

glm1 = b=glm(buy ~ ., TR, family=binomial)
predict(glm1, TS, type="response") %>% colAUC(TS$buy)

                [,1]
no vs. yes 0.7556038

• 執行完CV後，決策樹之AUC上升至0.7556038

---

線性迴歸之交叉驗證

Spliting Data

A2 = subset(A, A$buy == "yes") %>% mutate_at(c("m","rev","amount"), log10)
TR2 = A2[ spl2, c(2:10)]
TS2 = A2[!spl2, c(2:10)]

CV Control for Regression

ctrl2 = trainControl(
  method="repeatedcv", number=10,    # 10-fold, Repeated CV
  savePredictions = "final")

• 同樣將資料切為10等分

CV: rpart() Regression Tree

ctrl$repeats = 2
set.seed(2)
cv.rpart2 = train(
  amount ~ ., data=TR2, method="rpart", 
  trControl=ctrl2, metric="Rsquared",
  tuneGrid = expand.grid(cp = seq(0.0008,0.0024,0.0001) ) )
plot(cv.rpart2)

• 透過參數調校找出最佳複雜度

cv.rpart2$results

rpart(), Regression Tree Final Model

rpart2 = rpart(amount ~ ., data=TR2, cp=0.0016)
SST = sum((TS2$amount - mean(TR2$amount))^ 2)
SSE = sum((predict(rpart2, TS2) -  TS2$amount)^2)
(r2.ts.rpart2 = 1 - (SSE/SST))

[1] 0.2174555

CV: lm(), Linear Model

ctrl$repeats = 2
set.seed(2)
cv.lm2 = train(
  amount ~ ., data=TR2, method="lm", 
  trControl=ctrl2, metric="Rsquared",
    tuneGrid = expand.grid( intercept = seq(0,5,0.5) ) 
  )
plot(cv.lm2)

cv.lm2$results

lm() Final Model

lm2 = lm(amount ~ ., TR2)
SST = sum((TS2$amount - mean(TR2$amount))^ 2)
SSE = sum((predict(lm2, TS2) -  TS2$amount)^2)
(r2.ts.lm2 = 1 - (SSE/SST))

[1] 0.2381007

• 線性迴歸做完交叉驗證後之R^2為0.2381007

要記得關閉平行運算功能喔!

stopCluster(clust)

---