Train Test Split

Statistical Prediction

Outline

  1. Training and Testing
  2. Cross-validation

Goal: Learn the concepts, terminology, and techniques in statistical prediction!

Part 1: Training and Testing

Recall: Regression

We want to know the relationship between covariates (X’s) and response (Y), we can do this with a method called regression. Regression provides us with a statistical method to conduct:

Statistical Prediction

Often, we use a linear model even when we know that the “truth” is not a purely linear relationship with normally-distributed errors.

Statistical Prediction

Often, we use a linear model even when we know that the “truth” is not a purely linear relationship with normally-distributed errors.

Does this mean our model is wrong? Yes, in some sense.

Statistical Prediction

Often, we use a linear model even when we know that the “truth” is not a purely linear relationship with normally-distributed errors.

Does this mean our model is wrong? Yes, in some sense.

Does this mean our model is bad? No, not necessarily!

If we consider a linear model only to be a rough approximation, we can still make use of it for statistical prediction!

Regression Model Validation

Model Performance Metrics

Load Packages

 # library(tidyverse), for data manipulation and visualization

 # library(caret), for computing cross-validation methods

Example 1

# Load Data
Focus_New <- Focus %>%
select(GPA, SEX, AGE, MATH, COMP)

head(Focus_New)
##    GPA SEX AGE MATH COMP
## 1 2.39   F  22   28   26
## 2 3.37   M  21   23   24
## 3 2.63   F  22   25   21
## 4 2.77   M  21   19   19
## 5 3.79   M  22   28   32
## 6 3.34   M  21   26   23

Example 1 Continuation

set.seed(123)
training <- Focus_New$GPA %>%
    createDataPartition(p = 0.8, list = FALSE)
training
##        Resample1
##   [1,]         2
##   [2,]         4
##   [3,]         5
##   [4,]         7
##   [5,]         8
##   [6,]        10
##   [7,]        11
##   [8,]        12
##   [9,]        13
##  [10,]        14
##  [11,]        16
##  [12,]        17
##  [13,]        18
##  [14,]        19
##  [15,]        20
##  [16,]        21
##  [17,]        22
##  [18,]        23
##  [19,]        25
##  [20,]        26
##  [21,]        27
##  [22,]        28
##  [23,]        29
##  [24,]        30
##  [25,]        31
##  [26,]        32
##  [27,]        33
##  [28,]        36
##  [29,]        37
##  [30,]        39
##  [31,]        40
##  [32,]        42
##  [33,]        43
##  [34,]        44
##  [35,]        45
##  [36,]        46
##  [37,]        47
##  [38,]        48
##  [39,]        49
##  [40,]        50
##  [41,]        51
##  [42,]        53
##  [43,]        54
##  [44,]        55
##  [45,]        56
##  [46,]        57
##  [47,]        58
##  [48,]        59
##  [49,]        61
##  [50,]        62
##  [51,]        66
##  [52,]        67
##  [53,]        68
##  [54,]        69
##  [55,]        71
##  [56,]        72
##  [57,]        74
##  [58,]        75
##  [59,]        76
##  [60,]        77
##  [61,]        78
##  [62,]        79
##  [63,]        80
##  [64,]        81
##  [65,]        82
##  [66,]        83
##  [67,]        84
##  [68,]        85
##  [69,]        87
##  [70,]        88
##  [71,]        90
##  [72,]        91
##  [73,]        92
##  [74,]        93
##  [75,]        94
##  [76,]        95
##  [77,]        97
##  [78,]        98
##  [79,]        99
##  [80,]       100
##  [81,]       101
##  [82,]       102
##  [83,]       103
##  [84,]       104
##  [85,]       105
##  [86,]       106
##  [87,]       107
##  [88,]       108
##  [89,]       109
##  [90,]       112
##  [91,]       113
##  [92,]       114
##  [93,]       115
##  [94,]       116
##  [95,]       117
##  [96,]       118
##  [97,]       119
##  [98,]       120
##  [99,]       121
## [100,]       122
## [101,]       123
## [102,]       124
## [103,]       125
## [104,]       126
## [105,]       127
## [106,]       128
## [107,]       129
## [108,]       130
## [109,]       132
## [110,]       133
## [111,]       134
## [112,]       136
## [113,]       137
## [114,]       139
## [115,]       140
## [116,]       141
## [117,]       143
## [118,]       145
## [119,]       146
## [120,]       147
## [121,]       150
## [122,]       152
## [123,]       153
## [124,]       154
## [125,]       156
## [126,]       158
## [127,]       159
## [128,]       160
## [129,]       161
## [130,]       162
## [131,]       163
## [132,]       164
## [133,]       165
## [134,]       167
## [135,]       168
## [136,]       170
## [137,]       171
## [138,]       172
## [139,]       174
## [140,]       175
## [141,]       176
## [142,]       179
## [143,]       180
## [144,]       181
## [145,]       183
## [146,]       184
## [147,]       185
## [148,]       186
## [149,]       188
## [150,]       189
## [151,]       191
## [152,]       192
## [153,]       193
## [154,]       194
## [155,]       195
## [156,]       196
## [157,]       197
## [158,]       198
## [159,]       199
## [160,]       200

Example 1 Continuation

set.seed(123)
training <- Focus_New$GPA %>%
    createDataPartition(p = 0.8, list = FALSE)
train.data  <- Focus_New[training, ]
test.data <- Focus_New[-training, ]
head(train.data)
##     GPA SEX AGE MATH COMP
## 2  3.37   M  21   23   24
## 4  2.77   M  21   19   19
## 5  3.79   M  22   28   32
## 7  2.81   F  20   14   17
## 8  3.09   F  22   23   22
## 10 2.61   F  23   22   20
head(test.data)
##     GPA SEX AGE MATH COMP
## 1  2.39   F  22   28   26
## 3  2.63   F  22   25   21
## 6  3.34   M  21   26   23
## 9  1.91   M  20   18   20
## 15 3.12   F  21   18   21
## 24 2.10   F  23   25   22

Example 1 Continuation

# Build the model
model <- lm(GPA ~., data = train.data)
summary(model)
## 
## Call:
## lm(formula = GPA ~ ., data = train.data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.37983 -0.31607  0.04557  0.33755  1.15074 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  1.663208   0.433917   3.833 0.000184 ***
## SEXM        -0.147544   0.077751  -1.898 0.059602 .  
## AGE         -0.016347   0.015729  -1.039 0.300274    
## MATH         0.006034   0.014117   0.427 0.669655    
## COMP         0.065995   0.017793   3.709 0.000289 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.4564 on 155 degrees of freedom
## Multiple R-squared:  0.2153, Adjusted R-squared:  0.1951 
## F-statistic: 10.63 on 4 and 155 DF,  p-value: 1.218e-07

Example 1 Continuation

# Make predictions and compute the R2, RMSE and MAE
predictions <- model %>% predict(test.data)
data.frame(R2 = R2(predictions, test.data$GPA),
            RMSE = RMSE(predictions, test.data$GPA),
            MAE = MAE(predictions, test.data$GPA))
##          R2      RMSE      MAE
## 1 0.2484012 0.4799296 0.392288

Example 1 (New)

set.seed(123)
training1 <- Focus_New$GPA %>%
    createDataPartition(p = 0.6, list = F)
train.data1  <- Focus_New[training1, ]
test.data1 <- Focus_New[-training1, ]
training1
##        Resample1
##   [1,]         3
##   [2,]         6
##   [3,]        13
##   [4,]        14
##   [5,]        15
##   [6,]        16
##   [7,]        17
##   [8,]        18
##   [9,]        19
##  [10,]        20
##  [11,]        21
##  [12,]        22
##  [13,]        23
##  [14,]        25
##  [15,]        26
##  [16,]        27
##  [17,]        28
##  [18,]        29
##  [19,]        30
##  [20,]        31
##  [21,]        33
##  [22,]        34
##  [23,]        37
##  [24,]        38
##  [25,]        39
##  [26,]        40
##  [27,]        42
##  [28,]        43
##  [29,]        45
##  [30,]        47
##  [31,]        48
##  [32,]        49
##  [33,]        50
##  [34,]        51
##  [35,]        52
##  [36,]        54
##  [37,]        55
##  [38,]        58
##  [39,]        59
##  [40,]        62
##  [41,]        65
##  [42,]        66
##  [43,]        68
##  [44,]        69
##  [45,]        70
##  [46,]        76
##  [47,]        78
##  [48,]        79
##  [49,]        80
##  [50,]        83
##  [51,]        84
##  [52,]        85
##  [53,]        87
##  [54,]        88
##  [55,]        90
##  [56,]        92
##  [57,]        94
##  [58,]        97
##  [59,]        98
##  [60,]       100
##  [61,]       102
##  [62,]       103
##  [63,]       104
##  [64,]       105
##  [65,]       107
##  [66,]       109
##  [67,]       110
##  [68,]       112
##  [69,]       113
##  [70,]       114
##  [71,]       117
##  [72,]       118
##  [73,]       119
##  [74,]       120
##  [75,]       122
##  [76,]       123
##  [77,]       124
##  [78,]       126
##  [79,]       127
##  [80,]       131
##  [81,]       133
##  [82,]       140
##  [83,]       141
##  [84,]       142
##  [85,]       144
##  [86,]       145
##  [87,]       148
##  [88,]       149
##  [89,]       150
##  [90,]       152
##  [91,]       153
##  [92,]       155
##  [93,]       156
##  [94,]       158
##  [95,]       159
##  [96,]       160
##  [97,]       164
##  [98,]       165
##  [99,]       167
## [100,]       168
## [101,]       170
## [102,]       171
## [103,]       172
## [104,]       173
## [105,]       174
## [106,]       175
## [107,]       179
## [108,]       180
## [109,]       181
## [110,]       183
## [111,]       184
## [112,]       185
## [113,]       187
## [114,]       191
## [115,]       192
## [116,]       193
## [117,]       195
## [118,]       196
## [119,]       197
## [120,]       199

Example 1 (New)

# Build the model
model_1 <- lm(GPA ~., data = train.data1)
summary(model_1)
## 
## Call:
## lm(formula = GPA ~ ., data = train.data1)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.47079 -0.27613  0.04255  0.28418  0.87028 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  1.57359    0.46272   3.401 0.000925 ***
## SEXM        -0.14482    0.08541  -1.696 0.092684 .  
## AGE         -0.02846    0.01513  -1.882 0.062429 .  
## MATH         0.03207    0.01537   2.086 0.039212 *  
## COMP         0.05430    0.01907   2.847 0.005222 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.4235 on 115 degrees of freedom
## Multiple R-squared:  0.3043, Adjusted R-squared:  0.2801 
## F-statistic: 12.58 on 4 and 115 DF,  p-value: 1.606e-08

Example 1 (New)

# Make predictions and compute the R2, RMSE and MAE
predictions1 <- model_1 %>% predict(test.data1)
data.frame( R2 = R2(predictions1, test.data1$GPA),
            RMSE = RMSE(predictions1, test.data1$GPA),
            MAE = MAE(predictions1, test.data1$GPA))
##          R2      RMSE       MAE
## 1 0.1009643 0.5281377 0.4202099

Example 1 (New)

# Make predictions and compute the R2, RMSE and MAE
predictions <- model %>% predict(test.data)
data.frame( R2 = R2(predictions, test.data$GPA),
            RMSE = RMSE(predictions, test.data$GPA),
            MAE = MAE(predictions, test.data$GPA))
##          R2      RMSE      MAE
## 1 0.2484012 0.4799296 0.392288
predictions1 <- model_1 %>% predict(test.data1)
data.frame( R2 = R2(predictions1, test.data1$GPA),
            RMSE = RMSE(predictions1, test.data1$GPA),
            MAE = MAE(predictions1, test.data1$GPA))
##          R2      RMSE       MAE
## 1 0.1009643 0.5281377 0.4202099

Example 1 (New)

Cross-validation

K-Fold Cross Validation

K-Fold Cross Validation in R (Step-by-Step)

One commonly used method for doing this is known as k-fold cross-validation, which uses the following approach:

Example

Example

# library(caret)
# specify the cross-validation method
ctrl <- trainControl(method = "cv", number = 5)

Example

#fit a regression model and use k-fold CV to evaluate performance
model <- train(GPA ~ ., data = Focus_New, method = "lm", trControl = ctrl)
print(model)
## Linear Regression 
## 
## 200 samples
##   4 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (5 fold) 
## Summary of sample sizes: 160, 160, 160, 160, 160 
## Resampling results:
## 
##   RMSE       Rsquared  MAE      
##   0.4585786  0.236229  0.3732109
## 
## Tuning parameter 'intercept' was held constant at a value of TRUE

Interpretations:

Interpretations (cont’d):

Interpretations (cont’d):

Example (New Method)

# library(caret)
# specify the cross-validation method
ctrl <- trainControl(method = "repeatedcv", number = 5,  repeats = 3)

Example (New Method)

#fit a regression model and use k-fold CV to evaluate performance
model <- train(GPA ~ ., data = Focus_New, method = "lm", trControl = ctrl)
print(model)
## Linear Regression 
## 
## 200 samples
##   4 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (5 fold, repeated 3 times) 
## Summary of sample sizes: 160, 160, 160, 160, 160, 160, ... 
## Resampling results:
## 
##   RMSE     Rsquared   MAE      
##   0.46383  0.2053271  0.3768713
## 
## Tuning parameter 'intercept' was held constant at a value of TRUE

Applying k-Fold Cross-Validation to Logistic Regression

Recall

head(Binary)  
##   admit gre  gpa rank id
## 1     0 380 3.61    3  1
## 2     1 660 3.67    3  2
## 3     1 800 4.00    1  3
## 4     1 640 3.19    4  4
## 5     0 520 2.93    4  5
## 6     1 760 3.00    2  6

Recall

model <- glm(Binary$admit ~ Binary$gre + Binary$gpa + factor(Binary$rank),  family = "binomial")
summary(model)
## 
## Call:
## glm(formula = Binary$admit ~ Binary$gre + Binary$gpa + factor(Binary$rank), 
##     family = "binomial")
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.6268  -0.8662  -0.6388   1.1490   2.0790  
## 
## Coefficients:
##                       Estimate Std. Error z value Pr(>|z|)    
## (Intercept)          -3.989979   1.139951  -3.500 0.000465 ***
## Binary$gre            0.002264   0.001094   2.070 0.038465 *  
## Binary$gpa            0.804038   0.331819   2.423 0.015388 *  
## factor(Binary$rank)2 -0.675443   0.316490  -2.134 0.032829 *  
## factor(Binary$rank)3 -1.340204   0.345306  -3.881 0.000104 ***
## factor(Binary$rank)4 -1.551464   0.417832  -3.713 0.000205 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 499.98  on 399  degrees of freedom
## Residual deviance: 458.52  on 394  degrees of freedom
## AIC: 470.52
## 
## Number of Fisher Scoring iterations: 4

Logistic Regression

set.seed(123)
# Partition data and create index matrix of selected values
index <- createDataPartition(Binary$admit, p=.8, list=FALSE, times=1)

Create test and train data frames

Binary <- as.data.frame(Binary)

train <- Binary[index,]
test <- Binary[-index,]


# Verify number of rows (cases) in each data frame
nrow(train)
## [1] 320
nrow(test)
## [1] 80

Re-label values of outcome variable for train and test

train$admit[train$admit==1] <- "yes"
train$admit[train$admit==0] <- "no"

test$admit[test$admit==1] <- "yes"
test$admit[test$admit==0] <- "no"


# Convert outcome variable to factor for each data frame
train$admit <- as.factor(train$admit)
test$admit <- as.factor(test$admit)

Model

ctrlspecs <- trainControl(method="cv", 
                          number=10, 
                          savePredictions="all",
                          classProbs=TRUE)

set.seed(1234)
model_2 <- train(admit ~ gre + gpa + factor(rank),  data=train, method="glm", family = "binomial",  trControl=ctrlspecs)
summary(model_2)
## 
## Call:
## NULL
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.6790  -0.8721  -0.6442   1.1154   2.1120  
## 
## Coefficients:
##                  Estimate Std. Error z value Pr(>|z|)    
## (Intercept)     -4.363439   1.303098  -3.349 0.000812 ***
## gre              0.001809   0.001237   1.462 0.143666    
## gpa              1.019065   0.378182   2.695 0.007046 ** 
## `factor(rank)2` -0.791972   0.355269  -2.229 0.025799 *  
## `factor(rank)3` -1.424809   0.390766  -3.646 0.000266 ***
## `factor(rank)4` -1.541866   0.467099  -3.301 0.000964 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 403.57  on 319  degrees of freedom
## Residual deviance: 369.32  on 314  degrees of freedom
## AIC: 381.32
## 
## Number of Fisher Scoring iterations: 4

Model

print(model_2)
## Generalized Linear Model 
## 
## 320 samples
##   3 predictor
##   2 classes: 'no', 'yes' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 288, 289, 288, 287, 287, 288, ... 
## Resampling results:
## 
##   Accuracy   Kappa    
##   0.7058498  0.2123983

Estimate the importance of different predictors

varImp(model_2)
## glm variable importance
## 
##                 Overall
## `factor(rank)3`  100.00
## `factor(rank)4`   84.19
## gpa               56.43
## `factor(rank)2`   35.12
## gre                0.00

Predict outcome using model from training data based on testing data

predictions <- predict(model_2, newdata=test)

# Create confusion matrix to assess model fit/performance on test data
confusionMatrix(data=predictions, test$admit)
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction no yes
##        no  51  19
##        yes  6   4
##                                           
##                Accuracy : 0.6875          
##                  95% CI : (0.5741, 0.7865)
##     No Information Rate : 0.7125          
##     P-Value [Acc > NIR] : 0.7350          
##                                           
##                   Kappa : 0.0826          
##                                           
##  Mcnemar's Test P-Value : 0.0164          
##                                           
##             Sensitivity : 0.8947          
##             Specificity : 0.1739          
##          Pos Pred Value : 0.7286          
##          Neg Pred Value : 0.4000          
##              Prevalence : 0.7125          
##          Detection Rate : 0.6375          
##    Detection Prevalence : 0.8750          
##       Balanced Accuracy : 0.5343          
##                                           
##        'Positive' Class : no              
##