Goal: Learn the concepts, terminology, and techniques in statistical prediction!
We want to know the relationship between covariates (X’s) and response (Y), we can do this with a method called regression. Regression provides us with a statistical method to conduct:
Often, we use a linear model even when we know that the “truth” is not a purely linear relationship with normally-distributed errors.
Often, we use a linear model even when we know that the “truth” is not a purely linear relationship with normally-distributed errors.
Does this mean our model is wrong? Yes, in some sense.
Often, we use a linear model even when we know that the “truth” is not a purely linear relationship with normally-distributed errors.
Does this mean our model is wrong? Yes, in some sense.
Does this mean our model is bad? No, not necessarily!
If we consider a linear model only to be a rough approximation, we can still make use of it for statistical prediction!
Cross-validation refers to a set of methods for measuring the performance of a given predictive model on new test datasets.
The basic idea, behind cross-validation techniques, consists of dividing the data into two sets:
The training set, used to train (i.e. build) the model; and the testing set (or validation set), used to test (i.e. validate) the model by estimating the prediction error.
After building a model, we are interested in determining the accuracy of this model on predicting the outcome for new unseen observations not used to build the model. Put in other words, we want to estimate the prediction error.
To do so, the basic strategy is to:
** Build the model on a training data set
** Apply the model on a new test data set to make predictions
** Compute the prediction errors
## GPA SEX AGE MATH COMP
## 1 2.39 F 22 28 26
## 2 3.37 M 21 23 24
## 3 2.63 F 22 25 21
## 4 2.77 M 21 19 19
## 5 3.79 M 22 28 32
## 6 3.34 M 21 26 23
The example below splits the dataset so that 80% is used for training a linear regression model and 20% is used to evaluate the model performance.
You can use the createDataPartition() function from the caret package in R to partition a data frame into training and testing sets for model building.
## Resample1
## [1,] 2
## [2,] 4
## [3,] 5
## [4,] 7
## [5,] 8
## [6,] 10
## [7,] 11
## [8,] 12
## [9,] 13
## [10,] 14
## [11,] 16
## [12,] 17
## [13,] 18
## [14,] 19
## [15,] 20
## [16,] 21
## [17,] 22
## [18,] 23
## [19,] 25
## [20,] 26
## [21,] 27
## [22,] 28
## [23,] 29
## [24,] 30
## [25,] 31
## [26,] 32
## [27,] 33
## [28,] 36
## [29,] 37
## [30,] 39
## [31,] 40
## [32,] 42
## [33,] 43
## [34,] 44
## [35,] 45
## [36,] 46
## [37,] 47
## [38,] 48
## [39,] 49
## [40,] 50
## [41,] 51
## [42,] 53
## [43,] 54
## [44,] 55
## [45,] 56
## [46,] 57
## [47,] 58
## [48,] 59
## [49,] 61
## [50,] 62
## [51,] 66
## [52,] 67
## [53,] 68
## [54,] 69
## [55,] 71
## [56,] 72
## [57,] 74
## [58,] 75
## [59,] 76
## [60,] 77
## [61,] 78
## [62,] 79
## [63,] 80
## [64,] 81
## [65,] 82
## [66,] 83
## [67,] 84
## [68,] 85
## [69,] 87
## [70,] 88
## [71,] 90
## [72,] 91
## [73,] 92
## [74,] 93
## [75,] 94
## [76,] 95
## [77,] 97
## [78,] 98
## [79,] 99
## [80,] 100
## [81,] 101
## [82,] 102
## [83,] 103
## [84,] 104
## [85,] 105
## [86,] 106
## [87,] 107
## [88,] 108
## [89,] 109
## [90,] 112
## [91,] 113
## [92,] 114
## [93,] 115
## [94,] 116
## [95,] 117
## [96,] 118
## [97,] 119
## [98,] 120
## [99,] 121
## [100,] 122
## [101,] 123
## [102,] 124
## [103,] 125
## [104,] 126
## [105,] 127
## [106,] 128
## [107,] 129
## [108,] 130
## [109,] 132
## [110,] 133
## [111,] 134
## [112,] 136
## [113,] 137
## [114,] 139
## [115,] 140
## [116,] 141
## [117,] 143
## [118,] 145
## [119,] 146
## [120,] 147
## [121,] 150
## [122,] 152
## [123,] 153
## [124,] 154
## [125,] 156
## [126,] 158
## [127,] 159
## [128,] 160
## [129,] 161
## [130,] 162
## [131,] 163
## [132,] 164
## [133,] 165
## [134,] 167
## [135,] 168
## [136,] 170
## [137,] 171
## [138,] 172
## [139,] 174
## [140,] 175
## [141,] 176
## [142,] 179
## [143,] 180
## [144,] 181
## [145,] 183
## [146,] 184
## [147,] 185
## [148,] 186
## [149,] 188
## [150,] 189
## [151,] 191
## [152,] 192
## [153,] 193
## [154,] 194
## [155,] 195
## [156,] 196
## [157,] 197
## [158,] 198
## [159,] 199
## [160,] 200
set.seed(123)
training <- Focus_New$GPA %>%
createDataPartition(p = 0.8, list = FALSE)
train.data <- Focus_New[training, ]
test.data <- Focus_New[-training, ]
head(train.data)## GPA SEX AGE MATH COMP
## 2 3.37 M 21 23 24
## 4 2.77 M 21 19 19
## 5 3.79 M 22 28 32
## 7 2.81 F 20 14 17
## 8 3.09 F 22 23 22
## 10 2.61 F 23 22 20
## GPA SEX AGE MATH COMP
## 1 2.39 F 22 28 26
## 3 2.63 F 22 25 21
## 6 3.34 M 21 26 23
## 9 1.91 M 20 18 20
## 15 3.12 F 21 18 21
## 24 2.10 F 23 25 22
##
## Call:
## lm(formula = GPA ~ ., data = train.data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.37983 -0.31607 0.04557 0.33755 1.15074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.663208 0.433917 3.833 0.000184 ***
## SEXM -0.147544 0.077751 -1.898 0.059602 .
## AGE -0.016347 0.015729 -1.039 0.300274
## MATH 0.006034 0.014117 0.427 0.669655
## COMP 0.065995 0.017793 3.709 0.000289 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4564 on 155 degrees of freedom
## Multiple R-squared: 0.2153, Adjusted R-squared: 0.1951
## F-statistic: 10.63 on 4 and 155 DF, p-value: 1.218e-07
# Make predictions and compute the R2, RMSE and MAE
predictions <- model %>% predict(test.data)
data.frame(R2 = R2(predictions, test.data$GPA),
RMSE = RMSE(predictions, test.data$GPA),
MAE = MAE(predictions, test.data$GPA))## R2 RMSE MAE
## 1 0.2484012 0.4799296 0.392288
set.seed(123)
training1 <- Focus_New$GPA %>%
createDataPartition(p = 0.6, list = F)
train.data1 <- Focus_New[training1, ]
test.data1 <- Focus_New[-training1, ]
training1## Resample1
## [1,] 3
## [2,] 6
## [3,] 13
## [4,] 14
## [5,] 15
## [6,] 16
## [7,] 17
## [8,] 18
## [9,] 19
## [10,] 20
## [11,] 21
## [12,] 22
## [13,] 23
## [14,] 25
## [15,] 26
## [16,] 27
## [17,] 28
## [18,] 29
## [19,] 30
## [20,] 31
## [21,] 33
## [22,] 34
## [23,] 37
## [24,] 38
## [25,] 39
## [26,] 40
## [27,] 42
## [28,] 43
## [29,] 45
## [30,] 47
## [31,] 48
## [32,] 49
## [33,] 50
## [34,] 51
## [35,] 52
## [36,] 54
## [37,] 55
## [38,] 58
## [39,] 59
## [40,] 62
## [41,] 65
## [42,] 66
## [43,] 68
## [44,] 69
## [45,] 70
## [46,] 76
## [47,] 78
## [48,] 79
## [49,] 80
## [50,] 83
## [51,] 84
## [52,] 85
## [53,] 87
## [54,] 88
## [55,] 90
## [56,] 92
## [57,] 94
## [58,] 97
## [59,] 98
## [60,] 100
## [61,] 102
## [62,] 103
## [63,] 104
## [64,] 105
## [65,] 107
## [66,] 109
## [67,] 110
## [68,] 112
## [69,] 113
## [70,] 114
## [71,] 117
## [72,] 118
## [73,] 119
## [74,] 120
## [75,] 122
## [76,] 123
## [77,] 124
## [78,] 126
## [79,] 127
## [80,] 131
## [81,] 133
## [82,] 140
## [83,] 141
## [84,] 142
## [85,] 144
## [86,] 145
## [87,] 148
## [88,] 149
## [89,] 150
## [90,] 152
## [91,] 153
## [92,] 155
## [93,] 156
## [94,] 158
## [95,] 159
## [96,] 160
## [97,] 164
## [98,] 165
## [99,] 167
## [100,] 168
## [101,] 170
## [102,] 171
## [103,] 172
## [104,] 173
## [105,] 174
## [106,] 175
## [107,] 179
## [108,] 180
## [109,] 181
## [110,] 183
## [111,] 184
## [112,] 185
## [113,] 187
## [114,] 191
## [115,] 192
## [116,] 193
## [117,] 195
## [118,] 196
## [119,] 197
## [120,] 199
##
## Call:
## lm(formula = GPA ~ ., data = train.data1)
##
## Residuals:
## Min 1Q Median 3Q Max
## -1.47079 -0.27613 0.04255 0.28418 0.87028
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.57359 0.46272 3.401 0.000925 ***
## SEXM -0.14482 0.08541 -1.696 0.092684 .
## AGE -0.02846 0.01513 -1.882 0.062429 .
## MATH 0.03207 0.01537 2.086 0.039212 *
## COMP 0.05430 0.01907 2.847 0.005222 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4235 on 115 degrees of freedom
## Multiple R-squared: 0.3043, Adjusted R-squared: 0.2801
## F-statistic: 12.58 on 4 and 115 DF, p-value: 1.606e-08
# Make predictions and compute the R2, RMSE and MAE
predictions1 <- model_1 %>% predict(test.data1)
data.frame( R2 = R2(predictions1, test.data1$GPA),
RMSE = RMSE(predictions1, test.data1$GPA),
MAE = MAE(predictions1, test.data1$GPA))## R2 RMSE MAE
## 1 0.1009643 0.5281377 0.4202099
# Make predictions and compute the R2, RMSE and MAE
predictions <- model %>% predict(test.data)
data.frame( R2 = R2(predictions, test.data$GPA),
RMSE = RMSE(predictions, test.data$GPA),
MAE = MAE(predictions, test.data$GPA))## R2 RMSE MAE
## 1 0.2484012 0.4799296 0.392288
predictions1 <- model_1 %>% predict(test.data1)
data.frame( R2 = R2(predictions1, test.data1$GPA),
RMSE = RMSE(predictions1, test.data1$GPA),
MAE = MAE(predictions1, test.data1$GPA))## R2 RMSE MAE
## 1 0.1009643 0.5281377 0.4202099
To evaluate the performance of a model on a dataset, we need to measure how well the predictions made by the model match the observed data.
K-fold cross-validation (CV) is a robust method for estimating the accuracy of a model.
One commonly used method for doing this is known as k-fold cross-validation, which uses the following approach:
Randomly divide a dataset into k groups, or “folds”, of roughly equal size.
Choose one of the folds to be the holdout set. Fit the model on the remaining k-1 folds. Calculate the test MSE on the observations in the fold that was held out.
Repeat this process k times, using a different set each time as the holdout set.
Calculate the overall test MSE to be the average of the k test MSE’s.
The following code shows how to fit a multiple linear regression model to this dataset in R and perform k-fold cross validation with k = 5 folds to evaluate the model performance:
The easiest way to perform k-fold cross-validation in R is by using the trainControl() function from the caret library in R.
#fit a regression model and use k-fold CV to evaluate performance
model <- train(GPA ~ ., data = Focus_New, method = "lm", trControl = ctrl)
print(model)## Linear Regression
##
## 200 samples
## 4 predictor
##
## No pre-processing
## Resampling: Cross-Validated (5 fold)
## Summary of sample sizes: 160, 160, 160, 160, 160
## Resampling results:
##
## RMSE Rsquared MAE
## 0.4585786 0.236229 0.3732109
##
## Tuning parameter 'intercept' was held constant at a value of TRUE
No pre-processing occured. That is, we didn’t scale the data in any way before fitting the models.
The resampling method we used to evaluate the model was cross-validation with 5 folds.
The sample size for each training set was 160.
RMSE: The root mean squared error. This measures the average difference between the predictions made by the model and the actual observations. The lower the RMSE, the more closely a model can predict the actual observations.
Rsquared: This is a measure of the correlation between the predictions made by the model and the actual observations. The higher the R-squared, the more closely a model can predict the actual observations.
The process of splitting the data into k-folds can be repeated a number of times, this is called repeated k-fold cross validation.
The final model error is taken as the mean error from the number of repeats.
The following example uses 5-fold cross validation with 3 repeats:
#fit a regression model and use k-fold CV to evaluate performance
model <- train(GPA ~ ., data = Focus_New, method = "lm", trControl = ctrl)
print(model)## Linear Regression
##
## 200 samples
## 4 predictor
##
## No pre-processing
## Resampling: Cross-Validated (5 fold, repeated 3 times)
## Summary of sample sizes: 160, 160, 160, 160, 160, 160, ...
## Resampling results:
##
## RMSE Rsquared MAE
## 0.46383 0.2053271 0.3768713
##
## Tuning parameter 'intercept' was held constant at a value of TRUE
## admit gre gpa rank id
## 1 0 380 3.61 3 1
## 2 1 660 3.67 3 2
## 3 1 800 4.00 1 3
## 4 1 640 3.19 4 4
## 5 0 520 2.93 4 5
## 6 1 760 3.00 2 6
model <- glm(Binary$admit ~ Binary$gre + Binary$gpa + factor(Binary$rank), family = "binomial")
summary(model)##
## Call:
## glm(formula = Binary$admit ~ Binary$gre + Binary$gpa + factor(Binary$rank),
## family = "binomial")
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.6268 -0.8662 -0.6388 1.1490 2.0790
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -3.989979 1.139951 -3.500 0.000465 ***
## Binary$gre 0.002264 0.001094 2.070 0.038465 *
## Binary$gpa 0.804038 0.331819 2.423 0.015388 *
## factor(Binary$rank)2 -0.675443 0.316490 -2.134 0.032829 *
## factor(Binary$rank)3 -1.340204 0.345306 -3.881 0.000104 ***
## factor(Binary$rank)4 -1.551464 0.417832 -3.713 0.000205 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 499.98 on 399 degrees of freedom
## Residual deviance: 458.52 on 394 degrees of freedom
## AIC: 470.52
##
## Number of Fisher Scoring iterations: 4
set.seed(123)
# Partition data and create index matrix of selected values
index <- createDataPartition(Binary$admit, p=.8, list=FALSE, times=1)As the first argument, type the name of the data frame (df), followed by the $ operator and the name of the outcome variable (Admit).
As a the second argument, set p=.8 to note that you wish to randomly select 80% (.8) of the cases from the data frame.
As the third argument, type list=FALSE to indicate that we want the values in matrix form to facilitate, which will facilitate our ability to reference them in a subsequent step.
As the fourth argument, type times=1 to indicate that we only want to request a single partition of the data.
Binary <- as.data.frame(Binary)
train <- Binary[index,]
test <- Binary[-index,]
# Verify number of rows (cases) in each data frame
nrow(train)## [1] 320
## [1] 80
ctrlspecs <- trainControl(method="cv",
number=10,
savePredictions="all",
classProbs=TRUE)
set.seed(1234)
model_2 <- train(admit ~ gre + gpa + factor(rank), data=train, method="glm", family = "binomial", trControl=ctrlspecs)
summary(model_2)##
## Call:
## NULL
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.6790 -0.8721 -0.6442 1.1154 2.1120
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -4.363439 1.303098 -3.349 0.000812 ***
## gre 0.001809 0.001237 1.462 0.143666
## gpa 1.019065 0.378182 2.695 0.007046 **
## `factor(rank)2` -0.791972 0.355269 -2.229 0.025799 *
## `factor(rank)3` -1.424809 0.390766 -3.646 0.000266 ***
## `factor(rank)4` -1.541866 0.467099 -3.301 0.000964 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 403.57 on 319 degrees of freedom
## Residual deviance: 369.32 on 314 degrees of freedom
## AIC: 381.32
##
## Number of Fisher Scoring iterations: 4
## Generalized Linear Model
##
## 320 samples
## 3 predictor
## 2 classes: 'no', 'yes'
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 288, 289, 288, 287, 287, 288, ...
## Resampling results:
##
## Accuracy Kappa
## 0.7058498 0.2123983
## glm variable importance
##
## Overall
## `factor(rank)3` 100.00
## `factor(rank)4` 84.19
## gpa 56.43
## `factor(rank)2` 35.12
## gre 0.00
predictions <- predict(model_2, newdata=test)
# Create confusion matrix to assess model fit/performance on test data
confusionMatrix(data=predictions, test$admit)## Confusion Matrix and Statistics
##
## Reference
## Prediction no yes
## no 51 19
## yes 6 4
##
## Accuracy : 0.6875
## 95% CI : (0.5741, 0.7865)
## No Information Rate : 0.7125
## P-Value [Acc > NIR] : 0.7350
##
## Kappa : 0.0826
##
## Mcnemar's Test P-Value : 0.0164
##
## Sensitivity : 0.8947
## Specificity : 0.1739
## Pos Pred Value : 0.7286
## Neg Pred Value : 0.4000
## Prevalence : 0.7125
## Detection Rate : 0.6375
## Detection Prevalence : 0.8750
## Balanced Accuracy : 0.5343
##
## 'Positive' Class : no
##
Model Accuracy: We can see that the accuracy is 68.75% (95% CI[0.5741, 0.7865]).
The p-value is a one-sided test that helps us understand if the model is more accurate than a model with no information (for example, predictors); here, we see that that p-value is greater than .05, which implies that the model is not more accurate to a statistically significant extent.
Balanced Accuracy: Balanced accuracy is calculated by adding sensitivity to specificity and then dividing that sum by 2. In this example, the balanced accuracy is approximately 54%, and thus provides another estimate of the model’s overall classification accuracy.