LOOCV

Leave One Out CV (LOOCV):

Each observation is considered as a validation set and the rest n-1 observations are a training set. Fit the model and predict using 1 observation validation set. Repeat this for n times for each observation as a validation set.

Test-error rate is average of all n errors.

Advantages: takes care of both drawbacks of validation-set method

(1) No randomness of using some observations for training vs. validation set like in validation-set method as each observation is considered for both training and validation. So overall less variability than Validation-set method due to no randomness no matter how many times you run it. (it does not mean LOOCV does not have variability, see disadvantage no.1)

(2) Less bias than validation-set method as training-set is of n-1 size. Almost entire data set. Because of this reduced bias, reduced over-estimation of test-error, not as much compared to validation-set method.

Disadvantages:

(1) Even though each iterations test-error is un-biased, it has a high variability as only one-observation validation-set was used for prediction.

(2) Computationally expensive (time and power) especially if dataset is big with large n as it requires fitting the model n times. Also some statistical models have computationally intensive fitting so with large dataset and these models LOOCV might not be a good choice.

here we will fit linear regreesion model but using glm() function without passing family type. because later it can be used together with cv.glm() [ part of boot library ].

suppressMessages(library(ISLR))

data("Auto")
dim(Auto)

## [1] 392   9

library(boot)

## 
## Attaching package: 'boot'

## The following object is masked from 'package:survival':
## 
##     aml

## The following object is masked from 'package:lattice':
## 
##     melanoma

glm.fit = glm(mpg ~ horsepower, data=Auto)

summary(glm.fit)

## 
## Call:
## glm(formula = mpg ~ horsepower, data = Auto)
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -13.5710   -3.2592   -0.3435    2.7630   16.9240  
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 39.935861   0.717499   55.66   <2e-16 ***
## horsepower  -0.157845   0.006446  -24.49   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for gaussian family taken to be 24.06645)
## 
##     Null deviance: 23819.0  on 391  degrees of freedom
## Residual deviance:  9385.9  on 390  degrees of freedom
## AIC: 2363.3
## 
## Number of Fisher Scoring iterations: 2

cv.error = cv.glm(Auto, glm.fit)

names(cv.error)

## [1] "call"  "K"     "delta" "seed"

round(cv.error$delta,2)

## [1] 24.23 24.23

above numbers are CV Error (Test MSE). as we have only one model, both numbers are same.

cv.error = rep(0,5)

for (i in 1:5) {
        glm.fit = glm(mpg ~ poly(horsepower,i), data=Auto)
        cv.error[i] = round(cv.glm(Auto, glm.fit)$delta[1],2)
}

cv.error

## [1] 24.23 19.25 19.33 19.42 19.03

we see a huge drop in test-MSE from linear and quadratic fits (1st and 2nd), but then for other polynomial models the improvement in test-MSE is not that big.

all 5 est-MSEs are for different model using LOOCV.

No matter how many times you run LOOCV, you will get same results as the concept is to use each observation point as test set and the rest for training total n times. so no variation in estimate test-MSE from run to run (thats we did not set the seed, no need). though it has inherent variability in test-MSE as only one observation was used for test-set.