To implement k-fold cross‐validation, we randomly partition the data into k folds of roughly equal size. For each fold i, we treat that fold as the validation set and train the model on the remaining k-1 folds. We record the validation error on fold i, After doing this for all k folds, we average the validation errors to obtain the cross‐validation error estimate, which approximates the test error.
Compared to a single validation set approach, k fold CV makes more efficient use of the data and typically yields a more stable estimate of test error, although it requires more computation (fitting the model k times). LOOCV (k=n) is an extreme case in which we fit the model n times—this often has lower bias (because each training set is almost the entire data set) but can have higher variance in the estimated test error and is very computationally expensive. The validation‐set approach is simplest and fastest but can yield high‐variance estimates of test error and uses less data for training, often resulting in higher bias in the fitted model.
-To estimate the variability of a predicted value at X0, we can employ a bootstrap approach. First, we repeatedly draw bootstrap samples (each of size, n) from the original data with replacement. For each bootstrap sample, we refit the model and then obtain the predicted response at X0. Collecting these predictions across all bootstrap replications gives us a distribution of fitted values at X0. The sample standard deviation of these predictions serves as an estimate of the standard error (i.e., the uncertainty) associated with predicting Y at X0. Because the bootstrap re samples directly from the empirical distribution of the data this method applies equally well to linear, non‐linear, or non‐parametric learning methods.