A: K-fold cross validation is a resampling technique used to evaluate a model’s performance by repeatedly training and testing on different subsets of the data. It works as follows:
Divide the dataset into K equal-sized subsets.
For example, if K = 5, the dataset is split into 5 folds. Train and test the model 5 times: In each iteration, one fold is used as the test set, and the remaining K-1 folds are used as the training set. The model is trained on the training folds and evaluated on the test fold. Repeat the process for all K folds so that each data point gets to be in the test set once.
Compute the final performance metric or accuracy by averaging the test errors across all K iterations. As of an example of 5 folds, Fold 1: Train on Folds 2,3,4,5 and Test on Fold 1 Fold 2: Train on Folds 1,3,4,5 and Test on Fold 2 similiarly it is followed for all 5 folds.
Advantages of K-fold CV over validation set approach:
Every observation is used for both training and validation, increasing data efficiency. Averaging over K folds gives a more stable and reliable estimate of model performance. Reduces the risk of over fitting to a particular split
Disadvantages:
The model must be trained K times compared to just once in the validation set approach. So it is computationally expensive. Slightly more complex to implement.
Advantages of K-Fold CV over LOOCV:
Much faster i.e., K-Fold CV only requires training the model K times (e.g., K = 5 or 10), whereas LOOCV requires training n times where n is the no.of observations, which can be very slow for large datasets. Lower variance of error estimate. K-Fold CV generally has slightly higher bias but much lower variance than LOOCV, making it more suitable for model selection.
Disadvantages: Since each training set in K-Fold CV omits more data than LOOCV, it may underfit slightly more, especially with small datasets.
LOOCV uses nearly all available data for training in each iteration, which may be helpful when the dataset is very small.
A: When we use a statistical learning method to predict a response Y for a particular value of a predictor X, we want to know how uncertain that prediction is. This uncertainty is measured by the standard deviation of the prediction. There are two main components that contribute to the uncertainty: Model Uncertainty (Estimation Error) and Irreducible Error ( Residual Variability)
Estimation Error comes from the fact that the model was trained on a sample, not the entire population. The estimated model coefficients are themselves random variables.
Irreducible error represents natural randomness in the data. Even with perfect model knowledge, individual predictions still vary due to noise or unmeasured factors.
There are two main ways, and each has a different formula for estimating this standard deviation, They are:
Estimating the Standard Deviation of the Mean Response at X0. For Example: What is the average income of people with 10 years of education? This standard error tells us how much the predicted average response at X0 , might vary from sample to sample.
Estimating the Standard Deviation of a New Observation at X0. What is the income of this one person with 10 years of education? We now want to predict the actual Y for a new individual, not the average. This prediction has more variability, because it includes random noise in addition to model uncertainty.This is always larger than the standard error for the mean response, because it accounts for both the uncertainty in the estimated model, and the natural variability in individual responses (irreducible error).
And then finally to estimate the SD of our prediction we follow these steps. At first we’ve to decide whether we’re predicting the mean response or a new observation. Secondly, we’ve to use the appropriate formula to compute the standard error.
This standard deviation measures the uncertainty in your prediction due to model estimation error and randomness in the data.