Lesson 2.3 - The Bias Variance Trade-Off

Bias-Variance Trade-Off

Assume that we are considering a variety of training methods for a regression problem, and that the methods can be ordered in terms of their flexibility. As we saw in the last lesson, the training MSE score will typically decrease as the flexibility increases. The validation MSE, on the other hand, with usually display a U-shape relationship when plotted against flexibility. In this section, we will come to understand this aspect of the validation MSE by studying two related values: Bias and Variance.

Setting

Assume the following:

• We have access to a known population model, \(Y = f(X) + \varepsilon\), where \(\varepsilon \sim N(0, \sigma^2)\).

• The following values of the predictor \(X\) have been set: \(x_0, x_1, x_2, ..., x_n\)

• A specific regression method has been supplied to us.

Suppose that we wish to study the performance of the supplied regression method on samples generated from this population model. We might carry out our study by performing the following steps many times:

1. We will collect a sample of \(n+1\) observations \((x_i, y_i)\) for \(i = 0, 1, 2, ... n\).

2. We will designate the observations \((x_i, y_i)\) for \(i = 1, 2, ... n\) as our training set, and will use the single observation \((x_0, y_0)\) for validation.

3. Applying the selected regression method to the training set, we generate a fitted model \(\hat y = \hat f (x)\).

4. We will plug the pre-selected value \(X = x_0\) into our model to generate a prediction \(\hat y_0 = \hat f (x_0)\).

Because of the random variable \(\varepsilon\) in our population models, the \(y_i\) values will vary from one sample to the next. As a result, the fitted model generated in Step 3 will vary from one iteration to the next, even though we are applying the same fitted method each time. Since the fitted function \(f\) changes with each iteration, so too does the fitted value \(\hat y_0\). In this sense, the value of \(\hat y_0\) represents a single observation of a random variable, which we will denote by \(\hat Y_0\).

Since \(\hat Y_0\) is a random variable, we can (in theory) calculate \(\mathrm E \left[\hat Y_0 \right]\) and \(\mathrm {Var}\left[\hat Y_0 \right]\).

Bias

We will define the bias of our fitted value \(\hat Y_0\) as follows:

\[\mathrm{Bias} \left[\hat Y_0 \right] = \mathrm E \left[\hat Y_0 \right] - f(x_0)\]

Note that since \(\mathrm E \left[\varepsilon \right] = 0\), the expected value of \(Y\) given \(X=x_0\) is given by \(\mathrm E \left[Y \mid X=x_0 \right] = f(x_0)\). With this in mind, we can interpret \(\mathrm{Bias}\) as the amount that you would expect the prediction \(\hat Y_0\) to exceed the observed value of \(Y_0\) (on average). Regression methods that have a positive bias at \(X=x_0\) will tend to generate over-predictions for \(Y_0\), while methods with a negative will tend to generate under-predictions.

Decomposing Validation MSE

Since we are using a single observation \((x_0, Y_0)\) for our validation set, the validation MSE is given by:

\[MSE = \left(Y_0 - \hat Y_0 \right)^2\] Since \(Y_0\) and \(\hat Y_0\) are both random variables, \(MSE\) is also a random variable whose value depends on the sample drawn (and the fitted model resulting from that sample). It can be shown that the expected value of \(MSE\) can be decomposed in the following way:

\[\mathrm E \left[ MSE \right] = \mathrm{Var}\left[\hat Y_0 \right] + \left(\mathrm{Bias}\left[\hat Y_0 \right] \right)^2 + \mathrm{Var}\left(\varepsilon\right)\]

Note that all of the values in the decomposition of \(\mathrm E \left[ MSE \right]\) are positive, and that \(\mathrm{Var}\left(\varepsilon\right) = \sigma^2\) is a constant that comes from the population model, and is unaffected by our choice of regression method.

Minimizing E[MSE] (The Trade-Off)

Assume know that we have at our disposal several regression methods, with varying levels of flexibility. Our goal is to select a method that will minimize \(\mathrm E \left[ MSE \right]\). Since we have no control over \(\mathrm{Var}\left(\varepsilon\right)\), our goal amounts to trying to minimize the following two quantities:

• The Squared Bias, \(\left(\mathrm{Bias}\left[\hat Y_0 \right] \right)^2\)
• The Variance, \(\mathrm{Var}\left[\hat Y_0 \right]\)

Unfortunately, it tends to be the case that regression methods which produce models with very low values for one of these metrics will also produce very large values for the other metric. More specifically:

• Very flexible models tend to have low bias, but high variance.
• Very rigid models tend to have high bias, but low variance.

Our goal in selecting a regression method is to determine an appropriate level of flexibility in order to obtain a model with moderately low values of both squared bias and variance. This is referred to as the bias-variance trade-off.

Plotting Bias, Variance, and MSE

The following plots display typical ways in which bias, variance, and MSE depend on model flexibility.

Final Observations

It should be pointed out that it is impossible to calculate either bias or variance without having access to the population model. As a result, it is not feasible to calculate these values in practice. However, the way in which these quantities depend on the flexibility of our fitting method, and the way in which these values affect the value of MSE are import principles to keep in mind.