Lasso, Similarly to that of Ridge, tends to shrink some coefficients to zero. This effectively reduces the number of observations used in the model. Thus, compared to that of least squares, which uses all the predictors, the lasso is less flexible. This sacrifice in flexibility results in reduced variance.
Ridge Regression tends to shrink coefficients towards zero. This causes the model to be less flexible compared to ordinary least squares since it is restricting the range of coefficient values. While this tends to introduce bias, it also works to reduce variance. Thus, iii is the correct answer for this prompt.
Non-Linear methods are generally viewed as more flexible due to them having to account for relationships that aren’t linear and more complex thus it is required that they are more flexible. This does the opposite of the methods above in the sense that its flexibility reduces bias but increases variance. Thus, the correct answer for this prompt is ii.
As s increases from zero, we allow more coefficients to be non-zero, this allows the model to better fit the data. This will cause a increase in model flexibility, with a decrease in model flexibility training RSS decreases because we never lose any “fitting power” by allowing bigger coefficients. Thus, answer iv is the best for this prompt.
As s increases from 0, initially we reduce under fitting so the test RSS will typically decrease initially. After some optimal region, where the bias-variance trade off is balanced, if s were to become large we would observe a an increase in test RSS due to over fitting. Thus, answer ii is best for this prompt.
Variance tends to increase along side model flexibility. When s=0 it is very restrictive, thus has very little variance. However, if it is to increase the opposite is to happen and variance increases. The more coefficients that we introduce into the model increase its variance. Thus, iii is the correct answer for this prompt.
Bias refers to how far, on average, our model’s prediction are from the truth. Models with restricted s values have very high bias since they are unable to capture the true relationships with little info. However, when s increases the model becomes more flexible and can better approximate the truth, thus bias lower. So, answer iv is the best for this prompt.
Irreducible error, is noise within the data that cannot be explained or accounted for by a model. It is independent of the model that we create and fit, thus it remains constant despite increases in s. So, v is the best answer for this prompt.