Preliminary Remarks

The linear regression model is a useful starting point because:

• It is familiar from classical statistics.
• It illustrates all core ML concepts (loss, training, evaluation, interpretation).
• It is somewhat atypical for ML: \(\phi(.)\) is imposed rather than learnt; there are no tuning parameters and no automatic feature selection.

The model specification:

\[ y_i = \beta + \sum_{j=1}^{P} \omega_j x_{ij} + \varepsilon_i \]

• \(\beta\): bias (intercept)
• \(\omega_j\): weight (partial slope) for feature \(j\)
• \(P\): number of predictive features
• \(\varepsilon_i\): irreducible error

Loss Functions for Regression

The learning process selects weights \(\boldsymbol{\omega}\) that minimize the chosen loss function over the training set.

Performance Metrics (test set)

Three standard metrics, computed on the test set only:

\[\text{MAE} = \frac{1}{n_2} \sum_{i=1}^{n_2} |\psi_i - y_i|\]

\[\text{RMSE} = \sqrt{\frac{1}{n_2} \sum_{i=1}^{n_2} (\psi_i - y_i)^2}\]

\[R^2 = r_{\psi, y}^2\]

Interpretation: Partial Dependence Plots

After training, we want to understand which features drive predictions and how.

A partial dependence plot (PDP) isolates the relationship between a focal feature \(\boldsymbol{x}_f\) and the predicted label, averaging over all other (control) features \(\boldsymbol{x}_c\):

\[\hat{f}(\boldsymbol{x}_f) = \mathbb{E}_{\boldsymbol{x}_c}\left[f(\boldsymbol{x}_c, \boldsymbol{x}_f, \boldsymbol{q})\right] = \int f(\boldsymbol{x}_c, \boldsymbol{x}_f, \boldsymbol{q})\, dp(\boldsymbol{x}_c)\]

This is the ML equivalent of the ceteris paribus condition.

Gradient Descent

Most ML problems are too complex for closed-form solutions, i.e., we turn to gradient descent:

1. Initialize weights \(\boldsymbol{\omega}\).
2. Compute the gradient of the loss function with respect to \(\boldsymbol{\omega}\): \[\nabla_{\boldsymbol{\omega}} L\]
3. Update weights in the direction of steepest descent: \[\boldsymbol{\omega} \leftarrow \boldsymbol{\omega} - \eta \nabla_{\boldsymbol{\omega}} L\]
4. Repeat until convergence (loss ceases to decrease meaningfully).

\(\eta\) (eta) is the learning rate — a tuning parameter controlling step size.

Why a Good Optimizer Matters

We want optimizers that are:

• Simple in terms of calculus — avoid complex second-order derivatives
• Numerically evaluable — can be computed on real data
• Mini-batch capable — process small subsets of data at a time (important for large datasets)

Stochastic gradient descent (SGD): updates weights using a random mini-batch of instances at each step — much faster than full-batch gradient descent on large datasets.

Modern variants (Adam, RMSProp, AdaGrad) adapt the learning rate automatically and are standard in deep learning.

The three sources

Every predictive model produces error. There are exactly three sources:

\[\text{Total Error} = \underbrace{B^2}_{\text{Bias}^2} + \underbrace{V}_{\text{Variance}} + \underbrace{I}_{\text{Irreducible}}\]

The Bias-Variance Trade-Off

Bias and variance pull in opposite directions:

• To reduce bias: increase model complexity (add features, interactions, nonlinearities)
• To reduce variance: decrease model complexity (penalize, simplify, reduce features)

The goal is to find the sweet spot — the complexity level at which total error is minimized.

Diagnosing Over- and Under-Fitting

Compare training and test performance:

Tuning Parameters

A tuning parameter affects how an algorithm operates but cannot be estimated from the data — it must be set by the researcher.

Examples:

• \(M\): number of principal components to retain
• \(\lambda\): regularization strength in lasso/ridge
• \(k\): number of neighbors in kNN
• Learning rate \(\eta\) in neural networks

Although the researcher sets the final value, we can use re-sampling to let the data inform that decision.

Re-sampling provides out-of-sample estimates of performance for each candidate value — it is superior to re-substitution.

Three Re-Sampling Methods

The Three-Way Split of Data

When tuning is needed, we divide data into three parts:

\[ \underbrace{\text{Training proper}}_{\text{fit the model}} + \underbrace{\text{Validation}}_{\text{tune hyperparameters}} + \underbrace{\text{Test}}_{\text{final evaluation}} \]

Once the model is deployed on the test set, no further changes to the model are permitted.

What is Regularization?

Reminder: variance error arises when a model is too complex and overfits the training data. One powerful remedy is regularization.

Regularization adds a penalty for over-fitting to the loss function, which serves as a constraint on the optimization problem.

This can be understood as a shrinkage estimator, i.e. coefficients that contribute little to prediction are shrunk toward zero, reducing model complexity automatically.

The penalized optimization problem takes the general form:

\[\min_{\boldsymbol{\omega}} \underbrace{\sum_{i=1}^{n_1} (\beta + \boldsymbol{x}_i^\top \boldsymbol{\omega} - y_i)^2}_{\text{fit to training data}} + \underbrace{\lambda \cdot \text{Penalty}(\boldsymbol{\omega})}_{\text{complexity penalty}}\]

\(\lambda\) is a tuning parameter controlling the strength of regularization — selected by cross-validation.

The Lasso

Lasso = Least Absolute Shrinkage and Selection Operator (Tibshirani (1996)).

The penalty is the \(L_1\)-norm of the weight vector:

\[\min_{\boldsymbol{\omega}} \sum_{i=1}^{n_1} (\beta + \boldsymbol{x}_i^\top \boldsymbol{\omega} - y_i)^2 + \lambda \sum_{j=1}^{P} |\omega_j|\]

Key property: the lasso shrinks some coefficients exactly to zero, performing automatic feature selection.

Tikhonov Regularization (Ridge Regression)

The penalty is the \(L_2\)-norm (squared coefficients):

\[\min_{\boldsymbol{\omega}} \sum_{i=1}^{n_1} (\beta + \boldsymbol{x}_i^\top \boldsymbol{\omega} - y_i)^2 + \lambda \sum_{j=1}^{P} \omega_j^2\]

Key difference from lasso: ridge shrinks coefficients toward zero but rarely sets them exactly to zero — it retains all features.

Elastic Nets

Elastic nets combine lasso and ridge, offering a mixture tuned by \(\alpha \in [0,1]\):

\[\text{Penalty}(\alpha) = \sum_{j=1}^{P} \left[\frac{1}{2}(1-\alpha)\omega_j^2 + \alpha|\omega_j|\right]\]

\[\min_{\boldsymbol{\omega}} \sum_{i=1}^{n_1} (\beta + \boldsymbol{x}_i^\top \boldsymbol{\omega} - y_i)^2 + \lambda \cdot \text{Penalty}(\alpha)\]

• \(\alpha = 1\): reduces to the lasso
• \(\alpha = 0\): reduces to Tikhonov/ridge
• Both \(\lambda\) and \(\alpha\) are tuning parameters

Elastic nets are a flexible default when it is unclear whether lasso or ridge is more appropriate.