Introduction

Models to be discussed:

• Ordinary Linear Regression
• Partial Least Square (PLS)
• Penalized Models: Lasso, Ridge and Elastic Net

Introduction

Each model can be written in the form directly or indirectly:

• \(y_i\): response variable
• \(\beta_0\): estimated intercept
• \(\beta_i\): estimated coefficent
• \(x_{ij}\): predictor value
• \(e_i\): random error that cannot be explained by the model

Objective: Each of these models seeks to find estimates of the parameters so that the sum of the squared errors or a function of the sum of the squared errors is minimized. The estimates fall along the spectrum of the bias-variance trade-off.

• Ordinary linear regression, finds parameter estimates that have minimum bias

• Ridge, lasso, and the elastic net find estimates that have lower variance.

Introduction

Overall Advantages of models:

• Highly interpretable
• Relationships among predictors can be further interpreted through the estimated coefficients.
• Enables computation of standard errors of the coefficients.

Overall Disadvantages on models:

• Relationship between the predictor and response are expected to fall along a straight line (1 - predictor example) or a flat hyperplane for multiple predictors
• Augmentation or predictors may be required in the case on non-linear relationships and in some cases, linear models may not not be adequate/suitable

What is Ordinary Linear Regression?

• A type of Predictive Analysis

  • Main Players
    • Predictor Variables
    • Dependent / Response Variable

Objective:

• Finding the plane that minimizes the sum-of-squared errors (SSE) between the observed and predicted response

• Why? Reduces bias

What is Ordinary Linear Regression?

Preprocessing techniques:

• Addressing Multicollinearity: Use the variance inflation factor (VIF)
• Outlier Treatment: Remove or use Transformation techniques
• Missing Values: Imputation or removal

What is Ordinary Linear Regression?

If too many predictors remain after preprocessing steps… - Danger: Overfitting - Lacks degree of freedom

Alternatives:

• Principal Component Analysis (PCA)
  • Aims at reducing a large set of variables to a small set that explains most the variance.
• PLS
  • Simultaneous dimension reduction
• Ridge, Lasso and Elastic Net
  • Shrinking parameter estimates

What is Ordinary Linear Regression?

Limitations:

• Does not take into consideration if the data have curvature or nonlinear structure. Use diagnostic plots for visual
• Adding Polynomials - Quadratic (squared), cubic (cubed) terms turns a linear regression model into a curve. This makes it a straightforward way to model curves without having to model complicated non-linear models.
• Focusing on outliers
  • Very sensitive to outliers

Cousins Of Linear Regression

Penalized Methods (a.k.a. Regularization, Shrinkage Methods)

• Helps to reduce OLS high variance due to overfitting and/or multicollinearity
• Adds penalty to OLS Sum of Squared Residuals when estimates become too large
• Trades-off a little Bias for substantial drop in Variance

Regression Models: Ridge, Lasso, Elastic Net

• R Implementation: cv.glmnet, glmnet

  • \(\lambda\): parameter that controls shrinkage

  • \(\alpha\): parameter that identifies the penalty mode

Cousins Of Linear Regression

Ridge Regression

• penalizes sum of squared values (L2 penalty)

Cousins Of Linear Regression

Lasso Regression

• penalizes sum of absolute values (L1 penalty)
• can shrink the coefficients to absolute 0 (= feature selection)

Cousins Of Linear Regression

Elastic Net Regression

• Dynamic blending of Ridge & Lasso