• The Problem: High-dimensional data where the number of predictors (\(p\)) exceeds or approaches the number of observations (\(n\)).
• Sparsity Assumption: True underlying relationships are often driven by only a small subset of features (many coefficients are exactly zero).
• Objective: Simultaneously perform feature selection and parameter estimation to prevent overfitting.

LASSO (Least Absolute Shrinkage and Selection Operator) adds an \(L_1\) penalty to the ordinary least squares objective function:

\[\min_{\beta} \frac{1}{2n} \|y - X\beta\|_2^2 + \lambda \|\beta\|_1\]

Components:

• \(\|y - X\beta\|_2^2\): Residual sum of squares (data fidelity term).
• \(\|\beta\|_1 = \sum_{j=1}^{p} |\beta_j|\): The \(L_1\) regularization penalty enforcing sparsity.
• \(\lambda \ge 0\): Tuning parameter controlling the severity of regularization.

• Because the \(L_1\) norm is non-smooth at zero, traditional gradient descent fails.
• ISTA (Iterative Soft-Thresholding Algorithm) applies a proximal operator (soft-thresholding):

\[\beta^{(k)} = \eta_{\alpha \lambda} \left( \beta^{(k-1)} - \alpha X^T (X\beta^{(k-1)} - y) \right)\]

• FISTA (Fast Iterative Soft-Thresholding Algorithm) accelerates convergence from \(O(1/k)\) to \(O(1/k^2)\) using a momentum extrapolation step between iterations.

We simulate a high-dimensional design matrix where only 5 of 50 true coefficients are nonzero, then fit the LASSO path with glmnet:

set.seed(42)
X_mat <- matrix(rnorm(100 * 50), 100, 50)
true_beta <- c(rep(3, 5), rep(0, 45))
y_vec <- X_mat %*% true_beta + rnorm(100)

lasso_model <- cv.glmnet(X_mat, y_vec, alpha = 1)