Modern High-Dimensional Statistics and Regularization
When parameters outnumber observations
High-dimensional challenges
When is large relative to , classical estimators may become unstable or non-unique. Regularization imposes structure to obtain estimable solutions.
What does the penalty encourage?
The penalty in lasso tends to set some coefficients exactly to zero.
Correct answer: Sparsity
Write the general form of a penalized estimation problem.
Regularization trades fit for complexity control.
Correct answer: Minimize loss plus penalty, such as .
Examples of regularizers
Ridge uses penalty , lasso uses penalty , and elastic net combines both. These methods stabilize estimation and improve prediction.
Penalty comparison
| Method | Penalty | Primary effect |
|---|---|---|
| Ridge | Shrinkage | |
| Lasso | Sparsity | |
| Elastic net | Sparse plus grouped shrinkage |