Lasso and Elastic Net

What Are Lasso and Elastic Net?

Lasso is a regularization technique. Use lasso to:

• Reduce the number of predictors in a regression model.

• Identify important predictors.

• Select among redundant predictors.

• Produce shrinkage estimates with potentially lower predictive errors than ordinary least squares.

Elastic net is a related technique. Use elastic net when you have several highly correlated variables. lasso provides elastic net regularization when you set the Alpha name-value pair to a number strictly between 0 and 1.

See Lasso and Elastic Net Details.

For lasso regularization of regression ensembles, see regularize.

Lasso and Elastic Net Details

Overview of Lasso and Elastic Net

Lasso is a regularization technique for performing linear regression. Lasso includes a penalty term that constrains the size of the estimated coefficients. Therefore, it resembles ridge regression. Lasso is a shrinkage estimator: it generates coefficient estimates that are biased to be small. Nevertheless, a lasso estimator can have smaller mean squared error than an ordinary least-squares estimator when you apply it to new data.

Unlike ridge regression, as the penalty term increases, lasso sets more coefficients to zero. This means that the lasso estimator is a smaller model, with fewer predictors. As such, lasso is an alternative to stepwise regression and other model selection and dimensionality reduction techniques.

Elastic net is a related technique. Elastic net is a hybrid of ridge regression and lasso regularization. Like lasso, elastic net can generate reduced models by generating zero-valued coefficients. Empirical studies have suggested that the elastic net technique can outperform lasso on data with highly correlated predictors.

Definition of Lasso

The lasso technique solves this regularization problem. For a given value of λ, a nonnegative parameter, lasso solves the problem

$$\underset{{\beta }_0,\beta }{\min }\left(\frac{1}{2N}{\displaystyle \sum _{i=1}^N{\left(y_i-{\beta }_0-x_i^T\beta \right)}^2}+\lambda {\displaystyle \sum _{j=1}^p\left|{\beta }_j\right|}\right).$$

• N is the number of observations.

• y_i is the response at observation i.

• x_i is data, a vector of p values at observation i.

• λ is a positive regularization parameter corresponding to one value of Lambda.

• The parameters β₀ and β are scalar and p-vector respectively.

As λ increases, the number of nonzero components of β decreases.

The lasso problem involves the L¹ norm of β, as contrasted with the elastic net algorithm.

Definition of Elastic Net

The elastic net technique solves this regularization problem. For an α strictly between 0 and 1, and a nonnegative λ, elastic net solves the problem

$$\underset{{\beta }_0,\beta }{\min }\left(\frac{1}{2N}{\displaystyle \sum _{i=1}^N{\left(y_i-{\beta }_0-x_i^T\beta \right)}^2}+\lambda P_{\alpha }\left(\beta \right)\right),$$

where

$$P_{\alpha }\left(\beta \right)=\frac{(1-\alpha )}{2}{\Vert \beta \Vert }_2^2+\alpha {\Vert \beta \Vert }_1={\displaystyle \sum _{j=1}^p\left(\frac{(1-\alpha )}{2}{\beta }_j^2+\alpha \left|{\beta }_j\right|\right)}.$$

Elastic net is the same as lasso when α = 1. As α shrinks toward 0, elastic net approaches ridge regression. For other values of α, the penalty term P_α(β) interpolates between the L¹ norm of β and the squared L² norm of β.

References

[1] Tibshirani, R. "Regression shrinkage and selection via the lasso." Journal of the Royal Statistical Society, Series B, Vol 58, No. 1, pp. 267–288, 1996.

[2] Zou, H. and T. Hastie. "Regularization and variable selection via the elastic net." Journal of the Royal Statistical Society, Series B, Vol. 67, No. 2, pp. 301–320, 2005.

[3] Friedman, J., R. Tibshirani, and T. Hastie. "Regularization paths for generalized linear models via coordinate descent." Journal of Statistical Software, Vol 33, No. 1, 2010. https://www.jstatsoft.org/v33/i01

[4] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning, 2nd edition. Springer, New York, 2008.