# 9\. Regularization

In mathematics, statistics, and computer science, particularly in the fields of machine learning and inverse problems, regularization is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting ([Wikipedia Regularization](https://en.wikipedia.org/wiki/Regularization_(mathematics))).

Due to the sparsity within our data, our training sets will often be ill-posed (singular). Applying regularization to the regression has many advantages, including:

1.  Converting ill-posed problems to well-posed by adding additional information via the penalty parameter ![\lambda](img/84566f6949f9a2f8734318c284f441f7.jpg)
2.  Preventing overfitting
3.  Variable selection and the removal of correlated variables ([Glmnet Vignette](https://web.stanford.edu/~hastie/Papers/Glmnet_Vignette.pdf)). The Ridge method shrinks the coefficients of correlated variables while the LASSO method picks one variable and discards the others. The elastic net penalty is a mixture of these two; if variables are correlated in groups then ![\alpha=0.5](img/44d0db1ee959675768959ef02c868b32.jpg) tends to select the groups as in or out. If α is close to 1, the elastic net performs much like the LASSO method and removes any degeneracies and wild behavior caused by extreme correlations.

## 9.1\. Ordinary least squares regression

![\min _{\Bbeta\in \mathbb {R} ^{n}}{\frac {1}{n}}\|{\X}\Bbeta -{\y}\|^{2}](img/6869432b79fecd2af9dcc85625f6f356.jpg)

When ![\lambda=0](img/6170e4e4344d720bef3ff354a507f6fa.jpg) (i.e. `regParam` ![=0](img/9b8d35ed3fc944be3e432c47b447f92f.jpg)), then there is no penalty.

```
LinearRegression(featuresCol="features", labelCol="label", predictionCol="prediction", maxIter=100,
regParam=0.0, elasticNetParam=0.0, tol=1e-6, fitIntercept=True, standardization=True, solver="auto",
weightCol=None, aggregationDepth=2)

```

## 9.2\. Ridge regression

![\min _{\Bbeta\in \mathbb {R} ^{n}}{\frac {1}{n}}\|{\X}\Bbeta-{\y}\|^{2}+\lambda \|\Bbeta\|_{2}^{2}](img/007f623d1ec885d996b1b72689ba7cb1.jpg)

When ![\lambda&gt;0](img/85a26958e55acab88aef1ab37443b30b.jpg) (i.e. `regParam` ![&gt;0](img/08c6744e242573f80b23af5dbbf21a94.jpg)) and ![\alpha=0](img/946e2a245a8fae021860977280b52b44.jpg) (i.e. `elasticNetParam` ![=0](img/9b8d35ed3fc944be3e432c47b447f92f.jpg)) , then the penalty is an L2 penalty.

```
LinearRegression(featuresCol="features", labelCol="label", predictionCol="prediction", maxIter=100,
regParam=0.1, elasticNetParam=0.0, tol=1e-6, fitIntercept=True, standardization=True, solver="auto",
weightCol=None, aggregationDepth=2)

```

## 9.3\. Least Absolute Shrinkage and Selection Operator (LASSO)

![\min _{\Bbeta\in \mathbb {R} ^{n}}{\frac {1}{n}}\|{\X}\Bbeta-{\y}\|^{2}+\lambda\|\Bbeta\|_{1}](img/be4d423d387dbcd6a770d4bda5718082.jpg)

When ![\lambda&gt;0](img/85a26958e55acab88aef1ab37443b30b.jpg) (i.e. `regParam` ![&gt;0](img/08c6744e242573f80b23af5dbbf21a94.jpg)) and ![\alpha=1](img/361ef23b0743d01bb30ead2dccc9edca.jpg) (i.e. `elasticNetParam` ![=1](img/0702875bab1a20dbb9d95fab3813c019.jpg)), then the penalty is an L1 penalty.

```
LinearRegression(featuresCol="features", labelCol="label", predictionCol="prediction", maxIter=100,
regParam=0.0, elasticNetParam=0.0, tol=1e-6, fitIntercept=True, standardization=True, solver="auto",
weightCol=None, aggregationDepth=2)

```

## 9.4\. Elastic net

![\min _{\Bbeta\in \mathbb {R} ^{n}}{\frac {1}{n}}\|{\X}\Bbeta-{\y}\|^{2}+\lambda (\alpha \|\Bbeta\|_{1}+(1-\alpha )\|\Bbeta\|_{2}^{2}),\alpha \in (0,1)](img/0ac8a503cc147ea1ccb9c24bf83a5992.jpg)

When ![\lambda&gt;0](img/85a26958e55acab88aef1ab37443b30b.jpg) (i.e. `regParam` ![&gt;0](img/08c6744e242573f80b23af5dbbf21a94.jpg)) and `elasticNetParam` ![\in (0,1)](img/2fbe25eafed24324bfbde9c4d3dca1f4.jpg) (i.e. ![\alpha\in (0,1)](img/c45aab6ee1f6f00de1ac3f428e62b01c.jpg)) , then the penalty is an L1 + L2 penalty.

```
LinearRegression(featuresCol="features", labelCol="label", predictionCol="prediction", maxIter=100,
regParam=0.0, elasticNetParam=0.0, tol=1e-6, fitIntercept=True, standardization=True, solver="auto",
weightCol=None, aggregationDepth=2)

```