Intermediate30 min

Regularisation

L1 and L2 penalties, elastic net, the constrained-optimisation view, and why L1 induces sparsity.

Assumes you know

Regularisation

Intuition first

Overfitting happens when a model has enough freedom to chase noise. Regularisation removes some of that freedom — not by deleting parameters, but by making large parameters expensive.

Think of it as a tax on complexity. The model still wants to fit the data, but now every unit of coefficient magnitude costs something. Faced with two explanations that fit equally well, it picks the smaller one. Since noise-chasing generally requires large, finely-tuned coefficients that cancel each other out, taxing magnitude suppresses noise-fitting more than it suppresses genuine signal.

The tax rate is a hyperparameter. Set it to zero and you are back to unregularised fitting. Set it too high and the model is taxed into predicting nothing but the mean.

Notation used in this lesson
Symbol	Meaning	Read aloud
w	Vector of model weights (coefficients)	w
λ	Regularisation strength — the tax rate	lambda
‖w‖₂²	Sum of squared weights	L2 norm squared
‖w‖₁	Sum of absolute weights	L1 norm
J(w)	Regularised objective being minimised	J of w

The penalised objective

J(w) = \underbrace{\frac{1}{n}\sum_{i=1}^{n} L\big(y_i, h_w(x_i)\big)}_{\text{fit the data}} \;+\; \underbrace{\lambda\, \Omega(w)}_{\text{stay simple}}

The two standard penalties:

\text{L2 (ridge)}: \quad \Omega(w) = \lVert w \rVert_2^2 = \sum_{j=1}^{d} w_j^2

\text{L1 (lasso)}: \quad \Omega(w) = \lVert w \rVert_1 = \sum_{j=1}^{d} \lvert w_j \rvert

Why L1 produces exact zeros and L2 does not

This is the single most asked question about regularisation, and the geometric answer is the clearest.

The penalty defines a feasible region; the ellipses are contours of the data-fitting loss. L1's diamond has corners on the axes, so the first contact often lands exactly on an axis — a zero coefficient. L2's circle has no corners.

The algebraic version: soft thresholdingAdvanced

Consider the simplest case — one coefficient, orthonormal features, so the least-squares solution is some value $z$ . The L1-penalised problem is

\min_{w} \; \tfrac{1}{2}(w - z)^2 + \lambda \lvert w \rvert

For $w > 0$ the derivative is $(w - z) + \lambda$ , zero at $w = z - \lambda$ . For $w < 0$ it is $(w - z) - \lambda$ , zero at $w = z + \lambda$ . Neither is valid if it crosses zero, in which case the minimum sits at the kink $w = 0$ . Combining:

w^{\text{lasso}} = \text{sign}(z)\,\max\big(\lvert z \rvert - \lambda,\; 0\big)

This is the soft-thresholding operator: shift every coefficient towards zero by $\lambda$ , and clamp at zero. Any coefficient with $\lvert z \rvert \leq \lambda$ becomes exactly zero.

The L2 problem is smooth:

\min_w \; \tfrac{1}{2}(w - z)^2 + \lambda w^2 \quad\Longrightarrow\quad (w - z) + 2\lambda w = 0 \quad\Longrightarrow\quad w^{\text{ridge}} = \frac{z}{1 + 2\lambda}

A proportional shrinkage. It can make coefficients arbitrarily small but never exactly zero for finite $\lambda$ , because the penalty's gradient vanishes at the origin while L1's does not.

That asymmetry — L1 has a non-zero gradient at zero, L2 does not — is the entire mechanism behind sparsity.

Ridge in closed form

For linear regression the L2 solution is analytic:

\hat{w}_{\text{ridge}} = \big(X^{\mathsf{T}}X + \lambda I\big)^{-1} X^{\mathsf{T}} y

Why ridge fixes multicollinearityAdvanced

Take the SVD $X = U\Sigma V^{\mathsf{T}}$ with singular values $\sigma_j$ . The ordinary least-squares and ridge solutions become

\hat{w}_{\text{OLS}} = \sum_j \frac{1}{\sigma_j}\,(u_j^{\mathsf{T}}y)\, v_j, \qquad \hat{w}_{\text{ridge}} = \sum_j \frac{\sigma_j}{\sigma_j^2 + \lambda}\,(u_j^{\mathsf{T}}y)\, v_j

When features are nearly collinear, some $\sigma_j \approx 0$ . In OLS that $1/\sigma_j$ explodes, so tiny changes in the data produce enormous swings in the coefficients — the definition of high variance.

Ridge replaces $1/\sigma_j$ with $\sigma_j/(\sigma_j^2 + \lambda)$ , which tends to $\sigma_j / \lambda \to 0$ as $\sigma_j \to 0$ . The unstable directions are damped rather than amplified.

Adding $\lambda I$ also guarantees $X^{\mathsf{T}}X + \lambda I$ is invertible for any $\lambda > 0$ , so ridge has a unique solution even when $d > n$ and OLS has none.

Comparing the penalties

	L2 / Ridge	L1 / Lasso	Elastic Net
Shrinks coefficients	Proportionally	By a constant, then clamps	Both
Produces exact zeros	No	Yes	Yes
Correlated features	Shares weight between them	Picks one arbitrarily	Shares among the group
Solution unique	Always	Not always	Always
Closed form	Yes	No — needs coordinate descent	No
Use when	Many weak predictors	Few strong predictors, want selection	Correlated groups

\text{Elastic Net}: \quad \Omega(w) = \alpha \lVert w \rVert_1 + (1 - \alpha)\lVert w \rVert_2^2

Solved problem 1 · Ridge and lasso by hand

With orthonormal features, the unregularised coefficients are

z = (3.0,\; -1.4,\; 0.5,\; -0.2)

Compute the ridge solution with $\lambda = 0.5$ and the lasso solution with $\lambda = 0.6$ .

Step 1 — ridge: proportional shrinkage

w_j = \frac{z_j}{1 + 2\lambda} = \frac{z_j}{1 + 1.0} = \frac{z_j}{2}

w^{\text{ridge}} = (1.50,\; -0.70,\; 0.25,\; -0.10)

Every coefficient halved. None reached zero, and the relative ordering is preserved exactly.

Step 2 — lasso: soft thresholding, coefficient by coefficient

w_j = \text{sign}(z_j)\max(\lvert z_j \rvert - 0.6,\; 0)

z_1 = 3.0: \quad 3.0 - 0.6 = 2.4 \quad\Longrightarrow\quad w_1 = +2.40

z_2 = -1.4: \quad 1.4 - 0.6 = 0.8 \quad\Longrightarrow\quad w_2 = -0.80

z_3 = 0.5: \quad 0.5 - 0.6 = -0.1 < 0 \quad\Longrightarrow\quad w_3 = 0

z_4 = -0.2: \quad 0.2 - 0.6 = -0.4 < 0 \quad\Longrightarrow\quad w_4 = 0

w^{\text{lasso}} = (2.40,\; -0.80,\; 0,\; 0)

Step 3 — contrast the two

Ridge kept all four features with every coefficient halved. Lasso eliminated the two weakest and shrank the survivors by a fixed amount rather than proportionally.

Note that lasso shrank the large coefficient less in relative terms — $3.0 \to 2.4$ is a 20% reduction, while $1.4 \to 0.8$ is 43%. Ridge reduced both by exactly 50%.

Answer

Ridge: $(1.50, -0.70, 0.25, -0.10)$ — all retained, proportionally shrunk. Lasso: $(2.40, -0.80, 0, 0)$ — two coefficients eliminated, performing feature selection as a side effect of fitting.

Regularisation beyond penalties

Anything that reduces effective capacity regularises, whether or not it appears in the objective:

Early stopping — halting before the model has time to memorise. For linear models it is provably close to ridge.
Dropout — randomly deactivating units, approximately averaging an ensemble.
Data augmentation — enlarging the effective dataset with label-preserving transformations.
Batch normalisation — the noise from batch statistics acts as a mild regulariser.
Weight sharing — convolution is a hard constraint that a filter applies identically everywhere.
Adding noise to inputs — for linear regression, equivalent to ridge exactly.

Choosing λ

Cross-validate over a logarithmic grid. Never a linear one: the interesting behaviour spans orders of magnitude.

python

import numpy as np
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=300, n_features=40, n_informative=8,
                       noise=15.0, random_state=0)

alphas = np.logspace(-3, 3, 25)          # log grid, 0.001 → 1000

# StandardScaler inside the pipeline: the penalty is scale-sensitive, and the
# scaler must be fitted per fold to avoid leakage.
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas, cv=5)).fit(X, y)
lasso = make_pipeline(StandardScaler(), LassoCV(alphas=alphas, cv=5, max_iter=5000)).fit(X, y)

r_coef = ridge[-1].coef_
l_coef = lasso[-1].coef_

print(f"ridge  alpha={ridge[-1].alpha_:.4f}  zeros={np.sum(r_coef == 0):2d}/40")
print(f"lasso  alpha={lasso[-1].alpha_:.4f}  zeros={np.sum(l_coef == 0):2d}/40")
print(f"true informative features: 8")

Ridge reports zero exact zeros; lasso typically recovers close to the 8 informative features. That contrast is the whole lesson in two lines of output.

Exercise 1

A model has 5,000 features and 300 training examples. Which penalty, and why?

Show solution

L1, or elastic net. With $d \gg n$ the priority is reducing the effective number of parameters, and at most 300 can be identified from 300 examples regardless.

Lasso sets most coefficients to exactly zero, producing an interpretable subset and a model whose effective capacity matches the data available. Ridge would retain all 5,000 features with small coefficients — numerically stable, since $\lambda I$ guarantees invertibility, but neither sparse nor interpretable.

Elastic net is the safer default if features are correlated. Pure lasso picks one member of a correlated group essentially arbitrarily, and which one it picks can change with a small perturbation of the data — unstable, and misleading if you interpret the selection as "these features matter". The L2 component makes correlated features share weight, stabilising the selection.

Exercise 2

Training accuracy 0.71, validation accuracy 0.69. A colleague suggests increasing L2 strength. Assess.

Show solution

Wrong direction. The gap is 2 points — there is essentially no variance problem. Both numbers being low means high bias: the model cannot represent the pattern.

Increasing $\lambda$ further constrains an already over-constrained model, raising bias and pushing both numbers down.

The right moves are the opposite: decrease $\lambda$ , add capacity or features, or use a more expressive model family. If $\lambda$ is already small and the model is still underfitting, regularisation is not the lever at all.

The general rule: diagnose from the gap before choosing a remedy. Regularisation treats variance; it makes bias worse.

Next: The Curse of Dimensionality, which explains why high-dimensional data needs this kind of constraint so badly.