Intermediate32 min

The Bias–Variance Trade-off

Full algebraic decomposition of expected squared error into bias, variance and noise, with a simulation.

Assumes you know

Generalisation, Overfitting and Underfitting

The Bias–Variance Trade-off

Intuition first

Imagine repeating your whole project many times: each time you collect a fresh training set from the same source, fit the same kind of model, and predict at one particular input $x$ . You get a spread of predictions.

Two things can be wrong with that spread.

Bias — the centre of the spread sits away from the truth. Every version of the model makes the same systematic mistake, so averaging them would not help. A straight line fitted to a curve is biased at almost every point.

Variance — the spread is wide. Individual models are wildly different depending on which sample they happened to see. Any single one of them is unreliable even if the average is perfect.

And a third thing that is not your fault at all: the target itself may be noisy. Even a perfect model cannot predict the unpredictable part.

The trade-off is that the knobs which reduce one usually increase the other.

Four regimes at a target. Bias is how far the cluster centre sits from the bullseye; variance is how spread out the shots are.

Notation used in this lesson
Symbol	Meaning	Read aloud
f(x)	The true underlying function	f of x
ε	Noise in the observed target, mean 0 and variance σ²	epsilon
f̂(x)	Prediction of the model fitted to one training sample	f hat of x
E[f̂(x)]	Average prediction over all possible training samples	expected f hat
σ²	Irreducible noise variance — the error floor	sigma squared

Setup

Assume the data is generated as

y = f(x) + \varepsilon, \qquad \E[\varepsilon] = 0, \quad \Var(\varepsilon) = \sigma^2

with $\varepsilon$ independent of $x$ . We fit $\hat f$ on a random training sample. The randomness we take expectations over is which training sample we happened to get, plus the noise in the test point.

The decomposition

\E\Big[\big(y - \hat{f}(x)\big)^2\Big] = \underbrace{\Big(f(x) - \E[\hat{f}(x)]\Big)^{2}}_{\text{Bias}^2} \;+\; \underbrace{\E\Big[\big(\hat{f}(x) - \E[\hat{f}(x)]\big)^2\Big]}_{\text{Variance}} \;+\; \underbrace{\sigma^2}_{\text{irreducible}}

Full derivationAdvanced

Write $\bar f = \E[\hat f(x)]$ , the average prediction across training samples, and suppress the argument $x$ . Start from the definition and insert $\bar f$ :

\E\big[(y - \hat f)^2\big] = \E\Big[\big((f + \varepsilon) - \hat f\big)^2\Big] = \E\Big[\big(\underbrace{(f - \bar f)}_{A} + \underbrace{(\bar f - \hat f)}_{B} + \underbrace{\varepsilon}_{C}\big)^2\Big]

Expand the square of the three-term sum:

= \E[A^2] + \E[B^2] + \E[C^2] + 2\E[AB] + 2\E[AC] + 2\E[BC]

Take the six terms one at a time.

$\E[A^2]$ . Both $f$ and $\bar f$ are fixed constants (not random), so

\E[A^2] = (f - \bar f)^2 = \text{Bias}^2

$\E[B^2]$ . This is the definition of the variance of the prediction:

\E[B^2] = \E\big[(\bar f - \hat f)^2\big] = \Var(\hat f)

$\E[C^2]$ . Since $\E[\varepsilon] = 0$ , $\E[\varepsilon^2] = \Var(\varepsilon) = \sigma^2$ .

$\E[AB]$ . $A$ is constant, so it factors out, and $\E[B] = \E[\bar f - \hat f] = \bar f - \bar f = 0$ :

\E[AB] = A\,\E[B] = A \cdot 0 = 0

$\E[AC]$ . $A$ constant and $\E[\varepsilon] = 0$ :

\E[AC] = A\,\E[\varepsilon] = 0

$\E[BC]$ . The noise on the test point is independent of the training sample that produced $\hat f$ , so $B$ and $\varepsilon$ are independent:

\E[BC] = \E[B]\,\E[\varepsilon] = 0 \cdot 0 = 0

All three cross terms vanish, leaving

\E\big[(y - \hat f)^2\big] = \text{Bias}^2 + \Var(\hat f) + \sigma^2

Note which assumption did the work: $\E[BC] = 0$ required the test-point noise to be independent of the training data. If the same noisy observation appears in both training and test — the definition of leakage — the cross term is non-zero and the decomposition, along with your error estimate, is invalid.

Reading the trade-off

Change	Bias	Variance
More flexible model (deeper tree, higher degree)	↓	↑
Stronger regularisation	↑	↓
More training data	–	↓
More features	↓	↑
Bagging / averaging many models	–	↓↓
Boosting (sequential fitting)	↓↓	↑
Early stopping	↑	↓

Two rows are worth dwelling on.

More data reduces variance without touching bias. That is why data is the highest-leverage intervention when you can get it — every other row involves a trade.

Bagging reduces variance without touching bias, which is the entire reason random forests work: take a high-variance low-bias model (a deep tree) and average away the variance.

Solved problem 1 · Computing bias and variance from repeated fits

The true function is $f(x) = 2x$ and we predict at $x = 3$ , so $f(3) = 6$ . Noise has $\sigma^2 = 0.25$ . Two models are each fitted on five independent training samples, giving these predictions at $x = 3$ :

Model A (a constant predictor): $5.0,\; 5.0,\; 5.1,\; 4.9,\; 5.0$
Model B (a flexible fit): $6.4,\; 5.1,\; 7.2,\; 4.8,\; 6.5$

Compute bias², variance and total expected error for each.

Step 1 — Model A: average prediction

\bar f_A = \frac{5.0 + 5.0 + 5.1 + 4.9 + 5.0}{5} = \frac{25.0}{5} = 5.00

Step 2 — Model A: bias²

\text{Bias} = f(3) - \bar f_A = 6.00 - 5.00 = 1.00

\text{Bias}^2 = 1.00

Step 3 — Model A: variance

Deviations from $5.00$ : $0,\; 0,\; +0.1,\; -0.1,\; 0$ .

\Var = \frac{0^2 + 0^2 + 0.1^2 + (-0.1)^2 + 0^2}{5} = \frac{0.02}{5} = 0.004

Step 4 — Model A: total

\text{Total} = 1.00 + 0.004 + 0.25 = 1.254

Step 5 — Model B: average prediction

\bar f_B = \frac{6.4 + 5.1 + 7.2 + 4.8 + 6.5}{5} = \frac{30.0}{5} = 6.00

Step 6 — Model B: bias²

\text{Bias} = 6.00 - 6.00 = 0 \quad\Longrightarrow\quad \text{Bias}^2 = 0

Model B is unbiased — on average it is exactly right.

Step 7 — Model B: variance

Deviations from $6.00$ : $+0.4,\; -0.9,\; +1.2,\; -1.2,\; +0.5$ .

0.4^2 = 0.16,\quad (-0.9)^2 = 0.81,\quad 1.2^2 = 1.44,\quad (-1.2)^2 = 1.44,\quad 0.5^2 = 0.25

\text{sum} = 0.16 + 0.81 + 1.44 + 1.44 + 0.25 = 4.10

\Var = \frac{4.10}{5} = 0.82

Step 8 — Model B: total, and the comparison

\text{Total} = 0 + 0.82 + 0.25 = 1.07

Model B wins, $1.07$ against $1.254$ — but not because it is unbiased. Being unbiased is worth nothing on its own; what matters is the sum.

Answer

Model A: bias² $= 1.00$ , variance $= 0.004$ , total $= 1.254$ . Model B: bias² $= 0$ , variance $= 0.82$ , total $= 1.07$ .

Model B is better overall, though it is far less stable. Note that if Model A's bias were only $0.5$ instead of $1.0$ , its total would be $0.25 + 0.004 + 0.25 = 0.504$ and it would win decisively — a small systematic error can beat a large random one.

Measuring it yourself

The decomposition is defined over repeated samples, so simulate that directly:

python

import numpy as np

rng = np.random.default_rng(0)

def true_f(x):
    return np.sin(1.5 * x)

SIGMA = 0.3
X_TEST = np.linspace(0, 4, 60)
TRUTH = true_f(X_TEST)

def fit_predict(degree, n=25):
    """Fit a polynomial of the given degree to one fresh noisy sample."""
    x = rng.uniform(0, 4, n)
    y = true_f(x) + rng.normal(0, SIGMA, n)
    coeffs = np.polyfit(x, y, degree)
    return np.polyval(coeffs, X_TEST)

for degree in (1, 3, 9, 15):
    preds = np.array([fit_predict(degree) for _ in range(200)])  # 200 repeats
    mean_pred = preds.mean(axis=0)

    bias2 = np.mean((TRUTH - mean_pred) ** 2)
    variance = np.mean(preds.var(axis=0))
    total = bias2 + variance + SIGMA ** 2

    print(f"degree {degree:2d}  bias²={bias2:7.4f}  var={variance:8.4f}  total={total:8.4f}")

Running this shows bias falling and variance rising with degree, with the total minimised at a middle value — the U-curve, measured rather than asserted.

Exercise 1

A random forest and a single deep decision tree are trained on the same data. The tree gets 0.01 training error and 0.24 validation error; the forest gets 0.03 and 0.14. Explain in bias–variance terms.

Show solution

A single deep tree has very low bias — it can carve the input space finely enough to fit almost anything — and very high variance, since a slightly different sample produces an entirely different tree. Its 0.23 gap is variance.

A random forest averages many such trees, each fitted to a bootstrap sample with a random feature subset. Averaging $B$ roughly independent predictors cuts variance by approximately a factor of $B$ while leaving bias essentially unchanged, because each tree is individually low-bias.

The numbers match: training error rose slightly, 0.01 to 0.03, which is the small bias cost of averaging and of restricting features per split. Validation error fell from 0.24 to 0.14, which is the large variance gain. Trading a little bias for a lot of variance is exactly the intended bargain.

Exercise 2

The irreducible error at some input is $\sigma^2 = 0.4$ . A model achieves expected squared error of 0.45. How much room for improvement remains, and what does this imply about further tuning?

Show solution

\text{Bias}^2 + \Var = 0.45 - 0.40 = 0.05

Only 0.05 of the error is attributable to the model; the remaining 0.40 is noise in the target and no model can remove it.

Even a perfect model would score 0.40, so the best possible improvement is about 11% of the current error. Effort is better spent elsewhere — reducing label noise, finding features that genuinely reduce $\sigma^2$ by explaining part of what currently looks like noise, or accepting the model and shipping.

The broader lesson: without an estimate of the noise floor you cannot tell an excellent model from a mediocre one. Estimating it — via repeated measurements, annotator agreement, or a known physical limit — should precede a tuning campaign.

Next: Train, Validation and Test Splits, where we set up the machinery that lets these quantities be estimated honestly.