Beginner24 min

Regression Metrics

MSE, RMSE, MAE, MAPE, R² and adjusted R², and which to report for which audience.

Assumes you know

Train, Validation and Test Splits

Regression Metrics

Intuition first

When the target is a number, "wrong" has a size. A prediction off by £2 and one off by £2,000 are both errors, and any metric has to decide how much worse the second one is.

Squared error says it is a million times worse, because it squares the gap. Absolute error says it is a thousand times worse, in direct proportion. That single choice changes which model wins, which outliers dominate the fit, and — less obviously — what quantity the model ends up predicting: squared error pulls predictions towards the mean of the plausible outcomes, absolute error towards the median.

So the metric is not a scoring afterthought. It determines the model's behaviour.

Notation used in this lesson
Symbol	Meaning	Read aloud
yᵢ	True value of example i	y i
ŷᵢ	Predicted value of example i	y hat i
ȳ	Mean of the true values	y bar
eᵢ	Residual, yᵢ − ŷᵢ	e i
n	Number of examples	n
p	Number of predictors in the model	p

The core metrics

\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}\big(y_i - \hat{y}_i\big)^2 \qquad \text{RMSE} = \sqrt{\text{MSE}}

\text{MAE} = \frac{1}{n}\sum_{i=1}^{n}\big\lvert y_i - \hat{y}_i \big\rvert

\text{MAPE} = \frac{100}{n}\sum_{i=1}^{n}\left\lvert \frac{y_i - \hat{y}_i}{y_i} \right\rvert

R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}

Metric	Units	Outlier sensitivity	Predicts
MSE	squared target units	Very high	Mean
RMSE	target units	High	Mean
MAE	target units	Low	Median
MAPE	percent	High for small $y$	—
$R^2$	unitless	High	—

Why the choice changes the model

Squared error targets the mean, absolute error the medianAdvanced

Suppose you must predict a single constant $c$ for a random target $Y$ .

Under squared loss, minimise $g(c) = \E[(Y - c)^2]$ . Differentiate and set to zero:

g'(c) = \E\big[-2(Y - c)\big] = -2\big(\E[Y] - c\big) = 0 \quad\Longrightarrow\quad c = \E[Y]

The second derivative is $2 > 0$ , so this is the minimum. The optimal constant is the mean.

Under absolute loss, minimise $h(c) = \E\lvert Y - c \rvert$ . The derivative of $\lvert y - c \rvert$ with respect to $c$ is $-1$ when $y > c$ and $+1$ when $y < c$ , so

h'(c) = \Prob(Y < c) - \Prob(Y > c) = 0 \quad\Longrightarrow\quad \Prob(Y < c) = \Prob(Y > c)

which is the definition of the median.

This generalises to conditional predictions: a model fitted with squared loss estimates $\E[Y \mid x]$ , and one fitted with absolute loss estimates the conditional median. On a skewed target — income, claim size, delivery time — those are materially different numbers, and the difference is not an artefact but the correct answer to two different questions.

RMSE versus MAE, concretely

Solved problem 1 · One outlier, two verdicts

Two models predict house prices in thousands for five houses.

House	Actual	Model A	Model B
1	200	210	202
2	250	240	248
3	300	310	298
4	350	340	352
5	900	620	500

Compute MAE and RMSE for both, and decide which model is better.

Step 1 — Model A residuals and absolute errors

e = y - \hat{y}: \quad -10,\; +10,\; -10,\; +10,\; +280

\lvert e \rvert: \quad 10,\; 10,\; 10,\; 10,\; 280

\text{MAE}_A = \frac{10 + 10 + 10 + 10 + 280}{5} = \frac{320}{5} = 64.0

Step 2 — Model A squared errors

e^2: \quad 100,\; 100,\; 100,\; 100,\; 78{,}400

\text{MSE}_A = \frac{100 + 100 + 100 + 100 + 78{,}400}{5} = \frac{78{,}800}{5} = 15{,}760

\text{RMSE}_A = \sqrt{15{,}760} \approx 125.5

Step 3 — Model B residuals and absolute errors

e: \quad -2,\; +2,\; +2,\; -2,\; +400

\lvert e \rvert: \quad 2,\; 2,\; 2,\; 2,\; 400

\text{MAE}_B = \frac{2 + 2 + 2 + 2 + 400}{5} = \frac{408}{5} = 81.6

Step 4 — Model B squared errors

e^2: \quad 4,\; 4,\; 4,\; 4,\; 160{,}000

\text{MSE}_B = \frac{4 + 4 + 4 + 4 + 160{,}000}{5} = \frac{160{,}016}{5} = 32{,}003.2

\text{RMSE}_B = \sqrt{32{,}003.2} \approx 178.9

Step 5 — compare

	MAE	RMSE
Model A	64.0	125.5
Model B	81.6	178.9

Model A wins on both — but for different reasons, and the gap differs sharply. On MAE A is better by 22%; on RMSE by 30%, because RMSE punishes B's larger single miss more heavily.

Step 6 — the detail both metrics hide

On the four ordinary houses, Model B is dramatically better: errors of 2 against A's 10, a five-fold improvement. Both aggregate metrics are dominated by house 5, which is an outlier in the target (900 against a 200–350 range), not necessarily a modelling failure.

So the honest report is: B is five times more accurate on typical houses and worse on the one atypical house. Which model to deploy depends on whether £900k houses are part of the intended use.

Answer

$\text{MAE}_A = 64.0$ , $\text{RMSE}_A \approx 125.5$ ; $\text{MAE}_B = 81.6$ , $\text{RMSE}_B \approx 178.9$ . Model A wins on both aggregates, but B is five times better on the four in-range houses. Segment the evaluation before choosing.

R² and what it does not mean

$R^2$ compares your model against the simplest possible baseline: always predicting $\bar y$ .

R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}

$R^2 = 1$ — perfect predictions.
$R^2 = 0$ — no better than predicting the mean.
$R^2 < 0$ — worse than predicting the mean. Entirely possible on a test set, and a clear signal that something is wrong.

Adjusted $R^2$ penalises adding predictors that do not help:

R^2_{\text{adj}} = 1 - \big(1 - R^2\big)\frac{n - 1}{n - p - 1}

Plain $R^2$ never decreases when you add a variable, even a random one, which makes it useless for comparing models of different size on training data.

Huber loss: the compromise

Squared near zero, linear in the tails — differentiable everywhere, and robust to outliers:

L_\delta(e) = \begin{cases} \tfrac{1}{2}e^2 & \text{if } \lvert e \rvert \leq \delta \\[4pt] \delta\big(\lvert e \rvert - \tfrac{1}{2}\delta\big) & \text{otherwise} \end{cases}

Loss against residual. Squared error grows without bound; absolute error has a kink at zero that complicates optimisation; Huber is smooth at zero and linear in the tails.

Computing them

python

import numpy as np
from sklearn.metrics import (
    mean_squared_error, mean_absolute_error, r2_score,
    mean_absolute_percentage_error,
)

y_true = np.array([200, 250, 300, 350, 900], dtype=float)
pred_a = np.array([210, 240, 310, 340, 620], dtype=float)
pred_b = np.array([202, 248, 298, 352, 500], dtype=float)

for name, pred in (("A", pred_a), ("B", pred_b)):
    mae = mean_absolute_error(y_true, pred)
    rmse = np.sqrt(mean_squared_error(y_true, pred))
    print(f"Model {name}: MAE={mae:6.1f}  RMSE={rmse:6.1f}  "
          f"ratio={rmse/mae:.2f}  R²={r2_score(y_true, pred):.3f}  "
          f"MAPE={mean_absolute_percentage_error(y_true, pred)*100:5.1f}%")

# Segment the evaluation: typical houses versus the outlier.
mask = y_true < 500
print("\nin-range houses only:")
for name, pred in (("A", pred_a), ("B", pred_b)):
    print(f"  Model {name}: MAE={mean_absolute_error(y_true[mask], pred[mask]):.1f}")

The final two lines are the ones that change the decision, and no single aggregate metric would have surfaced them.

Exercise 1

A delivery-time model reports RMSE 12 minutes and MAE 4 minutes. What does the ratio tell you, and what would you investigate?

Show solution

$\text{RMSE}/\text{MAE} = 3.0$ , which is high. For errors of roughly equal size the ratio is near 1; for normally distributed errors it is about 1.25. A ratio of 3 means the squared term is dominated by a small number of very large errors.

So the model is usually accurate — typical miss is 4 minutes — but occasionally catastrophically wrong, and those rare cases are what RMSE is reporting.

What to investigate:

Plot the residual distribution, not just the summary. Identify the tail cases.
Look for a segment. Large errors often cluster: a particular city, courier, time of day, or order type. If so, the fix is a feature or a separate model, not a global one.
Check for label errors. A recorded delivery time of 400 minutes may be a timestamp bug rather than a genuine delay, in which case the model is being penalised for being right.
Decide which metric matches the business. If a 90-minute miss causes a refund and a 4-minute miss causes nothing, RMSE is the metric to optimise and MAE is comfortingly irrelevant.

Exercise 2

A model achieves $R^2 = 0.93$ on training data and $R^2 = -0.15$ on test data. Explain what happened and what negative $R^2$ means.

Show solution

Severe overfitting. The model fits the training sample almost perfectly and fails on new data.

Negative $R^2$ has a precise meaning: the residual sum of squares exceeds the total sum of squares, so

\sum_i (y_i - \hat y_i)^2 > \sum_i (y_i - \bar y)^2

That is, the model's predictions are worse than simply predicting the training mean for every test point. It has not merely failed to learn — it has learned something actively misleading, and applying it is worse than applying nothing.

A likely mechanism: a high-capacity model on too little data has fitted noise into coefficients that extrapolate badly, so test predictions swing far from any plausible value.

Immediate actions: shrink the model heavily (strong regularisation, fewer features), verify the split was done correctly, and check that preprocessing was fitted on the training fold only — a scaler fitted on train and applied to differently-distributed test data can produce exactly this signature.

Next: Probability Calibration, which addresses the gap the AUC lesson flagged: a model that ranks well can still output probabilities that are badly wrong.