Intermediate28 min

ROC and Precision–Recall Curves

Threshold sweeps, AUC interpretation, and why PR curves beat ROC under heavy imbalance.

Assumes you know

Classification Metrics

ROC and Precision–Recall Curves

Intuition first

A classifier outputs scores. Where you cut those scores into "yes" and "no" is a separate decision from how good the scores are.

Cut low and you catch almost everything but flag a lot of innocents. Cut high and everything you flag is real but you miss most cases. Neither setting tells you whether the ranking underneath is any good.

A curve sweeps the cut point across its whole range and plots what happens. If the scores rank positives above negatives well, the curve bows sharply towards the good corner regardless of where you eventually cut. If the scores are noise, the curve is a diagonal.

Two curves are in common use, and choosing between them matters far more than most people realise: on heavily imbalanced data they can tell completely different stories about the same model.

The ROC curve

Plots the true positive rate against the false positive rate as the threshold sweeps from high to low.

\text{TPR} = \frac{TP}{TP + FN} = \text{recall}, \qquad \text{FPR} = \frac{FP}{FP + TN} = 1 - \text{specificity}

ROC space. The diagonal is random guessing; the top-left corner is perfect. Curve A dominates curve B everywhere, so A is better at every threshold.

AUC is the area under this curve. It has an exact probabilistic meaning, which is the reason it is worth knowing.

AUC equals the probability of correct rankingAdvanced

Claim: AUC is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative.

\text{AUC} = \Prob\big(s(x^+) > s(x^-)\big)

Sketch. Sweep the threshold $t$ from $+\infty$ downwards. At threshold $t$ ,

\text{TPR}(t) = \Prob\big(s(x^+) \geq t\big), \qquad \text{FPR}(t) = \Prob\big(s(x^-) \geq t\big)

The area under the ROC curve is $\int_0^1 \text{TPR}\, d(\text{FPR})$ . Substituting the distribution of negative scores for the measure $d(\text{FPR})$ turns the integral into

\int \Prob\big(s(x^+) \geq t\big)\, p_{-}(t)\, dt = \E_{t \sim p_-}\Big[\Prob\big(s(x^+) \geq t\big)\Big] = \Prob\big(s(x^+) > s(x^-)\big)

where $p_-$ is the density of scores on negatives. Ties contribute one half.

Two useful consequences fall out immediately:

AUC $= 0.5$ is random ranking, and AUC below 0.5 means the scores are anti-correlated with the label — invert them and you have a useful model.
AUC is invariant to any monotone transformation of the scores. Passing scores through a sigmoid, or squaring positive scores, leaves AUC untouched. It measures ranking only and says nothing about whether the numbers are calibrated probabilities.

The precision–recall curve

Plots precision against recall, again as the threshold sweeps.

\text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN}

Note what is absent: $TN$ appears nowhere. That single fact explains why PR curves behave so differently on imbalanced data.

Why the choice matters on imbalanced data

FPR divides by the number of negatives. When negatives vastly outnumber positives, a large absolute number of false positives is a small FPR — so the ROC curve barely notices them. Precision divides by the number of predictions, so those same false positives destroy it.

Solved problem 1 · The same model, two very different stories

A dataset has 1,000 positives and 100,000 negatives — a 1% positive rate. At a particular threshold a model achieves $TP = 900$ and $FP = 9{,}000$ .

Compute the ROC coordinates and the precision, and explain what each suggests.

Step 1 — fill in the confusion matrix

FN = 1{,}000 - 900 = 100

TN = 100{,}000 - 9{,}000 = 91{,}000

Step 2 — ROC coordinates

\text{TPR} = \frac{900}{900 + 100} = \frac{900}{1{,}000} = 0.900

\text{FPR} = \frac{9{,}000}{9{,}000 + 91{,}000} = \frac{9{,}000}{100{,}000} = 0.090

The point $(0.090,\, 0.900)$ sits close to the top-left corner. In ROC terms this looks excellent.

Step 3 — precision at the same threshold

\text{Precision} = \frac{900}{900 + 9{,}000} = \frac{900}{9{,}900} \approx 0.0909

Step 4 — reconcile the two views

Both numbers describe the identical 9,000 false positives.

ROC divides them by 100,000 negatives, giving a reassuring 0.09. Precision divides them by the 9,900 flagged cases, giving 0.09 — and here that means 91% of everything the model flags is wrong.

Operationally: a reviewer works through 9,900 cases to find 900 real ones, opening eleven files per genuine hit.

Step 5 — the baseline each metric is measured against

Random guessing gives AUC $= 0.5$ , so ROC has a fixed, class-balance-independent baseline.

The PR baseline is the positive rate itself, here $1{,}000/101{,}000 \approx 0.0099$ . So precision of $0.0909$ is about 9× better than random — genuinely informative, just not comfortable.

Answer

$\text{TPR} = 0.900$ , $\text{FPR} = 0.090$ , precision $\approx 0.0909$ .

ROC says "near the perfect corner"; PR says "9 out of 10 alerts are false". Both are correct. Report PR when positives are rare and the cost of a false positive falls on a human.

Comparing the two curves

	ROC / AUC	Precision–Recall / AP
Axes	TPR vs FPR	Precision vs Recall
Uses TN?	Yes	No
Baseline	0.5 always	Positive rate — changes with balance
Insensitive to class balance	Yes	No
Good when	Classes roughly balanced; both errors matter	Positives rare; false positives costly
Misleads when	Heavy imbalance	Comparing across datasets with different balance

Average precision (AP) summarises the PR curve as a weighted mean of precisions:

\text{AP} = \sum_{k} \big(R_k - R_{k-1}\big)\, P_k

Computing both

python

import numpy as np
from sklearn.metrics import (
    roc_auc_score, average_precision_score, roc_curve, precision_recall_curve,
)

rng = np.random.default_rng(0)

# 1% positive rate, mildly separable scores.
n_pos, n_neg = 1_000, 100_000
y_true = np.concatenate([np.ones(n_pos), np.zeros(n_neg)])
scores = np.concatenate([
    rng.normal(1.6, 1.0, n_pos),
    rng.normal(0.0, 1.0, n_neg),
])

auc = roc_auc_score(y_true, scores)
ap = average_precision_score(y_true, scores)
baseline_ap = y_true.mean()

print(f"ROC AUC          {auc:.4f}")
print(f"Average precision {ap:.4f}   (random baseline {baseline_ap:.4f})")

# Precision at the threshold giving 90% recall — the operational question.
prec, rec, thr = precision_recall_curve(y_true, scores)
i = np.argmin(np.abs(rec - 0.90))
print(f"at recall {rec[i]:.3f}: precision {prec[i]:.4f}, threshold {thr[min(i, len(thr)-1)]:.3f}")

Run it and the pattern from the solved problem appears: AUC looks strong while average precision sits far lower, because AP is judged against a 0.0099 baseline rather than 0.5.

Exercise 1

Model P has AUC 0.91 and AP 0.34. Model Q has AUC 0.88 and AP 0.52. The positive rate is 3%. Which do you deploy for a fraud queue with limited reviewer capacity?

Show solution

Model Q.

With a capacity-limited review queue, the quantity that matters is how many of the cases sent for review are genuine — precision at the operating threshold. AP summarises precision across thresholds, and Q's 0.52 against P's 0.34 means roughly half of Q's alerts are real versus a third of P's.

P's higher AUC reflects better ranking averaged over the whole threshold range, including regions of very high recall that a capacity-limited queue will never operate in. AUC weights all thresholds; the business only uses one.

Both beat the 0.03 baseline substantially, so both models are informative. The decision turns on the operating region, and in the high-precision region Q is better.

Worth doing before committing: plot both PR curves and compare precision at the exact recall your capacity implies, since AP is an average and the curves may cross.

Exercise 2

Explain why AUC is unchanged if you replace every predicted probability $\hat{p}$ with $\hat{p}^3$ , but $F_1$ at a fixed threshold of 0.5 does change.

Show solution

Cubing is strictly increasing on $[0, 1]$ , so it preserves the order of the scores. If $\hat p_a > \hat p_b$ then $\hat p_a^3 > \hat p_b^3$ . Since AUC is the probability that a random positive outranks a random negative, and every pairwise comparison is unchanged, AUC is identical.

$F_1$ at a fixed threshold depends on which examples fall above the cut, not on their order. Cubing compresses values towards zero: a score of $0.7$ becomes $0.343$ , and $0.8$ becomes $0.512$ . An example that was above 0.5 is now below it.

So the confusion matrix changes, and with it precision, recall and $F_1$ — even though the model's ranking ability is untouched.

The general lesson: threshold-free metrics measure ranking, threshold-dependent metrics measure ranking and calibration. If a transformation changes your $F_1$ but not your AUC, the model did not get better or worse — the threshold simply no longer sits where you thought. That is what the calibration lesson addresses.

Next: Regression Metrics, where the target is continuous and the choice of metric decides which errors you are implicitly willing to make.