Intermediate24 min

Covariance and Correlation

Measuring joint variation, Pearson correlation, and why zero correlation does not imply independence.

Assumes you know

Variance, Moments and Generating Functions

Covariance and Correlation

Intuition first

Variance measures how one variable spreads. Covariance measures whether two variables spread together.

The idea is simple: for each observation, look at whether $X$ is above or below its mean, and whether $Y$ is above or below its mean. Multiply the two deviations. If they usually share a sign — both high or both low — the products are positive and the average is positive. If one is typically high when the other is low, the products are negative.

The awkwardness is units. Covariance between height in centimetres and weight in kilograms is in centimetre-kilograms, which means nothing to anyone and changes if you switch to inches. Correlation fixes that by dividing out both standard deviations, producing a unitless number always between $-1$ and $+1$ .

And the essential caveat, which the last solved problem makes concrete: correlation measures linear association only. Two variables can be perfectly determined by one another and have correlation exactly zero.

Definitions

\Cov(X, Y) = \E\big[(X - \mu_X)(Y - \mu_Y)\big]

with the computational form, derived exactly as for variance:

\Cov(X, Y) = \E[XY] - \E[X]\,\E[Y]

Correlation standardises it:

\rho_{XY} = \frac{\Cov(X, Y)}{\sigma_X \sigma_Y}

Notation used in this lesson
Symbol	Meaning	Read aloud
Cov(X,Y)	Covariance — in units of X times units of Y	covariance of X and Y
ρ or ρₓᵥ	Pearson correlation coefficient, between −1 and 1	rho
σₓ, σᵥ	Standard deviations of X and Y	sigma X, sigma Y
Σ	Covariance matrix for a random vector	sigma (matrix)

Properties

\Cov(X, Y) = \Cov(Y, X) \qquad \text{(symmetric)}

\Cov(aX + b,\; cY + d) = ac\,\Cov(X, Y) \qquad \text{(constants shift, multipliers scale)}

\Var(X + Y) = \Var(X) + \Var(Y) + 2\Cov(X, Y)

The last identity is why covariance matters practically: it is the correction term that makes variances add or fail to add.

Why correlation is bounded by 1

The Cauchy–Schwarz argumentAdvanced

For any real $t$ , the variance of $X - tY$ is non-negative because it is the expectation of a square:

0 \leq \Var(X - tY) = \Var(X) - 2t\,\Cov(X,Y) + t^2 \Var(Y)

Read the right-hand side as a quadratic in $t$ :

q(t) = \Var(Y)\,t^2 - 2\Cov(X,Y)\,t + \Var(X)

A quadratic that is never negative has a non-positive discriminant:

\big(2\Cov(X,Y)\big)^2 - 4\,\Var(Y)\Var(X) \leq 0

\Cov(X,Y)^2 \leq \Var(X)\Var(Y)

Take square roots:

\lvert \Cov(X,Y) \rvert \leq \sigma_X \sigma_Y \quad\Longrightarrow\quad \lvert \rho \rvert \leq 1

When is $\rho = \pm 1$ ? Equality in the discriminant means $q(t)$ has a repeated root $t^*$ with $q(t^*) = 0$ , so $\Var(X - t^*Y) = 0$ , so $X - t^*Y$ is constant. That is $X = t^*Y + c$ — an exact linear relationship.

So $\lvert \rho \rvert = 1$ if and only if $X$ and $Y$ are exactly linearly related, with the sign of $\rho$ giving the sign of the slope.

Independence implies zero correlation — not the converse

If $X \ind Y$ then $\E[XY] = \E[X]\E[Y]$ , so $\Cov(X,Y) = 0$ and $\rho = 0$ .

The reverse fails, and this is the most important caveat in the lesson.

Solved problem 1 · Zero correlation with perfect dependence

Let $X$ take values $-1, 0, 1$ each with probability $1/3$ , and set $Y = X^2$ . Show $\rho = 0$ despite $Y$ being a deterministic function of $X$ .

Step 1 — the joint distribution

$Y = X^2$ , so:

$X$	$Y$	Probability
$-1$	1	1/3
0	0	1/3
1	1	1/3

Step 2 — the means

\E[X] = \tfrac13(-1) + \tfrac13(0) + \tfrac13(1) = 0

\E[Y] = \tfrac13(1) + \tfrac13(0) + \tfrac13(1) = \tfrac23

Step 3 — E[XY]

XY = X \cdot X^2 = X^3

\E[XY] = \E[X^3] = \tfrac13(-1) + \tfrac13(0) + \tfrac13(1) = 0

Step 4 — covariance

\Cov(X,Y) = \E[XY] - \E[X]\E[Y] = 0 - (0)\left(\tfrac23\right) = 0

Hence $\rho = 0$ .

Step 5 — confirm the dependence is total

$Y$ is completely determined by $X$ : observing $X = 0$ tells you $Y = 0$ with certainty. Check independence formally:

\Prob(X = 0, Y = 0) = \tfrac13

\Prob(X = 0)\,\Prob(Y = 0) = \tfrac13 \times \tfrac13 = \tfrac19

$\tfrac13 \neq \tfrac19$ , so they are emphatically not independent.

Step 6 — why correlation missed it

The relationship is symmetric about zero: increasing $\lvert X \rvert$ increases $Y$ , but $X$ going up from $-1$ to $0$ decreases $Y$ while $X$ going up from $0$ to $1$ increases it. The positive and negative contributions cancel exactly, leaving zero linear association.

Answer

$\rho = 0$ with $Y$ a deterministic function of $X$ . Zero correlation means no linear association; it does not mean independence. The implication runs one way only.

Four datasets. The first three have progressively weaker linear association. The fourth has correlation near zero and an obvious, strong, non-linear relationship — which is what correlation cannot see.

Solved problem 2 · Covariance from a joint table

$X$ and $Y$ have this joint distribution:

	$Y = 0$	$Y = 1$	$\Prob(X)$
$X = 0$	0.30	0.20	0.50
$X = 1$	0.10	0.40	0.50
$\Prob(Y)$	0.40	0.60	1.00

Find $\Cov(X,Y)$ and $\rho$ .

Step 1 — marginal means

Both are Bernoulli, so the mean is the probability of 1:

\E[X] = 0.50, \qquad \E[Y] = 0.60

Step 2 — E[XY]

$XY = 1$ only when both are 1, which has probability 0.40:

\E[XY] = 1 \times 0.40 = 0.40

Step 3 — covariance

\Cov(X,Y) = 0.40 - (0.50)(0.60) = 0.40 - 0.30 = 0.10

Positive: $X$ and $Y$ tend to be high together.

Step 4 — standard deviations

For Bernoulli( $p$ ), $\Var = p(1-p)$ :

\Var(X) = 0.5 \times 0.5 = 0.25 \quad\Longrightarrow\quad \sigma_X = 0.50

\Var(Y) = 0.6 \times 0.4 = 0.24 \quad\Longrightarrow\quad \sigma_Y = \sqrt{0.24} \approx 0.4899

Step 5 — correlation

\rho = \frac{0.10}{0.50 \times 0.4899} = \frac{0.10}{0.24495} \approx 0.4082

Step 6 — cross-check with conditional probabilities

\Prob(Y = 1 \mid X = 1) = \frac{0.40}{0.50} = 0.80, \qquad \Prob(Y = 1 \mid X = 0) = \frac{0.20}{0.50} = 0.40

Knowing $X = 1$ doubles the chance that $Y = 1$ , consistent with a moderate positive correlation.

Answer

$\Cov(X,Y) = 0.10$ and $\rho \approx 0.408$ — a moderate positive linear association.

The covariance matrix

For a random vector $\mathbf{X} = (X_1, \dots, X_d)$ :

\Sigma_{ij} = \Cov(X_i, X_j), \qquad \Sigma = \E\big[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^{\mathsf{T}}\big]

Variances sit on the diagonal, covariances off it. Two properties matter downstream:

Symmetric, since $\Cov(X_i, X_j) = \Cov(X_j, X_i)$ .
Positive semi-definite, because for any vector $a$ , $a^{\mathsf{T}}\Sigma a = \Var(a^{\mathsf{T}}\mathbf{X}) \geq 0$ .

Those two facts are what let PCA diagonalise $\Sigma$ by eigendecomposition, and what make the multivariate normal density well defined.

python

import numpy as np

rng = np.random.default_rng(0)

# The zero-correlation-yet-dependent example.
X = rng.choice([-1, 0, 1], size=300_000)
Y = X ** 2
print(f"corr(X, X²)     {np.corrcoef(X, Y)[0,1]:+.4f}   <- ~0, yet Y is determined by X")

# Correlation is invariant to linear rescaling; covariance is not.
A = rng.normal(0, 1, 100_000)
B = 2 * A + rng.normal(0, 1, 100_000)
print(f"\ncov(A,B)        {np.cov(A, B)[0,1]:.4f}")
print(f"cov(100A, B)    {np.cov(100*A, B)[0,1]:.4f}   <- scaled by 100")
print(f"corr(A,B)       {np.corrcoef(A, B)[0,1]:.4f}")
print(f"corr(100A, B)   {np.corrcoef(100*A, B)[0,1]:.4f}   <- unchanged")

# Variance of a sum needs the covariance term.
print(f"\nVar(A+B) empirical {np.var(A+B):.4f}")
print(f"Var(A)+Var(B)      {np.var(A)+np.var(B):.4f}  <- too small")
print(f"+ 2Cov(A,B)        {np.var(A)+np.var(B)+2*np.cov(A,B)[0,1]:.4f}  <- matches")

Exercise 1

$\Var(X) = 16$ , $\Var(Y) = 25$ , $\Cov(X,Y) = 12$ . Find $\rho$ and $\Var(X + Y)$ .

Show solution

\rho = \frac{12}{\sqrt{16}\sqrt{25}} = \frac{12}{4 \times 5} = \frac{12}{20} = 0.60

\Var(X+Y) = 16 + 25 + 2(12) = 41 + 24 = 65

Worth noting the size of the correction. Ignoring covariance would give 41, understating the variance by 37%. And check the bound: $\lvert \Cov \rvert \leq \sigma_X\sigma_Y = 20$ , so the largest possible covariance here is 20, giving $\rho = 1$ and $\Var(X+Y) = 81$ .

Exercise 2

Prove $\Var(X - Y) = \Var(X) + \Var(Y) - 2\Cov(X,Y)$ .

Show solution

Write $X - Y = X + (-1)Y$ and apply the variance-of-a-sum identity:

\Var\big(X + (-1)Y\big) = \Var(X) + \Var\big((-1)Y\big) + 2\Cov\big(X, (-1)Y\big)

Two substitutions. The scaling rule gives

\Var\big((-1)Y\big) = (-1)^2 \Var(Y) = \Var(Y)

and the bilinearity of covariance gives

\Cov\big(X, (-1)Y\big) = (-1)\Cov(X, Y) = -\Cov(X,Y)

Substituting:

\Var(X - Y) = \Var(X) + \Var(Y) - 2\Cov(X,Y)

The sign flips on the covariance term but not on the variance term, because variance carries a squared multiplier and covariance a linear one. This is why positively correlated variables have a less variable difference — the shared fluctuation cancels — which is the entire basis of paired experimental designs and of hedging.

Next: Bernoulli and Binomial Distributions, the first named distributions.