Advanced26 min

Conditional Expectation

E[X|Y] as a random variable, the tower property, and conditional variance decomposition.

Assumes you know

Expectation

Conditional Expectation

Intuition first

An average taken within a group. The average height of everyone is one number; the average height of everyone aged 12 is another. That second quantity is a conditional expectation.

The step that makes this subtle is treating the age itself as random. Before you know someone's age, "the average height of people their age" is not yet a number — it is a value that will be determined once the age is revealed. So $\E[X \mid Y]$ is a random variable, a function of $Y$ , and it has its own mean and its own variance.

Two results follow, and both are workhorses. Averaging those group averages, weighted by group size, recovers the overall average — the tower property. And the total variance splits cleanly into variation within groups plus variation between group means, which is the identity behind ANOVA and behind the bias–variance decomposition.

Two objects, carefully distinguished

\E[X \mid Y = y] \quad \text{is a \textbf{number} — the mean of } X \text{ among cases where } Y = y

\E[X \mid Y] \quad \text{is a \textbf{random variable} — the function } g(Y) \text{ where } g(y) = \E[X \mid Y = y]

Notation used in this lesson
Symbol	Meaning	Read aloud
E[X \| Y = y]	A number: the mean of X within the group Y = y	E of X given Y equals y
E[X \| Y]	A random variable: that mean, as a function of the random Y	E of X given Y
Var(X \| Y)	Conditional variance, also a random variable	variance of X given Y

For discrete variables:

\E[X \mid Y = y] = \sum_x x\, \Prob(X = x \mid Y = y)

The tower property

\boxed{\E\big[\E[X \mid Y]\big] = \E[X]}

Also called the law of total expectation or the law of iterated expectations. Averaging the group averages, weighted by how likely each group is, gives the overall average.

Proof of the tower propertyAdvanced

Work discretely. The outer expectation averages over the possible values of $Y$ :

\E\big[\E[X \mid Y]\big] = \sum_y \E[X \mid Y = y]\,\Prob(Y = y)

Expand the inner conditional expectation by its definition:

= \sum_y \left(\sum_x x\,\Prob(X = x \mid Y = y)\right)\Prob(Y = y)

Move $\Prob(Y=y)$ inside the inner sum and use the multiplication rule $\Prob(X = x \mid Y = y)\Prob(Y = y) = \Prob(X = x, Y = y)$ :

= \sum_y \sum_x x\, \Prob(X = x, Y = y)

Swap the order of summation and marginalise over $y$ :

= \sum_x x \sum_y \Prob(X = x, Y = y) = \sum_x x\, \Prob(X = x) = \E[X]

The whole proof is the multiplication rule followed by marginalisation. It is the law of total probability with values attached.

Why conditioning is a prediction

E[X | Y] minimises expected squared errorAdvanced

Among all functions $g(Y)$ , which minimises $\E\big[(X - g(Y))^2\big]$ ?

Write $m(Y) = \E[X \mid Y]$ and decompose:

X - g(Y) = \underbrace{\big(X - m(Y)\big)}_{A} + \underbrace{\big(m(Y) - g(Y)\big)}_{B}

Square and take expectations:

\E\big[(X - g(Y))^2\big] = \E[A^2] + 2\E[AB] + \E[B^2]

The cross term vanishes. Condition on $Y$ first — inside that conditioning, $B$ is a constant, so it factors out:

\E[AB \mid Y] = B \cdot \E\big[X - m(Y) \mid Y\big] = B \cdot \big(\E[X \mid Y] - m(Y)\big) = B \cdot 0 = 0

By the tower property $\E[AB] = \E\big[\E[AB \mid Y]\big] = 0$ . Therefore

\E\big[(X - g(Y))^2\big] = \underbrace{\E\big[(X - m(Y))^2\big]}_{\text{fixed}} + \underbrace{\E\big[(m(Y) - g(Y))^2\big]}_{\geq 0}

The first term does not involve $g$ ; the second is non-negative and is zero exactly when $g = m$ . So $g(Y) = \E[X \mid Y]$ is the minimiser.

This is the theoretical target of regression. A squared-error regression model is estimating $\E[Y \mid X]$ , which is why it predicts conditional means and why it is pulled around by outliers — the mean is.

The variance decomposition

\boxed{\Var(X) = \underbrace{\E\big[\Var(X \mid Y)\big]}_{\text{within-group}} + \underbrace{\Var\big(\E[X \mid Y]\big)}_{\text{between-group}}}

Sometimes called Eve's law, or the law of total variance.

Solved problem 1 · Both laws on a two-machine factory

Machine A produces 70% of items with mean weight 50 g and variance 4. Machine B produces 30% with mean weight 55 g and variance 9. An item is selected at random.

Find the overall mean and variance of its weight.

Step 1 — set up the conditioning variable

Let $M$ be the machine, $W$ the weight.

\Prob(M = A) = 0.7, \qquad \Prob(M = B) = 0.3

\E[W \mid A] = 50, \quad \Var(W \mid A) = 4

\E[W \mid B] = 55, \quad \Var(W \mid B) = 9

Step 2 — overall mean by the tower property

\E[W] = \E[W \mid A]\Prob(A) + \E[W \mid B]\Prob(B)

= 50(0.7) + 55(0.3) = 35 + 16.5 = 51.5 \text{ g}

Step 3 — within-group variance component

\E\big[\Var(W \mid M)\big] = 4(0.7) + 9(0.3) = 2.8 + 2.7 = 5.5

This is the average spread inside a machine's output.

Step 4 — between-group variance component

$\E[W \mid M]$ is a random variable taking value 50 with probability 0.7 and 55 with probability 0.3. Its variance:

\E\big[\E[W\mid M]^2\big] = 50^2(0.7) + 55^2(0.3) = 2500(0.7) + 3025(0.3) = 1750 + 907.5 = 2657.5

\Var\big(\E[W \mid M]\big) = 2657.5 - (51.5)^2 = 2657.5 - 2652.25 = 5.25

Step 5 — combine

\Var(W) = 5.5 + 5.25 = 10.75

So $\sigma_W = \sqrt{10.75} \approx 3.279$ g.

Step 6 — interpret the split

Within-group variation is 5.5 and between-group is 5.25 — almost equal. Roughly half the total variability in item weight comes from the machines differing in their averages, and half from scatter within each machine.

That is actionable: calibrating the two machines to a common mean would remove about 49% of total variance, while tightening each machine's own tolerance would address the other half.

Answer

$\E[W] = 51.5$ g and $\Var(W) = 10.75$ , decomposing as $5.5$ within-machine plus $5.25$ between-machine.

Solved problem 2 · Conditioning to avoid a hard calculation

A hen lays $N$ eggs, where $N \sim \text{Poisson}(\lambda = 6)$ . Each egg hatches independently with probability $p = 0.8$ . Let $H$ be the number that hatch. Find $\E[H]$ and $\Var(H)$ .

Step 1 — why direct calculation is unpleasant

$H$ has a compound distribution — a binomial whose number of trials is itself random. Writing its PMF requires summing over all possible $N$ :

\Prob(H = h) = \sum_{n \geq h} \binom{n}{h}p^h(1-p)^{n-h}\,\frac{e^{-\lambda}\lambda^n}{n!}

Doable, but conditioning is far easier.

Step 2 — condition on N

Given $N = n$ , the number hatching is binomial:

\E[H \mid N] = Np, \qquad \Var(H \mid N) = Np(1-p)

Step 3 — mean by the tower property

\E[H] = \E\big[\E[H \mid N]\big] = \E[Np] = p\,\E[N] = 0.8 \times 6 = 4.8

Step 4 — within-group variance term

\E\big[\Var(H \mid N)\big] = \E\big[Np(1-p)\big] = p(1-p)\E[N] = 0.8(0.2)(6) = 0.96

Step 5 — between-group variance term

For Poisson, $\Var(N) = \lambda = 6$ :

\Var\big(\E[H \mid N]\big) = \Var(Np) = p^2 \Var(N) = 0.64 \times 6 = 3.84

Step 6 — total

\Var(H) = 0.96 + 3.84 = 4.8

Step 7 — recognise the result

$\E[H] = \Var(H) = 4.8$ . Equal mean and variance is the signature of a Poisson distribution, and indeed $H \sim \text{Poisson}(\lambda p) = \text{Poisson}(4.8)$ .

This is Poisson thinning: independently keeping each event of a Poisson process with probability $p$ yields a Poisson process with rate $\lambda p$ . Conditioning gave us the mean and variance in six lines; the general theorem is worth remembering separately.

Answer

$\E[H] = 4.8$ and $\Var(H) = 4.8$ , with $H \sim \text{Poisson}(4.8)$ .

python

import numpy as np

rng = np.random.default_rng(0)
n = 2_000_000

# Factory example.
machine = rng.random(n) < 0.7                       # True = machine A
weight = np.where(machine,
                  rng.normal(50, 2, n),             # var 4
                  rng.normal(55, 3, n))             # var 9
print(f"E[W]   {weight.mean():.4f}   theory 51.5")
print(f"Var(W) {weight.var():.4f}   theory 10.75")
print(f"  within  {4*0.7 + 9*0.3:.4f}")
print(f"  between {50**2*0.7 + 55**2*0.3 - 51.5**2:.4f}")

# Poisson thinning.
N = rng.poisson(6, n)
H = rng.binomial(N, 0.8)
print(f"\nE[H]   {H.mean():.4f}   Var(H) {H.var():.4f}   both theory 4.8")

Exercise 1

$\E[X \mid Y] = 2Y$ and $Y$ has mean 5 and variance 3. Find $\E[X]$ .

Show solution

By the tower property and then linearity:

\E[X] = \E\big[\E[X \mid Y]\big] = \E[2Y] = 2\,\E[Y] = 2 \times 5 = 10

The variance of $Y$ is not needed for the mean — it would be needed for $\Var(X)$ , which also requires $\Var(X \mid Y)$ , information not supplied here. Without it you can compute only the between-group component, $\Var(2Y) = 4 \times 3 = 12$ , which is a lower bound on $\Var(X)$ .

Exercise 2

A coin has unknown bias $P$ , uniform on $[0,1]$ . Given $P = p$ , you flip it 10 times. Find the expected number of heads.

Show solution

Let $H$ be the number of heads. Given $P = p$ , $H \sim \text{Binomial}(10, p)$ , so

\E[H \mid P] = 10P

By the tower property:

\E[H] = \E[10P] = 10\,\E[P]

For $P \sim \text{Uniform}(0,1)$ , $\E[P] = 1/2$ , giving

\E[H] = 10 \times \tfrac12 = 5

The same answer as a fair coin — but the distribution is completely different. For a fair coin, $\Var(H) = 10(0.5)(0.5) = 2.5$ . Here, applying the variance decomposition:

\E\big[\Var(H\mid P)\big] = \E[10P(1-P)] = 10\big(\E[P] - \E[P^2]\big) = 10\left(\tfrac12 - \tfrac13\right) = \tfrac{10}{6} \approx 1.667

\Var\big(\E[H\mid P]\big) = \Var(10P) = 100 \times \tfrac{1}{12} \approx 8.333

\Var(H) = 1.667 + 8.333 = 10

Four times the variance of a fair coin, because the unknown bias adds substantial between-group spread. In fact $H$ is uniform on $\{0, 1, \dots, 10\}$ here — a genuinely surprising result, and the Bayesian reason a uniform prior on $p$ is sometimes called uninformative about the count.

Next: Bernoulli and Binomial Distributions.