Beginner24 min

Expectation

Expected value for discrete and continuous variables, linearity, and the law of the unconscious statistician.

Assumes you know

PMF, PDF and CDF

Expectation

Intuition first

The expected value is the long-run average. Roll a fair die many times and the mean of your rolls settles near 3.5 — even though 3.5 never appears on any face.

That last point is worth keeping: an expectation need not be an attainable value. It is the centre of mass of the distribution, the balance point if you placed weights equal to the probabilities along a ruler.

The property that makes expectation the workhorse of probability is linearity: the expected value of a sum is the sum of the expected values, always, whether or not the variables are independent. No other summary statistic is that well behaved, and problems that look hopeless directly often collapse once you write the quantity as a sum and take expectations term by term.

Definitions

\text{Discrete}: \quad \E[X] = \sum_{x} x\, p(x)

\text{Continuous}: \quad \E[X] = \int_{-\infty}^{\infty} x\, f(x)\, dx

Each value weighted by how likely it is.

Notation used in this lesson
Symbol	Meaning	Read aloud
E[X]	Expected value of X	E of X
μ or μₓ	Common alternative notation for E[X]	mu
E[g(X)]	Expected value of a function of X	E of g of X
E[X \| Y]	Conditional expectation — a separate lesson	E of X given Y

Linearity

For any random variables and any constants $a, b, c$ :

\E[aX + bY + c] = a\,\E[X] + b\,\E[Y] + c

Proof of linearity, and why independence is not neededAdvanced

Work with the discrete case; the continuous case replaces sums with integrals identically.

By definition, using the joint PMF $p(x,y)$ :

\E[X + Y] = \sum_x \sum_y (x + y)\, p(x, y)

Split the sum:

= \sum_x \sum_y x\, p(x,y) \;+\; \sum_x \sum_y y\, p(x,y)

In the first term $x$ does not depend on the inner index, so factor it out and marginalise:

\sum_x x \sum_y p(x,y) = \sum_x x\, p_X(x) = \E[X]

The step $\sum_y p(x,y) = p_X(x)$ is just marginalisation — summing a joint distribution over one variable. It holds for any joint distribution, dependent or not. The second term gives $\E[Y]$ symmetrically.

\E[X + Y] = \E[X] + \E[Y]

Independence never appeared. Contrast with the product rule, where it is essential:

\E[XY] = \E[X]\E[Y] \qquad \textbf{only if } X \ind Y

For dependent variables $\E[XY] = \E[X]\E[Y] + \Cov(X,Y)$ , and the covariance term is exactly what independence kills.

The law of the unconscious statistician

To find $\E[g(X)]$ you do not need the distribution of $g(X)$ — just weight $g(x)$ by the distribution of $X$ :

\E[g(X)] = \sum_x g(x)\, p(x) \qquad\text{or}\qquad \int g(x) f(x)\, dx

Solved problem 1 · Expectation of a die, and of its square

A fair six-sided die is rolled. Find $\E[X]$ , $\E[X^2]$ , and $\E[3X + 2]$ .

Step 1 — E[X] by definition

Each value has probability $1/6$ :

\E[X] = \frac{1}{6}(1 + 2 + 3 + 4 + 5 + 6) = \frac{21}{6} = 3.5

Step 2 — E[X²] using LOTUS

Weight the squares by the same probabilities:

\E[X^2] = \frac{1}{6}(1 + 4 + 9 + 16 + 25 + 36) = \frac{91}{6} \approx 15.1667

Step 3 — check that it is not the square of the mean

(\E[X])^2 = 3.5^2 = 12.25 \neq 15.1667

The difference $15.1667 - 12.25 = 2.9167$ is the variance of a fair die, $35/12$ .

Step 4 — E[3X + 2] by linearity

\E[3X + 2] = 3\,\E[X] + 2 = 3(3.5) + 2 = 10.5 + 2 = 12.5

Verify the hard way, computing the distribution of $3X + 2 \in \{5, 8, 11, 14, 17, 20\}$ :

\frac{5 + 8 + 11 + 14 + 17 + 20}{6} = \frac{75}{6} = 12.5 \;\checkmark

Answer

$\E[X] = 3.5$ ; $\E[X^2] = 91/6 \approx 15.167$ ; $\E[3X+2] = 12.5$ . Note 3.5 is not a face of the die — expectations need not be attainable.

Solved problem 2 · Linearity making a hard problem easy

Ten letters are placed at random into ten addressed envelopes, one per envelope. What is the expected number of letters in the correct envelope?

Step 1 — why the direct route is unpleasant

Let $X$ be the number of correct matches. Finding $\Prob(X = k)$ requires counting permutations with exactly $k$ fixed points, which involves derangements:

\Prob(X = k) = \frac{1}{k!}\sum_{j=0}^{10-k} \frac{(-1)^j}{j!}

Correct, and far more work than necessary.

Step 2 — write X as a sum of indicators

Define $\mathbb{1}_i = 1$ if letter $i$ lands in its own envelope, 0 otherwise. Then

X = \mathbb{1}_1 + \mathbb{1}_2 + \dots + \mathbb{1}_{10}

Step 3 — expectation of a single indicator

Letter $i$ is equally likely to land in any of the 10 envelopes, so

\E[\mathbb{1}_i] = \Prob(\text{letter } i \text{ correct}) = \frac{1}{10}

Step 4 — sum by linearity

\E[X] = \sum_{i=1}^{10} \E[\mathbb{1}_i] = 10 \times \frac{1}{10} = 1

Step 5 — note what was not needed

The indicators are not independent — if nine letters are correct the tenth must be too. That would wreck any variance calculation, but linearity does not care, so the answer stands.

Answer

Exactly 1 letter on average, and remarkably this is independent of $n$ : with a million letters the expected number of correct matches is still 1.

When expectation does not exist

Not every distribution has a finite mean. The Cauchy distribution,

f(x) = \frac{1}{\pi(1 + x^2)}

has $\int \lvert x \rvert f(x)\,dx = \infty$ , so $\E[X]$ is undefined. Sample means of Cauchy draws never settle — they keep jumping, because the tails are heavy enough that extreme values keep arriving.

python

import numpy as np

rng = np.random.default_rng(0)

# Die: empirical vs theoretical.
rolls = rng.integers(1, 7, 500_000)
print(f"E[X]  {rolls.mean():.4f}  theory 3.5")
print(f"E[X²] {np.mean(rolls**2):.4f}  theory {91/6:.4f}")

# Envelope problem: linearity says the answer is 1 for any n.
for n in (10, 100, 10_000):
    matches = [np.sum(rng.permutation(n) == np.arange(n)) for _ in range(2000)]
    print(f"n={n:<6} mean matches {np.mean(matches):.4f}  theory 1.0")

# Cauchy: the sample mean never converges.
print("\nrunning mean of Cauchy draws (no finite expectation):")
c = rng.standard_cauchy(1_000_000)
for k in (10**3, 10**4, 10**5, 10**6):
    print(f"  first {k:>7,} draws: {c[:k].mean():+10.3f}")

Exercise 1

A game costs £2 to play. You roll a die and win £1 per pip. Should you play?

Show solution

Expected winnings are $\E[X] = 3.5$ pounds, and the cost is £2, so expected profit per play is

3.5 - 2 = +1.50

Yes, play — the game is favourable by £1.50 per roll on average.

Two caveats worth stating. Expectation is a long-run average: on any single play you lose money whenever the roll is 1 or 2, which happens a third of the time. And expectation ignores risk entirely. A game with expected profit £1.50 but a small chance of a catastrophic loss may be one to decline, which is why utility theory weights outcomes by more than their monetary value.

Here, with bounded outcomes between £1 and £6 and a cost of £2, the risk is trivial and the expectation is decisive.

Exercise 2

A fair coin is flipped 100 times. Use indicators to find the expected number of times the pattern $HH$ appears in consecutive positions (overlapping allowed).

Show solution

There are 99 adjacent pairs of positions: $(1,2), (2,3), \dots, (99,100)$ .

Define $\mathbb{1}_i = 1$ if positions $i$ and $i+1$ are both heads. Two independent fair flips give

\E[\mathbb{1}_i] = \Prob(HH) = \tfrac12 \times \tfrac12 = \tfrac14

The total count is $X = \sum_{i=1}^{99} \mathbb{1}_i$ , so by linearity

\E[X] = 99 \times \tfrac14 = 24.75

The indicators are dependent — $\mathbb{1}_i$ and $\mathbb{1}_{i+1}$ share the flip at position $i+1$ , so knowing one raises the chance of the other. Linearity is unaffected, which is exactly why this approach works where a variance calculation would need the covariances.

Exercise 3

Show that $\E[X^2] \geq (\E[X])^2$ for any random variable with finite second moment, and say when equality holds.

Show solution

Consider the variance, which is an expectation of a squared quantity and therefore non-negative:

\Var(X) = \E\big[(X - \mu)^2\big] \geq 0

Expand the square and use linearity, remembering $\mu = \E[X]$ is a constant:

\E\big[X^2 - 2\mu X + \mu^2\big] = \E[X^2] - 2\mu\,\E[X] + \mu^2 = \E[X^2] - 2\mu^2 + \mu^2 = \E[X^2] - \mu^2

So $\Var(X) = \E[X^2] - (\E[X])^2 \geq 0$ , which rearranges to the claim.

Equality requires $\Var(X) = 0$ , meaning $(X - \mu)^2$ has expectation zero. Since it is non-negative, it must be zero with probability 1 — so $X$ is constant, $X = \mu$ almost surely.

This is the simplest instance of Jensen's inequality, with the convex function $g(x) = x^2$ , and the identity $\Var(X) = \E[X^2] - (\E[X])^2$ derived along the way is the computational formula used throughout the next lesson.

Next: Conditional Expectation, where the expectation itself becomes a random variable.