Advanced24 min

Laws of Large Numbers

Weak and strong laws, convergence in probability vs almost surely, and what they justify about sample means.

Assumes you know

Probability Inequalities

Laws of Large Numbers

Intuition first

Flip a fair coin ten times and you might see 7 heads. Flip it ten thousand times and the proportion will be very close to 0.5. The law of large numbers is the formal version of that observation: sample averages converge to the true mean as the sample grows.

This is the theorem that licenses statistics. Without it, there would be no reason to think a sample mean tells you anything about a population mean.

Two versions exist, differing in the sense in which convergence happens. The weak law says the sample mean is probably close to $\mu$ for large $n$ . The strong law says the sequence of sample means actually converges to $\mu$ along almost every infinite run. The weak law allows rare excursions to keep occurring forever; the strong law does not.

Both require a finite mean. Neither says anything about how fast convergence happens — that is the Central Limit Theorem's job.

The weak law

For i.i.d. $X_1, \dots, X_n$ with finite mean $\mu$ , for every $\epsilon > 0$ :

\lim_{n \to \infty} \Prob\big(\lvert \bar{X}_n - \mu \rvert \geq \epsilon\big) = 0

Written $\bar{X}_n \xrightarrow{P} \mu$ , read "converges in probability".

Proof from Chebyshev, assuming finite varianceAdvanced

Assume additionally that $\Var(X_i) = \sigma^2 < \infty$ . We know exactly:

\E[\bar{X}_n] = \mu, \qquad \Var(\bar{X}_n) = \frac{\sigma^2}{n}

Apply Chebyshev's inequality to $\bar{X}_n$ , whose standard deviation is $\sigma/\sqrt{n}$ :

\Prob\big(\lvert \bar{X}_n - \mu \rvert \geq \epsilon\big) \;\leq\; \frac{\Var(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}

For fixed $\epsilon$ , the right side tends to 0 as $n \to \infty$ . That is the weak law.

Three lines, and it also hands you a quantitative version: to make the failure probability at most $\delta$ ,

\frac{\sigma^2}{n\epsilon^2} \leq \delta \quad\Longrightarrow\quad n \geq \frac{\sigma^2}{\delta\epsilon^2}

The finite-variance assumption is convenient but not necessary — Khinchine's theorem proves the weak law under a finite mean alone, using characteristic functions rather than Chebyshev.

The strong law

\Prob\left(\lim_{n \to \infty} \bar{X}_n = \mu\right) = 1

Written $\bar{X}_n \xrightarrow{a.s.} \mu$ , read "converges almost surely".

Notation used in this lesson
Symbol	Meaning	Read aloud
X̄ₙ	Sample mean of the first n observations	X bar n
→ᴾ	Convergence in probability (weak law)	converges in probability
→ᵃ·ˢ·	Almost sure convergence (strong law)	converges almost surely
ε	Tolerance for being 'close'	epsilon

Solved problem 1 · How much data does the LLN actually need?

A fair die is rolled $n$ times. Using Chebyshev, find $n$ such that the sample mean is within $0.1$ of $3.5$ with probability at least $0.95$ . Then compare with the CLT's answer.

Step 1 — the parent moments

From earlier lessons:

\mu = 3.5, \qquad \sigma^2 = \frac{35}{12} \approx 2.9167

Step 2 — apply the Chebyshev bound

We need $\Prob(\lvert \bar X - 3.5\rvert \geq 0.1) \leq 0.05$ , and Chebyshev gives

\frac{\sigma^2}{n\epsilon^2} \leq 0.05

Step 3 — solve

n \geq \frac{\sigma^2}{0.05\,\epsilon^2} = \frac{2.9167}{0.05 \times 0.01} = \frac{2.9167}{0.0005} = 5833.3

So $n \geq 5834$ .

Step 4 — the CLT answer, for comparison

Assuming $\bar X$ is approximately normal with SE $= \sigma/\sqrt{n}$ , a 95% interval needs

1.96\,\frac{\sigma}{\sqrt{n}} \leq 0.1 \quad\Longrightarrow\quad \sqrt{n} \geq \frac{1.96 \times 1.7078}{0.1} = 33.47

n \geq 33.47^2 \approx 1121

Step 5 — interpret the gap

\text{Chebyshev}: 5834 \qquad \text{CLT}: 1121

Chebyshev demands 5.2 times more data. The difference is the price of making no shape assumption: Chebyshev must hold even for the most adversarial distribution with variance $35/12$ , while the CLT exploits the fact that averages of 1,000 dice rolls really are near-normal.

Answer

Chebyshev requires $n \geq 5{,}834$ ; the CLT-based calculation gives $n \geq 1{,}121$ . Both are correct — the first is a guarantee, the second an approximation that is very accurate at this sample size.

Where the law fails

Infinite or undefined mean. The Cauchy distribution has no finite mean, and its sample mean has the same Cauchy distribution as a single draw. Averaging a million Cauchy values is no more informative than taking one.

Dependence. Strongly correlated observations break the $\Var(\bar X) = \sigma^2/n$ step. If every observation is identical, $\bar X = X_1$ and no convergence occurs at all. Weak-dependence versions of the law exist, with an effective sample size smaller than $n$ .

Heavy but finite variance. The law still holds, but convergence can be so slow as to be useless in practice — a Pareto distribution with tail index just above 2 may need millions of observations before the sample mean stabilises.

python

import numpy as np

rng = np.random.default_rng(0)

# Convergence of the running mean for a fair die.
rolls = rng.integers(1, 7, 1_000_000)
running = np.cumsum(rolls) / np.arange(1, len(rolls) + 1)
print("n          running mean   |error|")
for n in (10, 100, 1_000, 10_000, 100_000, 1_000_000):
    print(f"{n:>9,}  {running[n-1]:12.5f}   {abs(running[n-1] - 3.5):.5f}")

# Sample size: Chebyshev guarantee vs CLT approximation.
sigma2, eps, delta = 35/12, 0.1, 0.05
print(f"\nChebyshev n >= {np.ceil(sigma2/(delta*eps**2)):.0f}")
print(f"CLT       n >= {np.ceil((1.96*np.sqrt(sigma2)/eps)**2):.0f}")

# The gambler's fallacy: proportion converges, absolute surplus does not.
flips = rng.integers(0, 2, 1_000_000)
surplus = np.cumsum(2*flips - 1)              # heads minus tails
print("\nn          proportion   heads-minus-tails")
for n in (100, 10_000, 1_000_000):
    print(f"{n:>9,}  {flips[:n].mean():10.5f}   {surplus[n-1]:+8d}")

# Cauchy: the law does not apply.
c = rng.standard_cauchy(1_000_000)
cr = np.cumsum(c) / np.arange(1, len(c) + 1)
print("\nCauchy running mean (never settles):")
for n in (1_000, 100_000, 1_000_000):
    print(f"  n={n:>9,}  {cr[n-1]:+10.4f}")

The third block is the instructive one: the proportion of heads marches towards 0.5 while the absolute surplus drifts further from zero — both at once, and not in contradiction.

Exercise 1

Explain why the law of large numbers does not guarantee that a casino always profits on any given night.

Show solution

The law is asymptotic. It says the average outcome per bet converges to the expected value as the number of bets grows without bound, and says nothing about any finite number.

For a game with a 2% house edge, the casino's average profit per unit staked tends to 0.02. But over a single night's $n$ bets, the realised average has standard error $\sigma/\sqrt{n}$ , and for modest $n$ that can easily exceed 0.02 — so a losing night is entirely possible.

The relevant quantity is the ratio of edge to noise:

\frac{\text{edge} \times n}{\sigma\sqrt{n}} = \frac{\text{edge}\sqrt{n}}{\sigma}

which grows like $\sqrt{n}$ . The edge accumulates linearly in $n$ while the fluctuation grows only as $\sqrt{n}$ , so the probability of an overall loss falls as volume rises — but it is never zero for finite $n$ .

This is also why casinos impose table limits. A single enormous bet makes $n$ effectively 1, where the edge is irrelevant and variance dominates. Their business model requires many small independent bets, which is precisely the setting the law of large numbers describes.

Exercise 2

Give an example of a sequence converging in probability but not almost surely, to show the two laws genuinely differ.

Show solution

Let $X_n$ be independent with

\Prob(X_n = 1) = \frac{1}{n}, \qquad \Prob(X_n = 0) = 1 - \frac{1}{n}

Convergence in probability to 0. For any $\epsilon \in (0,1)$ ,

\Prob(\lvert X_n - 0 \rvert \geq \epsilon) = \Prob(X_n = 1) = \frac{1}{n} \to 0

So $X_n \xrightarrow{P} 0$ .

No almost sure convergence. By the second Borel–Cantelli lemma, since the $X_n$ are independent and

\sum_{n=1}^{\infty} \Prob(X_n = 1) = \sum_{n=1}^{\infty}\frac{1}{n} = \infty

the event $X_n = 1$ occurs infinitely often with probability 1. So along almost every infinite sequence there are arbitrarily late times at which $X_n = 1$ , and the sequence never settles at 0.

\Prob\left(\lim_{n\to\infty} X_n = 0\right) = 0

The two notions differ exactly here: the probability of a deviation at each fixed time goes to zero, yet deviations keep happening forever because $\sum 1/n$ diverges. Had the probabilities been $1/n^2$ instead, the sum would converge and Borel–Cantelli would give almost sure convergence too.

Next: The Central Limit Theorem, which describes not just that the sample mean converges but the shape of its distribution on the way.