Intermediate30 min

The Central Limit Theorem

Why sample means go normal regardless of the parent distribution, with simulation evidence and sample-size rules.

Assumes you know

Laws of Large Numbers

The Central Limit Theorem

Intuition first

Take any distribution you like — skewed, bimodal, discrete, ugly. Draw $n$ values from it, average them, and write down the result. Repeat many times. The histogram of those averages will be approximately normal, and the approximation improves as $n$ grows.

This is genuinely remarkable. The shape of the original distribution is irrelevant. Roll dice, measure exponential waiting times, flip biased coins — the distribution of the sample mean converges to the same bell curve in every case. Only two things about the parent distribution survive: its mean, which becomes the centre, and its variance, which sets the width.

This is why the normal distribution is everywhere, and why inference works at all. It lets you put a confidence interval around a sample mean without knowing anything about the population's shape.

The limits matter too. The theorem is about averages, not individual observations, and it requires finite variance — which is why it says nothing about heavy-tailed data.

Statement

Let $X_1, \dots, X_n$ be i.i.d. with finite mean $\mu$ and finite variance $\sigma^2$ . Then

\frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \;\xrightarrow{d}\; \mathcal{N}(0, 1) \qquad \text{as } n \to \infty

Equivalently, for large $n$ :

\bar{X} \;\approx\; \mathcal{N}\!\left(\mu,\; \frac{\sigma^2}{n}\right), \qquad \sum_{i=1}^n X_i \;\approx\; \mathcal{N}\big(n\mu,\; n\sigma^2\big)

Notation used in this lesson
Symbol	Meaning	Read aloud
X̄	Sample mean, (X₁ + … + Xₙ)/n	X bar
μ, σ²	Mean and variance of the parent distribution	mu, sigma squared
σ/√n	Standard error — the SD of the sample mean	standard error
→ᵈ	Convergence in distribution	converges in distribution

Why the standard error is σ/√nAdvanced

This is exact and does not require the CLT, only linearity and independence.

\E[\bar{X}] = \E\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n \E[X_i] = \frac{n\mu}{n} = \mu

For the variance, apply the scaling rule $\Var(aX) = a^2\Var(X)$ with $a = 1/n$ , then independence to drop the covariance terms:

\Var(\bar{X}) = \frac{1}{n^2}\Var\left(\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\sum_{i=1}^n \Var(X_i) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}

\text{SE} = \sqrt{\Var(\bar{X})} = \frac{\sigma}{\sqrt{n}}

The $\sqrt{n}$ is the single most consequential number in statistics. To halve your uncertainty you need four times the data; to reduce it tenfold, a hundred times the data. This is why large-scale data collection has diminishing returns, and it is the same $1/\sqrt{n}$ that appeared in the generalisation bounds.

Proof sketch via moment generating functionsAdvanced

Standardise each observation: let $Y_i = (X_i - \mu)/\sigma$ , so $\E[Y_i] = 0$ and $\Var(Y_i) = 1$ . Define

Z_n = \frac{1}{\sqrt{n}}\sum_{i=1}^n Y_i

which is exactly the standardised sample mean. Let $M(t) = \E[e^{tY}]$ be the common MGF of the $Y_i$ . Because the $Y_i$ are independent and $Z_n$ scales each by $1/\sqrt{n}$ :

M_{Z_n}(t) = \left[M\!\left(\frac{t}{\sqrt{n}}\right)\right]^n

Now Taylor-expand $M$ about zero. Using $M(0) = 1$ , $M'(0) = \E[Y] = 0$ and $M''(0) = \E[Y^2] = 1$ :

M(s) = 1 + 0 \cdot s + \frac{s^2}{2} + o(s^2)

Substitute $s = t/\sqrt{n}$ :

M\!\left(\frac{t}{\sqrt{n}}\right) = 1 + \frac{t^2}{2n} + o\!\left(\frac{1}{n}\right)

Raise to the $n$ th power:

M_{Z_n}(t) = \left(1 + \frac{t^2}{2n} + o(1/n)\right)^{n} \;\longrightarrow\; e^{t^2/2}

using $\lim_{n\to\infty}(1 + x/n)^n = e^x$ . And $e^{t^2/2}$ is precisely the MGF of the standard normal.

Two things the proof reveals. The first-order term vanished because we centred, and the second-order term survived because we scaled by $\sqrt{n}$ — that specific power is what balances the expansion. And the argument needs $\E[Y^2]$ to be finite, which is the finite-variance condition. Distributions with infinite variance have no such expansion, and the theorem fails.

How large must n be?

There is no universal answer; it depends on how far the parent distribution is from normal.

Parent distribution	Adequate $n$
Already normal	1 — exact, not approximate
Symmetric, light tails (uniform)	5–10
Moderately skewed (exponential)	30–50
Heavily skewed (lognormal, Pareto with finite variance)	100s to 1000s
Binary with $p$ near 0 or 1	Need $np \geq 5$ and $n(1-p) \geq 5$
Infinite variance (Cauchy)	Never

Sample means of an exponential parent. n = 1 is the parent itself, strongly right-skewed. By n = 30 the distribution of the mean is visibly symmetric and close to normal.

Solved problem 1 · A skewed parent, a normal mean

Customer service times have mean 4 minutes and standard deviation 6 minutes — strongly right-skewed, since $\sigma > \mu$ for a positive quantity. A sample of 100 customers is taken.

(a) What is the distribution of the sample mean? (b) Find $\Prob(\bar{X} > 5)$ . (c) Find the probability the total time exceeds 450 minutes.

Step 1 — part (a): apply the CLT

$\mu = 4$ , $\sigma = 6$ , $n = 100$ . The standard error is

\text{SE} = \frac{\sigma}{\sqrt{n}} = \frac{6}{\sqrt{100}} = \frac{6}{10} = 0.6

\bar{X} \approx \mathcal{N}(4,\; 0.6^2) = \mathcal{N}(4,\, 0.36)

Step 2 — note the shape change

The parent has $\sigma/\mu = 1.5$ , strongly skewed. The sample mean has $\text{SE}/\mu = 0.15$ and is approximately symmetric. Averaging 100 observations shrank the relative spread by a factor of 10 and removed most of the skew.

Step 3 — part (b): standardise

z = \frac{5 - 4}{0.6} = \frac{1}{0.6} \approx 1.6667

\Prob(\bar{X} > 5) = \Prob(Z > 1.6667) \approx 1 - 0.9522 = 0.0478

Step 4 — part (c): totals, not means

For the sum $S = \sum_{i=1}^{100} X_i$ :

\E[S] = n\mu = 400, \qquad \Var(S) = n\sigma^2 = 100 \times 36 = 3600, \qquad \text{SD}(S) = 60

z = \frac{450 - 400}{60} = \frac{50}{60} \approx 0.8333

\Prob(S > 450) = \Prob(Z > 0.8333) \approx 1 - 0.7977 = 0.2023

Step 5 — check the two answers are consistent

$\bar{X} > 5$ is the same event as $S > 500$ , not $S > 450$ — so the two answers should differ, and they do. As a check on part (c) via the mean: $S > 450$ means $\bar{X} > 4.5$ , giving $z = (4.5-4)/0.6 = 0.8333$ , the same $z$ ✓.

Note the asymmetry in scaling: the standard deviation of the sum grows like $\sqrt{n}$ ( $60 = 6\sqrt{100}$ ) while the standard deviation of the mean shrinks like $\sqrt{n}$ .

Answer

(a) $\bar{X} \approx \mathcal{N}(4, 0.36)$ with SE $= 0.6$ ; (b) $\approx 0.0478$ ; (c) $\approx 0.2023$ .

Where the CLT fails

Infinite variance. The Cauchy distribution has undefined mean and infinite variance. Its sample mean has the same Cauchy distribution as a single observation — averaging achieves nothing at all. More generally, for Pareto tails with index $\alpha < 2$ the variance is infinite and the limit is a stable distribution, not normal.

Dependence. Time series, spatially correlated data, and clustered observations violate independence. Variants of the CLT exist under weak dependence, but the effective sample size is smaller than $n$ , so naive standard errors are too small.

Not identically distributed. Mild violations are fine under the Lindeberg condition, but if one observation dominates the variance the limit is not normal.

python

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
REPS = 40_000

# Exponential parent: mean 4, sd 4. Watch skewness of the mean decay.
print(f"{'n':>5}  {'mean':>7}  {'SE obs':>8}  {'SE theory':>9}  {'skew':>7}")
for n in (1, 5, 10, 30, 100):
    means = rng.exponential(4.0, size=(REPS, n)).mean(axis=1)
    print(f"{n:5d}  {means.mean():7.3f}  {means.std():8.4f}  "
          f"{4/np.sqrt(n):9.4f}  {stats.skew(means):7.3f}")

# The worked example.
print(f"\nP(X̄ > 5)  {stats.norm.sf(5, loc=4, scale=0.6):.4f}")
print(f"P(S > 450) {stats.norm.sf(450, loc=400, scale=60):.4f}")

# Where it fails: Cauchy has no finite variance, so averaging does nothing.
print("\nCauchy — sample mean does not concentrate:")
for n in (10, 100, 1000, 10_000):
    m = rng.standard_cauchy(size=(2000, n)).mean(axis=1)
    print(f"  n={n:<6} IQR of sample means {np.subtract(*np.percentile(m, [75, 25])):.3f}")

Running the first block, skewness of the sample mean falls from about 2.0 at $n=1$ towards 0.2 at $n=100$ — the CLT happening numerically. The Cauchy block shows the interquartile range of the sample mean refusing to shrink, no matter how large $n$ becomes.

Exercise 1

A population has $\mu = 50$ , $\sigma = 15$ . For $n = 36$ , find $\Prob(48 < \bar{X} < 53)$ .

Show solution

\text{SE} = \frac{15}{\sqrt{36}} = \frac{15}{6} = 2.5

z_1 = \frac{48 - 50}{2.5} = -0.80, \qquad z_2 = \frac{53 - 50}{2.5} = 1.20

\Prob(-0.80 < Z < 1.20) = \Phi(1.20) - \Phi(-0.80) = 0.8849 - 0.2119 = 0.6730

About 67.3%.

Note that no assumption about the parent distribution's shape was needed — only its mean and standard deviation, plus $n = 36$ being large enough for the CLT to bite. If the parent were heavily skewed, 36 might not suffice and this answer would be unreliable.

Exercise 2

Explain why the CLT does not justify modelling individual insurance claims as normal, even with a million historical claims.

Show solution

The CLT concerns the distribution of the sample mean, not the distribution of individual observations. Collecting more claims makes the average claim size more precisely known and more normally distributed. It does nothing to the shape of the underlying claim distribution, which is a fixed property of the world.

Claim sizes are strongly right-skewed: most are small, a few are enormous. That remains true with a million observations — indeed more observations reveal the tail more clearly rather than softening it.

Two concrete consequences of getting this wrong:

Negative claims. A normal model assigns positive probability to negative claim amounts, which is impossible.

Catastrophic underestimation of tail risk. A normal fitted to the mean and variance of claims will put negligible probability beyond $4\sigma$ , while the real distribution produces such claims regularly. This is precisely how insurers and banks under-reserve.

There is also a deeper problem: many heavy-tailed claim distributions have such large or infinite variance that the CLT converges extremely slowly or not at all, so even the mean may not be reliably normal at a million observations.

The right approach is to model individual claims with a distribution that has the correct shape — lognormal, gamma, Pareto, or a spliced body-and-tail model — and reserve the CLT for statements about aggregate averages.

Next: Markov Chains, the last lesson in this module and the bridge to hidden Markov models and reinforcement learning.