Advanced26 min

Gamma and Beta Distributions

Sums of exponentials, the gamma function, and the beta distribution as a conjugate prior over probabilities.

Assumes you know

Uniform and Exponential Distributions

Gamma and Beta Distributions

Intuition first

Two flexible families that fill important gaps.

The gamma generalises the exponential. An exponential times a single event; a gamma times the $k$ -th event. Waiting for one customer is exponential, waiting for the tenth is gamma. It lives on $(0, \infty)$ , is right-skewed, and is the natural model for durations, insurance claim sizes and rainfall totals.

The beta lives on $[0, 1]$ , which makes it the distribution for modelling a probability. If you are uncertain about a conversion rate, the beta describes that uncertainty. Its two shape parameters let it be flat, bell-shaped, U-shaped or J-shaped.

The beta's real importance is that it is the conjugate prior for the binomial: start with a beta belief about $p$ , observe some successes and failures, and your updated belief is again a beta with the counts simply added on. Bayesian updating reduces to arithmetic.

The gamma function

\Gamma(\alpha) = \int_0^{\infty} t^{\alpha - 1}e^{-t}\,dt

Key properties:

\Gamma(n) = (n-1)! \text{ for positive integers}, \qquad \Gamma(\alpha + 1) = \alpha\Gamma(\alpha), \qquad \Gamma(\tfrac12) = \sqrt{\pi}

It interpolates the factorial to non-integer arguments, which is what lets the gamma and beta distributions have continuous shape parameters.

The gamma distribution

X \sim \text{Gamma}(\alpha, \beta), \qquad f(x) = \frac{1}{\Gamma(\alpha)\beta^\alpha}x^{\alpha-1}e^{-x/\beta}, \qquad x > 0

\E[X] = \alpha\beta, \qquad \Var(X) = \alpha\beta^2, \qquad M_X(t) = (1 - \beta t)^{-\alpha}

Notation used in this lesson
Symbol	Meaning	Read aloud
α	Shape — controls skewness; α = 1 gives the exponential	alpha
β	Scale — stretches the distribution	beta
λ = 1/β	Rate, the alternative parametrisation	lambda
Γ(α)	Gamma function, the normalising constant	gamma of alpha

The gamma as a sum of exponentialsAdvanced

Let $X_1, \dots, X_\alpha$ be independent $\text{Exponential}(\text{rate } \lambda)$ , each with MGF

M_{X_i}(t) = \frac{\lambda}{\lambda - t} = (1 - t/\lambda)^{-1}

MGFs of independent sums multiply, so for $S = \sum_{i=1}^{\alpha} X_i$ :

M_S(t) = \left[(1 - t/\lambda)^{-1}\right]^{\alpha} = (1 - \beta t)^{-\alpha}, \qquad \beta = 1/\lambda

which is exactly the gamma MGF. So a sum of $\alpha$ i.i.d. exponentials is $\text{Gamma}(\alpha, \beta)$ — sometimes called the Erlang distribution when $\alpha$ is an integer.

Mean and variance follow immediately without integration:

\E[S] = \alpha \cdot \frac{1}{\lambda} = \alpha\beta, \qquad \Var(S) = \alpha \cdot \frac{1}{\lambda^2} = \alpha\beta^2

Interpretation: in a Poisson process at rate $\lambda$ , the waiting time until the $\alpha$ -th event is $\text{Gamma}(\alpha, 1/\lambda)$ . The gamma is to the exponential what the negative binomial is to the geometric.

Two special cases worth recognising:

$\alpha = 1$ : the exponential.
$\alpha = k/2$ , $\beta = 2$ : the chi-squared distribution with $k$ degrees of freedom, which is why the next lesson's distributions are all gamma relatives.

The beta distribution

X \sim \text{Beta}(\alpha, \beta), \qquad f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)}, \qquad 0 \leq x \leq 1

where $B(\alpha,\beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$ normalises it.

\E[X] = \frac{\alpha}{\alpha + \beta}, \qquad \Var(X) = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}

$\alpha, \beta$	Shape	Interpretation
$1, 1$	Flat — uniform on $[0,1]$	No information about $p$
$2, 2$	Symmetric bump	$p$ probably near 0.5
$5, 2$	Left-skewed, mass near 1	$p$ probably high
$2, 5$	Right-skewed, mass near 0	$p$ probably low
$0.5, 0.5$	U-shaped	$p$ probably extreme
$50, 50$	Tight around 0.5	Strong belief $p \approx 0.5$

Beta–binomial conjugacy

Deriving the posteriorAdvanced

Prior: $p \sim \text{Beta}(\alpha, \beta)$ , so

\pi(p) \propto p^{\alpha-1}(1-p)^{\beta-1}

Observe $k$ successes in $n$ trials. The likelihood is binomial in $p$ :

L(p) \propto p^{k}(1-p)^{n-k}

By Bayes' theorem the posterior is proportional to the product:

\pi(p \mid \text{data}) \propto p^{\alpha-1}(1-p)^{\beta-1} \cdot p^{k}(1-p)^{n-k} = p^{(\alpha+k)-1}(1-p)^{(\beta+n-k)-1}

That is the kernel of a Beta distribution. Hence

\boxed{p \mid \text{data} \sim \text{Beta}(\alpha + k,\; \beta + n - k)}

The update is pure addition: add successes to $\alpha$ , failures to $\beta$ . No integration is needed, because the beta and binomial have matching functional forms in $p$ — which is what "conjugate" means.

The posterior mean is a weighted average of prior and data:

\E[p \mid \text{data}] = \frac{\alpha + k}{\alpha + \beta + n} = \underbrace{\frac{\alpha+\beta}{\alpha+\beta+n}}_{\text{weight on prior}}\cdot\frac{\alpha}{\alpha+\beta} \;+\; \underbrace{\frac{n}{\alpha+\beta+n}}_{\text{weight on data}}\cdot\frac{k}{n}

As $n$ grows the data weight tends to 1 and the prior is progressively ignored — the mathematical statement that evidence eventually overwhelms prior belief.

Solved problem 1 · Updating a conversion rate

You believe a landing page's conversion rate is around 10%, with moderate uncertainty, modelled as $\text{Beta}(2, 18)$ . You then observe 47 conversions in 300 visitors.

Find the prior mean, the posterior distribution, the posterior mean, and compare with the raw sample proportion.

Step 1 — prior mean and strength

\E[p] = \frac{\alpha}{\alpha+\beta} = \frac{2}{20} = 0.10 \;\checkmark

Prior strength $\alpha + \beta = 20$ , so the prior carries the weight of about 20 observations.

Step 2 — posterior parameters

$k = 47$ successes, $n - k = 300 - 47 = 253$ failures.

\alpha' = 2 + 47 = 49, \qquad \beta' = 18 + 253 = 271

p \mid \text{data} \sim \text{Beta}(49, 271)

Step 3 — posterior mean

\E[p \mid \text{data}] = \frac{49}{49 + 271} = \frac{49}{320} = 0.153125

Step 4 — compare with the raw proportion

\hat{p}_{\text{MLE}} = \frac{47}{300} = 0.156667

The posterior mean $0.1531$ sits slightly below the sample proportion, pulled towards the prior mean of $0.10$ .

Step 5 — check the weighting decomposition

\text{weight on prior} = \frac{20}{20 + 300} = 0.0625, \qquad \text{weight on data} = \frac{300}{320} = 0.9375

0.0625 \times 0.10 + 0.9375 \times 0.156667 = 0.00625 + 0.14688 = 0.15313 \;\checkmark

With 300 observations against a prior worth 20, the data dominates at 94% weight — which is why the shrinkage is only 0.35 percentage points.

Step 6 — posterior uncertainty

\Var(p \mid \text{data}) = \frac{49 \times 271}{320^2 \times 321} = \frac{13{,}279}{32{,}870{,}400} \approx 4.039\times10^{-4}

\text{SD} \approx 0.0201

So a rough 95% credible interval is $0.153 \pm 2(0.020) = (0.113,\, 0.193)$ .

Answer

Posterior $\text{Beta}(49, 271)$ , mean $0.1531$ , SD $0.0201$ . The sample proportion was $0.1567$ ; the prior shrank it towards 0.10 by a modest amount because 300 observations outweigh a prior worth 20.

python

import numpy as np
from scipy import stats

# Gamma: sum of exponentials.
rng = np.random.default_rng(0)
alpha, beta_scale = 5, 2.0
sums = rng.exponential(beta_scale, size=(200_000, alpha)).sum(axis=1)
G = stats.gamma(a=alpha, scale=beta_scale)
print(f"sum of {alpha} exponentials: mean {sums.mean():.4f}  var {sums.var():.4f}")
print(f"Gamma({alpha}, {beta_scale}):        mean {G.mean():.4f}  var {G.var():.4f}")

# Beta-binomial conjugacy.
a0, b0 = 2, 18
k, n = 47, 300
a1, b1 = a0 + k, b0 + (n - k)
prior, post = stats.beta(a0, b0), stats.beta(a1, b1)

print(f"\nprior  Beta({a0},{b0})   mean {prior.mean():.4f}  sd {prior.std():.4f}")
print(f"post   Beta({a1},{b1}) mean {post.mean():.4f}  sd {post.std():.4f}")
print(f"MLE                   {k/n:.4f}")
print(f"95% credible interval ({post.ppf(0.025):.4f}, {post.ppf(0.975):.4f})")

# Shrinkage is strong when n is small.
print(f"\n{'n':>5} {'k':>3}  {'MLE':>7}  {'posterior mean':>15}")
for n_, k_ in ((3, 1), (30, 10), (300, 100), (3000, 1000)):
    print(f"{n_:5d} {k_:3d}  {k_/n_:7.4f}  {(a0+k_)/(a0+b0+n_):15.4f}")

The final table is the point: the MLE is $0.333$ at every sample size, while the posterior mean climbs from $0.130$ towards $0.333$ as evidence accumulates.

Exercise 1

Calls arrive at 4 per hour as a Poisson process. Find the mean and standard deviation of the waiting time until the 3rd call, and name the distribution.

Show solution

Waiting time until the $\alpha$ -th event in a Poisson process of rate $\lambda$ is $\text{Gamma}(\alpha, 1/\lambda)$ .

Here $\alpha = 3$ and $\lambda = 4$ per hour, so $\beta = 1/4 = 0.25$ hours:

T \sim \text{Gamma}(3,\; 0.25)

\E[T] = \alpha\beta = 3 \times 0.25 = 0.75 \text{ hours} = 45 \text{ minutes}

\Var(T) = \alpha\beta^2 = 3 \times 0.0625 = 0.1875 \quad\Longrightarrow\quad \text{SD} = \sqrt{0.1875} \approx 0.4330 \text{ hours} \approx 26 \text{ minutes}

Sanity check by the sum-of-exponentials view: each gap has mean $1/4$ hour, and three gaps give $3/4$ hour ✓. The standard deviation is $\sqrt{3}$ times a single gap's SD of 15 minutes, giving $15\sqrt{3} \approx 26$ minutes ✓ — variances add, so SDs grow like $\sqrt{3}$ , not 3.

Exercise 2

You use a $\text{Beta}(1,1)$ prior for a coin's bias and observe 8 heads in 10 flips. Give the posterior, its mean, and explain why the mean is not $0.8$ .

Show solution

$\text{Beta}(1,1)$ is the uniform prior on $[0,1]$ . Updating with $k = 8$ successes and $n - k = 2$ failures:

p \mid \text{data} \sim \text{Beta}(1 + 8,\; 1 + 2) = \text{Beta}(9, 3)

\E[p \mid \text{data}] = \frac{9}{9 + 3} = \frac{9}{12} = 0.75

The MLE is $8/10 = 0.80$ , so the posterior mean is lower.

The reason is that Beta(1,1) is uniform but not weightless. It contributes the equivalent of one prior success and one prior failure, so the posterior mean is

\frac{k + 1}{n + 2} = \frac{9}{12} = 0.75

which is Laplace's rule of succession — the classical add-one smoothing used in naive Bayes and n-gram language models. Its purpose is exactly this: to avoid assigning probability 0 or 1 on the basis of a small sample. With 0 heads in 2 flips the MLE would be $0$ , claiming heads is impossible, while the posterior mean gives $1/4$ .

A truly weightless prior would be $\text{Beta}(0,0)$ , which is improper — it does not integrate to a finite value — and recovers the MLE in the limit.

Next: t, Chi-Squared and F Distributions, all of which are gamma relatives built from normal samples.