Advanced26 min

Transformations of Random Variables

The CDF method, change-of-variables with Jacobians, and inverse transform sampling.

Assumes you know

PMF, PDF and CDF

Transformations of Random Variables

Intuition first

You know the distribution of $X$ and you need the distribution of $g(X)$ — a squared value, a logarithm, a reciprocal. This comes up constantly: log-transforming skewed data, converting between units, deriving the chi-squared from the normal, simulating any distribution from a uniform.

There are two reliable methods.

The CDF method always works: write $\Prob(Y \leq y)$ , rearrange the inequality in terms of $X$ , then read off or differentiate. Slower, but it never lies to you about monotonicity or about multiple branches.

The change-of-variables formula is faster but only applies when $g$ is monotone. Its essential ingredient is the $\lvert dx/dy \rvert$ factor, which accounts for the transformation stretching or compressing the axis. Forgetting it is the single most common error, and it produces a "density" that does not integrate to 1.

The CDF method

Three steps, in order:

Write $F_Y(y) = \Prob(Y \leq y) = \Prob(g(X) \leq y)$ .
Rearrange the inequality into a statement about $X$ .
Express in terms of $F_X$ , then differentiate for the density.

Solved problem 1 · The square of a uniform

$X \sim \text{Uniform}(0,1)$ and $Y = X^2$ . Find the density of $Y$ .

Step 1 — the support

$X \in (0,1)$ so $Y = X^2 \in (0,1)$ as well.

Step 2 — write the CDF and rearrange

F_Y(y) = \Prob(X^2 \leq y)

Since $X > 0$ on the support, taking square roots preserves the inequality:

= \Prob(X \leq \sqrt{y}) = F_X(\sqrt{y})

Step 3 — substitute the uniform CDF

For $\text{Uniform}(0,1)$ , $F_X(x) = x$ , so

F_Y(y) = \sqrt{y}, \qquad 0 < y < 1

Step 4 — differentiate

f_Y(y) = \frac{d}{dy}\,y^{1/2} = \frac{1}{2\sqrt{y}}, \qquad 0 < y < 1

Step 5 — verify it is a density

\int_0^1 \frac{1}{2\sqrt{y}}\,dy = \left[\sqrt{y}\right]_0^1 = 1 \;\checkmark

Step 6 — interpret

The density blows up as $y \to 0$ , since $1/(2\sqrt{y}) \to \infty$ , yet the integral is finite. This is the squaring compressing the interval near zero: values of $X$ in $(0, 0.1)$ — 10% of the probability — get squeezed into $Y \in (0, 0.01)$ , just 1% of the range, so the density there must be about ten times larger.

Note also that $f_Y$ exceeds 1 on part of its range, which is allowed: densities are not probabilities.

Answer

$f_Y(y) = \dfrac{1}{2\sqrt{y}}$ for $0 < y < 1$ . This is $\text{Beta}(\tfrac12, 1)$ .

The change-of-variables formula

If $g$ is monotone and differentiable with inverse $x = g^{-1}(y)$ :

f_Y(y) = f_X\big(g^{-1}(y)\big)\left\lvert \frac{dx}{dy} \right\rvert

Where the Jacobian factor comes fromAdvanced

Probability must be conserved: the chance of landing in a small interval cannot change just because we relabelled the axis. For a small interval $dx$ mapping to $dy$ :

f_X(x)\,\lvert dx \rvert = f_Y(y)\,\lvert dy \rvert

Dividing:

f_Y(y) = f_X(x)\left\lvert \frac{dx}{dy} \right\rvert

The absolute value is needed because a decreasing $g$ has negative derivative, while densities must be non-negative — and because the direction of the interval is irrelevant to its probability.

Why monotonicity is required. If $g$ is not monotone, several $x$ values map to the same $y$ , and probability arrives at $y$ from each branch. Take $Y = X^2$ with $X$ on $(-1, 1)$ : both $x = +\sqrt y$ and $x = -\sqrt y$ contribute, so

f_Y(y) = \sum_{\text{branches } i} f_X\big(g_i^{-1}(y)\big)\left\lvert \frac{dx_i}{dy}\right\rvert

Applying the single-branch formula would miss half the probability. In Solved problem 1 the issue did not arise because $X > 0$ made the squaring monotone on its support — worth checking every time.

Solved problem 2 · Deriving the lognormal

$X \sim \mathcal{N}(\mu, \sigma^2)$ and $Y = e^X$ . Find the density of $Y$ , and its mean.

Step 1 — check monotonicity and support

$g(x) = e^x$ is strictly increasing on all of $\mathbb{R}$ , so the single-branch formula applies. Its range is $(0, \infty)$ , so $Y > 0$ .

Step 2 — invert and differentiate

y = e^x \quad\Longrightarrow\quad x = \ln y, \qquad \frac{dx}{dy} = \frac{1}{y}

Since $y > 0$ , the absolute value is just $1/y$ .

Step 3 — apply the formula

f_Y(y) = f_X(\ln y)\cdot\frac{1}{y} = \frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{(\ln y - \mu)^2}{2\sigma^2}\right)\cdot\frac{1}{y}

f_Y(y) = \frac{1}{y\sigma\sqrt{2\pi}}\exp\left(-\frac{(\ln y - \mu)^2}{2\sigma^2}\right), \qquad y > 0

This is the lognormal density.

Step 4 — the mean, using the normal MGF

\E[Y] = \E[e^X] = M_X(1)

The normal MGF is $M_X(t) = \exp(\mu t + \tfrac12\sigma^2 t^2)$ , so at $t = 1$ :

\E[Y] = \exp\left(\mu + \frac{\sigma^2}{2}\right)

Step 5 — note the trap

\E[e^X] = e^{\mu + \sigma^2/2} \;\neq\; e^{\E[X]} = e^{\mu}

The gap is the factor $e^{\sigma^2/2} > 1$ . This is Jensen's inequality: $e^x$ is convex, so $\E[e^X] > e^{\E[X]}$ .

Concretely, with $\mu = 0$ and $\sigma = 1$ : the median of $Y$ is $e^0 = 1$ , but the mean is $e^{0.5} \approx 1.649$ — 65% higher. The lognormal is right-skewed, so its mean substantially exceeds its median.

Answer

$f_Y(y) = \dfrac{1}{y\sigma\sqrt{2\pi}}\exp\!\left(-\dfrac{(\ln y - \mu)^2}{2\sigma^2}\right)$ for $y > 0$ , with $\E[Y] = e^{\mu + \sigma^2/2}$ and median $e^{\mu}$ .

The standard transformations

Transformation	Result
$aX + b$ , $X$ normal	Normal: $\mathcal{N}(a\mu+b,\; a^2\sigma^2)$
$(X - \mu)/\sigma$ , $X$ normal	Standard normal
$Z^2$ , $Z$ standard normal	$\chi^2_1$
$\sum_{i=1}^k Z_i^2$	$\chi^2_k$
$e^X$ , $X$ normal	Lognormal
$F^{-1}(U)$ , $U$ uniform	Any distribution with CDF $F$
$-\ln(U)/\lambda$ , $U$ uniform	$\text{Exponential}(\lambda)$
$\sum_{i=1}^{\alpha} E_i$ , $E_i$ exponential	$\text{Gamma}(\alpha, \beta)$

Solved problem 3 · Z² is chi-squared with one degree of freedom

Show that if $Z \sim \mathcal{N}(0,1)$ then $Z^2 \sim \chi^2_1$ .

Step 1 — recognise the non-monotonicity

$g(z) = z^2$ is not monotone on $\mathbb{R}$ : both $+\sqrt y$ and $-\sqrt y$ map to $y$ . Use the CDF method, which handles this automatically.

Step 2 — write the CDF

For $y > 0$ :

F_Y(y) = \Prob(Z^2 \leq y) = \Prob(-\sqrt{y} \leq Z \leq \sqrt{y}) = \Phi(\sqrt{y}) - \Phi(-\sqrt{y})

By the symmetry of the standard normal, $\Phi(-a) = 1 - \Phi(a)$ :

F_Y(y) = 2\Phi(\sqrt{y}) - 1

Step 3 — differentiate

f_Y(y) = 2\,\phi(\sqrt{y})\cdot\frac{d}{dy}\sqrt{y} = 2\,\phi(\sqrt{y})\cdot\frac{1}{2\sqrt{y}} = \frac{\phi(\sqrt{y})}{\sqrt{y}}

Step 4 — substitute the normal density

With $\phi(z) = \frac{1}{\sqrt{2\pi}}e^{-z^2/2}$ and $z = \sqrt y$ , so $z^2 = y$ :

f_Y(y) = \frac{1}{\sqrt{y}}\cdot\frac{1}{\sqrt{2\pi}}e^{-y/2} = \frac{1}{\sqrt{2\pi}}\,y^{-1/2}e^{-y/2}

Step 5 — match against the chi-squared density

The $\chi^2_k$ density is

f(y) = \frac{1}{2^{k/2}\Gamma(k/2)}y^{k/2 - 1}e^{-y/2}

Set $k = 1$ . Then $y^{k/2-1} = y^{-1/2}$ ✓, and the constant is

\frac{1}{2^{1/2}\Gamma(1/2)} = \frac{1}{\sqrt{2}\cdot\sqrt{\pi}} = \frac{1}{\sqrt{2\pi}} \;\checkmark

using $\Gamma(\tfrac12) = \sqrt\pi$ . The two expressions are identical.

Answer

$Z^2 \sim \chi^2_1$ . The two branches of the squaring produced the factor of 2 that, combined with $\Gamma(\tfrac12) = \sqrt\pi$ , makes the constants match exactly.

python

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)

# Y = X² for X uniform: density 1/(2√y).
U = rng.random(500_000)
Y = U ** 2
q = np.linspace(0.05, 0.95, 5)
print("quantile  empirical   theory √q")
for p in q:
    print(f"{p:8.2f}  {np.quantile(Y, p):9.4f}   {p**2:9.4f}")

# Lognormal: mean vs median, the Jensen gap.
mu, sigma = 0.0, 1.0
X = rng.normal(mu, sigma, 1_000_000)
L = np.exp(X)
print(f"\nlognormal mean   {L.mean():.4f}   theory e^(μ+σ²/2) = {np.exp(mu + sigma**2/2):.4f}")
print(f"lognormal median {np.median(L):.4f}   theory e^μ       = {np.exp(mu):.4f}")
print(f"exp(E[X])        {np.exp(X.mean()):.4f}   <- NOT the mean of Y")

# Z² is chi-squared with 1 df.
Z = rng.standard_normal(1_000_000)
print(f"\nZ²: mean {np.mean(Z**2):.4f} (theory 1)  var {np.var(Z**2):.4f} (theory 2)")
print(f"KS test against chi2(1): p = {stats.kstest(Z**2, 'chi2', args=(1,)).pvalue:.3f}")

# Inverse transform sampling for the exponential.
E = -np.log(rng.random(500_000)) / 3.0
print(f"\ninverse transform exponential(3): mean {E.mean():.4f} (theory {1/3:.4f})")

Exercise 1

$X \sim \text{Exponential}(\lambda)$ and $Y = \sqrt{X}$ . Find the density of $Y$ .

Show solution

$g(x) = \sqrt x$ is increasing on $(0,\infty)$ , so the change-of-variables formula applies.

Invert and differentiate:

y = \sqrt x \quad\Longrightarrow\quad x = y^2, \qquad \frac{dx}{dy} = 2y

Apply the formula with $f_X(x) = \lambda e^{-\lambda x}$ :

f_Y(y) = \lambda e^{-\lambda y^2} \cdot 2y = 2\lambda y\, e^{-\lambda y^2}, \qquad y > 0

Check normalisation by substituting $u = y^2$ , $du = 2y\,dy$ :

\int_0^\infty 2\lambda y e^{-\lambda y^2}dy = \int_0^\infty \lambda e^{-\lambda u}\,du = 1 \;\checkmark

This is a Rayleigh distribution with scale $1/\sqrt{2\lambda}$ . It is the distribution of the magnitude of a two-dimensional vector with independent normal components, which is how it arises in signal processing and wind-speed modelling.

Exercise 2

Explain why a regression fitted to $\log(\text{revenue})$ under-predicts total revenue when predictions are exponentiated, and give the correction.

Show solution

A least-squares regression on $\log Y$ estimates $\E[\log Y \mid x]$ — the conditional mean of the log. Exponentiating gives

\exp\big(\E[\log Y \mid x]\big)

which, by the lognormal result, is the conditional median of $Y$ , not its mean:

\E[Y \mid x] = \exp\left(\E[\log Y \mid x] + \frac{\sigma^2}{2}\right)

where $\sigma^2$ is the variance of the residuals on the log scale. Since $e^{\sigma^2/2} > 1$ , the naive back-transform is always too small.

Because revenue is right-skewed, the mean exceeds the median, and summing medians across customers systematically undershoots the total. The error compounds: with residual standard deviation $\sigma = 0.6$ on the log scale, the correction factor is

e^{0.36/2} = e^{0.18} \approx 1.197

so totals are understated by about 20%.

The correction is to multiply predictions by $e^{\hat\sigma^2/2}$ , using the residual variance from the fitted log-scale model. A more robust alternative that does not assume lognormal residuals is Duan's smearing estimator: multiply by the mean of $e^{\hat\varepsilon_i}$ over the residuals.

The cleanest option when you need means rather than medians is to avoid the log transform altogether and fit a generalised linear model with a log link — a gamma or Poisson GLM — which models $\E[Y \mid x]$ directly and requires no back-transformation at all.

Next: Joint Continuous Distributions.