Bayes' Theorem

Intuition first

You know how often a symptom appears in sick people. You want to know how likely someone is sick given that they have the symptom. Those are different questions, and Bayes' theorem is the machinery for turning one into the other.

The reason the answer surprises people is that it depends on a third number they tend to ignore: how common the disease is in the first place. If almost nobody has the disease, then even a good test produces far more false alarms than true ones — simply because there are so many more healthy people to generate false alarms from.

That is the entire content of the theorem. Reversing a conditional probability requires knowing the base rate, and ignoring the base rate is the most expensive mistake in applied probability.

The statement

For events $A$ and $B$ with $\Prob(B) > 0$:

$$\Prob(A \mid B) = \frac{\Prob(B \mid A)\,\Prob(A)}{\Prob(B)}$$

Expanding the denominator with the law of total probability gives the form you will actually compute with:

$$\Prob(A \mid B) = \frac{\Prob(B \mid A)\Prob(A)}{\Prob(B \mid A)\Prob(A) + \Prob(B \mid A^c)\Prob(A^c)}$$

Derivation — two linesAdvanced

The multiplication rule can be written two ways, because $A \cap B$ and $B \cap A$ are the same event:

$$\Prob(A \cap B) = \Prob(A \mid B)\,\Prob(B)$$$$\Prob(A \cap B) = \Prob(B \mid A)\,\Prob(A)$$

Set the right-hand sides equal:

$$\Prob(A \mid B)\,\Prob(B) = \Prob(B \mid A)\,\Prob(A)$$

Divide by $\Prob(B)$, which is legitimate since $\Prob(B) > 0$:

$$\Prob(A \mid B) = \frac{\Prob(B \mid A)\,\Prob(A)}{\Prob(B)}$$

That is the whole proof. Bayes' theorem is not a deep new assumption — it is the symmetry of the joint probability, rearranged. The depth is in how it is used, not in what it says.

Why the base rate dominates

Write out the numerator and denominator as counts rather than probabilities. In a population of $N$ people:

• true positives: $N \times \Prob(A) \times \Prob(B \mid A)$
• false positives: $N \times \Prob(A^c) \times \Prob(B \mid A^c)$

The posterior is true positives divided by all positives. When $\Prob(A)$ is tiny, $\Prob(A^c) \approx 1$, so the false-positive count is governed almost entirely by the false-positive rate applied to nearly the whole population. A small rate times a large group easily exceeds a large rate times a small group.

Solved problem 1 · The medical test, in probabilities

A screening test detects 99% of true cases (sensitivity) and is 95% specific. The disease affects 1 in 200 people. A patient tests positive. What is the probability they have the disease?

Step 1 — translate the words into probabilities

Let $D$ = has the disease, $+$ = tests positive.

$$\Prob(D) = \tfrac{1}{200} = 0.005 \qquad \text{(prevalence, the prior)}$$$$\Prob(+ \mid D) = 0.99 \qquad \text{(sensitivity)}$$$$\Prob(- \mid D^c) = 0.95 \qquad \text{(specificity)}$$

Step 2 — derive the two quantities not stated directly

$$\Prob(D^c) = 1 - \Prob(D) = 1 - 0.005 = 0.995$$$$\Prob(+ \mid D^c) = 1 - \Prob(- \mid D^c) = 1 - 0.95 = 0.05$$

That second line is the false-positive rate. Specificity is about negatives, so it must be complemented before use.

Step 3 — the evidence, by the law of total probability

$$\Prob(+) = \Prob(+ \mid D)\Prob(D) + \Prob(+ \mid D^c)\Prob(D^c)$$$$= 0.99(0.005) + 0.05(0.995)$$$$= 0.00495 + 0.04975$$$$= 0.0547$$

Note the two terms: true positives contribute $0.00495$, false positives $0.04975$ — ten times more.

Step 4 — apply Bayes' theorem

$$\Prob(D \mid +) = \frac{\Prob(+ \mid D)\Prob(D)}{\Prob(+)} = \frac{0.00495}{0.0547} \approx 0.0905$$

Step 5 — check against the count version

Take 20,000 people: 100 ill, 19,900 healthy.

$$\text{true positives} = 0.99 \times 100 = 99$$$$\text{false positives} = 0.05 \times 19{,}900 = 995$$$$\Prob(D \mid +) = \frac{99}{99 + 995} = \frac{99}{1094} \approx 0.0905 \;\checkmark$$

Answer

$\Prob(D \mid +) \approx 0.0905$, about 9%. Despite a test that is right 99% of the time on sick people, a positive result leaves the patient more than 90% likely to be healthy.

The odds form

For repeated updates, the odds form is far more convenient. Divide Bayes' theorem for $A$ by Bayes' theorem for $A^c$ — the $\Prob(B)$ cancels:

$$\underbrace{\frac{\Prob(A \mid B)}{\Prob(A^c \mid B)}}_{\text{posterior odds}} = \underbrace{\frac{\Prob(B \mid A)}{\Prob(B \mid A^c)}}_{\text{likelihood ratio}} \times \underbrace{\frac{\Prob(A)}{\Prob(A^c)}}_{\text{prior odds}}$$

Deriving the odds form and why the constant vanishesAdvanced

Write Bayes' theorem for both $A$ and its complement:

$$\Prob(A \mid B) = \frac{\Prob(B \mid A)\Prob(A)}{\Prob(B)}, \qquad \Prob(A^c \mid B) = \frac{\Prob(B \mid A^c)\Prob(A^c)}{\Prob(B)}$$

Divide the first by the second. The $\Prob(B)$ in each denominator cancels:

$$\frac{\Prob(A \mid B)}{\Prob(A^c \mid B)} = \frac{\Prob(B \mid A)\Prob(A)}{\Prob(B \mid A^c)\Prob(A^c)}$$

which is the stated result. The practical gain is that $\Prob(B)$ — usually the most tedious part — never has to be computed. Recover the probability at the end with

$$\Prob(A \mid B) = \frac{\text{odds}}{1 + \text{odds}}$$

Taking logarithms makes it additive, which is exactly the structure of logistic regression: log-posterior-odds equals log-prior-odds plus a sum of log-likelihood ratios, one per feature. That connection is not a coincidence — naive Bayes and logistic regression are the same linear form fitted differently.

Solved problem 2 · The same problem in odds

Redo Solved problem 1 using the odds form.

Step 1 — prior odds

$$\frac{\Prob(D)}{\Prob(D^c)} = \frac{0.005}{0.995} = \frac{1}{199}$$

Odds of 1 to 199 against.

Step 2 — likelihood ratio

$$\frac{\Prob(+ \mid D)}{\Prob(+ \mid D^c)} = \frac{0.99}{0.05} = 19.8$$

A positive result is 19.8 times more likely in a sick person. This is the diagnostic value of the test, in one number.

Step 3 — multiply

$$\text{posterior odds} = 19.8 \times \frac{1}{199} = \frac{19.8}{199} \approx 0.09950$$

Step 4 — convert back to a probability

$$\Prob(D \mid +) = \frac{0.09950}{1 + 0.09950} \approx 0.0905 \;\checkmark$$

Answer

Same $\approx 0.0905$, with less arithmetic — and the evidence term never appeared.

Solved problem 3 · A second independent test

The patient from Solved problem 1 takes a second, independent test with the same characteristics. It is also positive. Now what?

Step 1 — the posterior becomes the new prior

After one positive, the odds were $19.8/199 \approx 0.09950$. Bayesian updating is sequential: yesterday's posterior is today's prior.

Step 2 — multiply by the likelihood ratio again

Conditional independence of the two tests given disease status means the likelihood ratio applies unchanged:

$$\text{posterior odds} = 19.8 \times 0.09950 \approx 1.9701$$

Step 3 — convert

$$\Prob(D \mid +, +) = \frac{1.9701}{1 + 1.9701} \approx 0.663$$

Answer

About 66% — up from 9%. Two independent positives move the conclusion from "probably fine" to "probably ill", which is precisely why confirmatory testing exists.

Common misreadings

Where this reappears

Bayes' theorem is not one topic among many; it is load-bearing across the whole curriculum:

• Naive Bayes classifiers apply it directly, with a conditional-independence assumption across features.
• Bayesian inference replaces events with parameters: $p(\theta \mid \text{data}) \propto p(\text{data} \mid \theta)\,p(\theta)$.
• MAP estimation maximises that posterior; maximum likelihood is the special case of a flat prior.
• Bayesian networks are factorised joint distributions queried with Bayes' rule and the law of total probability.
• Variational autoencoders approximate an intractable posterior $p(z \mid x)$, which is where the ELBO comes from.
• Precision in classification is a posterior: $\Prob(\text{positive class} \mid \text{predicted positive})$ — which is why precision collapses on imbalanced data exactly as the medical test does.

Exercise 1

A factory has two machines. Machine A makes 70% of items with a 1% defect rate; machine B makes 30% with a 4% defect rate. An item is found defective. What is the probability it came from machine B?

Show solutionHide solution

$$\Prob(D) = 0.01(0.70) + 0.04(0.30) = 0.007 + 0.012 = 0.019$$$$\Prob(B \mid D) = \frac{0.04(0.30)}{0.019} = \frac{0.012}{0.019} \approx 0.632$$

Machine B makes under a third of the output but produces nearly two thirds of the defects. Useful check: the two posteriors must sum to 1, and $0.007/0.019 \approx 0.368$, with $0.368 + 0.632 = 1 \;\checkmark$

Exercise 2

A spam filter flags 95% of spam and wrongly flags 2% of legitimate mail. 30% of incoming mail is spam. An email is flagged. What is the probability it is spam?

Show solutionHide solution

$$\Prob(\text{flag}) = 0.95(0.30) + 0.02(0.70) = 0.285 + 0.014 = 0.299$$$$\Prob(\text{spam} \mid \text{flag}) = \frac{0.285}{0.299} \approx 0.953$$

Contrast with the medical case: here the base rate is 30% rather than 0.5%, so the posterior is high. Same theorem, utterly different conclusion — the prior is doing the work.

Exercise 3

A test has sensitivity 0.90 and specificity 0.90. For which prevalence does a positive result give exactly $\Prob(D \mid +) = 0.5$?

Show solutionHide solution

Use the odds form. The likelihood ratio is

$$\frac{0.90}{1 - 0.90} = \frac{0.90}{0.10} = 9$$

A posterior probability of $0.5$ means posterior odds of exactly $1$. So

$$1 = 9 \times \text{prior odds} \quad\Longrightarrow\quad \text{prior odds} = \tfrac19$$

Converting odds to a probability:

$$\Prob(D) = \frac{1/9}{1 + 1/9} = \frac{1}{10} = 0.10$$

At 10% prevalence a positive result is a coin flip. Below that it is more likely a false alarm than a true one — a useful rule of thumb: a positive is only "probably right" when prevalence exceeds the reciprocal of the likelihood ratio.

---

Next: Random Variables, where we stop talking about events and start attaching numbers to outcomes — the step that makes expectation and variance possible.