Bayes' Theorem

Bayes' theorem reverses the direction of a conditional probability. It connects the chance of evidence under a proposed event, $P(B\mid A)$, with the chance of that event after seeing the evidence, $P(A\mid B)$. These quantities use different reference groups. Updating requires both the evidence rate within the event and the event's initial probability, together with the total rate of the evidence across all relevant cases.

The theorem is:

$$ P(A|B) = \frac{P(B|A) \times P(A)}{P(B)} $$

where:

• $P(A|B)$ is the probability of event $A$ given that event $B$ has occurred,
• $P(B|A)$ is the probability of event $B$ given that event $A$ has occurred,
• $P(A)$ is the prior probability of event $A$,
• $P(B)$ is the overall probability of event $B$.

The formula assumes that $P(B) > 0$.

Derivation from Conditional Probability

Bayes' theorem follows directly from the definition of conditional probability. Recall that:

$$ P(A|B) = \frac{P(A \cap B)}{P(B)} $$

Similarly:

$$ P(B|A) = \frac{P(A \cap B)}{P(A)} $$

From the second equation, we can solve for the joint probability:

$$ P(A \cap B) = P(B|A) \times P(A) $$

Substituting this expression for $P(A \cap B)$ into the first equation gives Bayes' theorem:

$$ P(A|B) = \frac{P(B|A) \times P(A)}{P(B)} $$

When $P(B)$ is not known directly, it can often be calculated using the law of total probability. If $A$ and $A^c$, the complement of $A$, partition the sample space, then:

$$ P(B) = P(B|A) \times P(A) + P(B|A^c) \times P(A^c) $$

Substituting this into Bayes' theorem gives:

$$ P(A|B) = \frac{P(B|A)P(A)} {P(B|A)P(A) + P(B|A^c)P(A^c)} $$

This form is especially useful in diagnostic and classification problems, where the prior probability $P(A)$ and the likelihoods $P(B|A)$ and $P(B|A^c)$ are known.

For example, suppose $P(A)=0.3$ and $P(B)=0.4$. The diagram below can help visualize the relationship between the events:

However, these two marginal probabilities alone are not enough to calculate $P(A|B)$. We also need information about how $A$ and $B$ overlap, such as $P(A \cap B)$ or $P(B|A)$.

Example: Medical Diagnosis

Consider a hypothetical diagnostic test with the characteristics below. These values define a teaching probability model rather than describe a particular clinical test. Assume the rates apply to the population from which the person is selected:

• True positive rate (sensitivity): 95%. If a person has the disease, the test is positive 95% of the time.
• False positive rate: 5%. If a person does not have the disease, the test is positive 5% of the time.
• Prevalence of the disease in the general population: 2%.

Let $A$ be the event "the person has the disease" and $B$ be the event "the person tests positive."

We want to calculate:

$$ P(A|B) $$

Using Bayes' theorem:

$$ P(A|B) = \frac{P(B|A) \times P(A)}{P(B)} $$

First, calculate the total probability of a positive test result. A positive result can come from either a true positive or a false positive:

$$ P(B) = P(B|A) \times P(A) + P(B|A^c) \times P(A^c) $$

The given values are:

• True positive rate: $P(B|A) = 0.95$
• Disease prevalence: $P(A) = 0.02$
• False positive rate: $P(B|A^c) = 0.05$
• Probability of not having the disease: $P(A^c) = 0.98$

The total probability of a positive test is:

$$ P(B) = (0.95 \times 0.02) + (0.05 \times 0.98) $$

Therefore:

$$ P(B) = 0.019 + 0.049 = 0.068 $$

Now apply Bayes' theorem:

$$ P(A|B) = \frac{0.95 \times 0.02} {(0.95 \times 0.02) + (0.05 \times 0.98)} \approx 0.2794 $$

Therefore, given a positive test result, the probability that the person actually has the disease is approximately 27.94%.

This example shows why the prevalence of a condition matters. Even a test with high sensitivity can produce a substantial proportion of false positives when the condition itself is rare.

A frequency table gives the same result without reversing a conditional mentally. In a hypothetical population of 10,000 people, the model gives 200 with the condition and 9,800 without it:

Among the 680 positives, 190 have the condition, so the desired conditional probability is $190/680\approx0.2794$. Sensitivity instead uses the 200 people with the condition as its denominator: $190/200=0.95$. The shared numerator and different denominators explain why the two percentages differ.

Example: BSE "Mad Cow" Disease Test

As a second hypothetical calculation, suppose the event of interest is BSE (bovine spongiform encephalopathy) in a cow with a positive test. The rates below are assumed for the example; the calculation illustrates how sensitivity, false-positive rate, and prevalence jointly determine the posterior probability.

Given Quantities:

• True positive rate: $P(T|B) = 0.70$
• False positive rate: $P(T|B^c) = 0.10$
• Prior probability, or prevalence, of BSE: $P(B) = 0.02$

Here:

• $B$ is the event that the cow has BSE,
• $B^c$ is the event that the cow does not have BSE,
• $T$ is the event that the test result is positive.

Complementary Probability:

The probability that a cow does not have BSE is:

$$ P(B^c) = 1 - P(B) $$

so:

$$ P(B^c) = 1 - 0.02 = 0.98 $$

To find the conditional probability $P(B|T)$, we use Bayes' theorem:

$$ P(B|T) = \frac{P(T|B) \cdot P(B)} {P(T)} $$

The denominator $P(T)$ is the total probability of a positive test. It includes both true positives and false positives:

$$ P(T) = P(T|B) \cdot P(B) + P(T|B^c) \cdot P(B^c) $$

Substituting the given values:

$$ P(T) = 0.70 \cdot 0.02 + 0.10 \cdot 0.98 $$

Therefore:

$$ P(T) = 0.014 + 0.098 = 0.112 $$

Now apply Bayes' theorem:

$$ P(B|T) = \frac{0.70 \cdot 0.02} {0.70 \cdot 0.02 + 0.10 \cdot 0.98} = \frac{0.014}{0.112} = 0.125 $$

Thus, given a positive BSE test result, the probability that the cow actually has BSE is 12.5%.

As in the medical diagnosis example, the result depends not only on the accuracy of the test but also on the prior prevalence of the disease. When the disease is uncommon, false positives can make up a significant share of all positive test results.

Updating odds separates prevalence from evidence

When the relevant probabilities are positive, dividing the posterior probability of $A$ by that of $A^c$ cancels $P(B)$:

The first ratio on the right is a likelihood ratio. In the first example it is $0.95/0.05=19$, while prior odds are $0.02/0.98=1/49$. Posterior odds are therefore $19/49$, which convert to probability $19/(19+49)=19/68$. Strong evidence multiplies the prior odds; it does not erase the base rate. Applying the same test twice would require the joint conditional behavior of the two results. Squaring the likelihood ratio is justified only with conditional independence given the condition and its absence.