The Idea of Changing Beliefs

If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing else about the person’s symptoms or signs?

Every once in a while, some variation of this question goes viral on social media, flooding the comments section with intuitive but incorrect answers until someone shows up and solves the problem mathematically (or @AskPerplexity it these days). This time, the question appeared in a reply to a tweet dunking on the new leadership of the U.S. Department of Health and Human Services (HHS).

When this question was originally proposed, an overwhelming majority of people, including doctors and medical students from Harvard teaching hospitals, thought the patient in the question had a 95% chance of actually having the disease when the correct answer is closer to just 2%.

While this question is phrased awkwardly and has some missing data, the reason why people make guesses closer to 95% instead of 2% is because of the counterintuitive nature of probability. After all, if the test only has a 5% false positive rate, i.e., incorrectly detecting a person has the disease while they don’t, it feels reasonable to guess that 95% of the time, the test would correctly detect the person has the disease.

But even if we assume the test could correctly detect a person has the disease when they actually have the disease with complete certainty, i.e., a 100% true positive rate, the answer is still 2%. Let’s break down the mathematics to see why.

Here’s what we know:

$$\text{Prevalence of the disease: } P(D) = \frac{1}{1000}$$

$$\text{False positive rate: } P(+ \mid \neg D) = 5\% = \frac{5}{100}$$

Note: $P(D)$ is the probability of having the disease before knowing the test result.

What we need to find is the probability that a person actually has the disease given that they tested positive, i.e., $P(D \mid +)$.

If we consider a sample of 1000 people where exactly 1 person has the disease (because $P(D) = \frac{1}{1000}$) and assume the test is perfectly accurate when detecting the disease, i.e., a 100% true positive rate ($P(+ \mid D)$), this is what the test results would look like:

1 person has the disease and tested positive (because $P(D) = \frac{1}{1000}$ and $P(+ \mid D) = 1$).

49 people don't have the disease but tested positive (because $P(+ \mid \neg D) = 5\% = \frac{5}{100} \approx \frac{49}{999}$, for easier calculation).

950 people don't have the disease and tested negative.

From our sample of 1000 people, 50 tested positive for the disease ( and ). So, if a person tests positive, we know they are either the 1 true positive or somewhere in the 49 false positives. Hence, the probability of having the disease, given a person tested positive, can be written as:

$$\begin{aligned} P(D \mid +) &= \frac{Number \ of \ true \ positives}{Total \ number \ of \ positives} \\[3ex] &= \frac{\color{#DC3545} Number \ of \ true \ positives}{\color{#DC3545} Number \ of \ true \ positives \color{normalcolor} + \color{#FFC107} Number \ of \ false \ positives} \\[3ex] &= \frac{\color{#DC3545} 1}{\color{#DC3545} 1 \color{normalcolor} + \color{#FFC107} 49} \\[3ex] &= \boxed{2\%} \end{aligned}$$

We can arrive at the same result through Bayes' theorem:

$$\begin{aligned} P(D \mid +) &= \frac{P(D \cap +)}{P(+)} \\[3ex] &= \frac{P(D \cap +)}{P(D \cap +) + P(\neg D \cap +)} \\[3ex] &= \frac{\frac{1}{1000}}{\frac{1}{1000} + \frac{49}{1000}} \\[3ex] &= \frac{1}{50} \\[3ex] &= \boxed{2\%} \end{aligned}$$

When we imagine a sample like this and apply the probabilities, arriving at 2% feels clear and intuitive. Then why do so many people get it wrong? When presented with a question like this, they often focus only on the test’s false positive rate (5%) and assume it means a 95% chance of being correct, ignoring the disease’s rarity.

Bayes’ theorem teaches us to update our prior beliefs—here, the 1 in 1000 prevalence of the disease—with new evidence, like the test result.

To make this idea stick, try the interactive example below and watch how the probability shifts. You can adjust the disease prevalence, the test’s sensitivity to detect the disease, and its specificity to accurately rule out the disease in healthy people.

Let's take this a step further and see what happens to the probability when a person with a positive test result takes another test like you usually do for confirmation.

We can use Bayes' Theorem again, but this time, the prior probability ($P(D)$) is 9.18%, i.e., the probability of having the disease before taking the second test. Assuming that we take the same test, the probability of having the disease given the person tested positive twice can be written as:

This result makes intuitive sense. When you test positive twice, the chances of having the disease are much higher than when you initially tested positive. This is the essence of Bayes’ theorem—with each bit of new information, we can update our beliefs incrementally. Try changing the probabilities again in the interactive example and see if you can build intuition.

Grant Sanderson, a.k.a 3Blue1Brown, has a series of videos on this topic that can help you learn more. I also recommend Veritasium (Derek Muller)'s video, where he talks specifically about the medical test paradox.