The confusion matrix
A handy little table that tells you how much to trust a medical test, and why "accurate" is never the whole story.
Introduction
Imagine you are an engineer trying to develop a test that identifies patients who have a disease. Every patient is either healthy or sick, and your test returns either a negative result ("healthy") or a positive result ("sick"). Cross those two facts and you get four possible outcomes. Laying them out in a two-by-two grid gives you a confusion matrix.
The problem is that there are several different ways a test can be right, and just as many ways it can be wrong. A test that never misses a sick patient might also raise a lot of false alarms. A test that never bothers a healthy person might let sick patients slip through. The confusion matrix lets us articulate those strengths and weaknesses quickly, which is why doctors, engineers, and regulators all lean on it.
Terminology
- Healthy: the patient does not have the disease.
- Sick: the patient does have the disease.
- Positive test: the test says the patient is sick.
- Negative test: the test says the patient is healthy.
- True positive \((TP)\): a correct call that the patient is sick.
- True negative \((TN)\): a correct call that the patient is healthy.
- False positive \((FP)\): the test says sick, but the patient is healthy. A false alarm.
- False negative \((FN)\): the test says healthy, but the patient is sick. A missed diagnosis.
"True" and "false" describe whether the test got it right. "Positive" and "negative" describe what the test said. Keep those two axes separate and the rest follows.
A worked example
Say there are 10 patients: 5 healthy and 5 sick. Our "test" is a single question, "Have you thrown up today?" Not a great test, but stick with me. Suppose 3 of the 5 sick patients have thrown up and none of the healthy patients have.
- True positives: 3 (sick, and the test caught them)
- False negatives: 2 (sick, but the test said healthy)
- True negatives: 5 (healthy, and the test agreed)
- False positives: 0 (no healthy patient was flagged)
This test is excellent at identifying healthy people (5 of 5) and mediocre at identifying sick people (3 of 5). In the language below, it is very specific but not very sensitive.
The metrics
Sensitivity: of all the sick patients, what fraction did the test catch? A sensitive test rarely misses disease.
$$Se = \frac{TP}{TP + FN}$$Specificity: of all the healthy patients, what fraction did the test correctly clear? A specific test rarely raises false alarms.
$$Sp = \frac{TN}{TN + FP}$$Positive predictive value: if the test comes back positive, what is the chance the patient is actually sick? This is the number a patient really wants to know, and it depends on how common the disease is.
$$PPV = \frac{TP}{TP + FP}$$Negative predictive value: if the test comes back negative, what is the chance the patient is actually healthy?
$$NPV = \frac{TN}{TN + FN}$$Accuracy: what fraction of all calls were right? Beware: if a disease is rare, a test that says "healthy" to everyone can be extremely accurate and completely useless.
$$Acc = \frac{TP + TN}{TP + TN + FP + FN}$$For the example above: sensitivity is 60%, specificity is 100%, PPV is 100%, NPV is 71%, and accuracy is 80%.
Which matters more?
In a perfect world every test would be right 100% of the time, and the only thing a clinician would need to know is which test to order. Unfortunately very few tests are perfect, so clinicians must weigh a test's accuracy against its cost and its risks, and we have to make trade-offs depending on what we are trying to detect.
Neither sensitivity nor specificity is more important in general. It depends on the situation. A screening test for a dangerous, treatable disease should be sensitive: missing a case is costly, and a false alarm just leads to a follow-up test. A confirmatory test before a risky surgery should be specific: you do not want to operate on someone who was never sick. Think about a pregnancy test. Is it more important to correctly identify someone who is pregnant, or to be able to tell someone definitively that they are not? Different people in different situations will answer that differently, and that is exactly the point.
Further reading
Wikipedia's confusion matrix article covers the full zoo of derived metrics. This page is my attempt at a clearer, more focused introduction from the standpoint of a diagnostic test.