Search :

2006-02-27-15 ROC Curves © Pedroli  www.thefetus.net/

ROC Curves

Gianluca Pedroli, MD

Humanitas Gavazzeni - Bergamo - Italy

A very useful test in medicine is to decide if a disease is present or not in a patient. Every test, like every other statistical measure, could have diagnostic errors and to assess the value of a diagnostic test we have to discuss some statistical terms.

Sensitivity is the proportion of patients with disease who test positive. (TP = true positive). In other words, sensitivity gives us the proportion of cases picked out by the test, relative to all cases that actually have the disease. This is the true positive rate of the test. The Positive Predictive value of a test is the proportion of patients with positive tests who have disease.

Specificity is the proportion of patients without disease who test negative. The specificity (true negative) is the probability to have a negative test for healthy people (true negative rate). The Negative Predictive value of a test is the proportion of patients with negative tests who do not have disease

These two rates should be the same in an ideal test but generally by increasing one measure, the other one will decrease; thus increasing the Sensitivity will result in a decrease of the Specificity. A good diagnostic test is that one that has small false positive and negative rates across a reasonable range of cut off values. A bad diagnostic test is one where the only cut offs that make the false positive rate low, have a high false negative rate and vice versa.

In the figure 1, we can see how a typical test does not distinguish, with 100% accuracy, the healthy person from the sick one. The plot represents the distributions of number of patients (frequency), affected and not, versus test value. The overlap area of these distributions is constituted by false positive (FP) and false negative (FN). The vertical line determines the cut off: There will be different cut off for different clinical situations.

Figure 1 ROC curves help us understanding the relationship between these two values and decide the best threshold for a particular application.

What is a ROC curves?
ROC is an acronym for “Receiver Operating Characteristic” which derives its name from a military field procedure developped during the Second World War to analyze sonar and radar signals.

An ROC curve is a graphical representation of the trade off between the false negative and false positive rates for every possible cut off. This graphical technique consists in plotting the true positive rate (TP) on y-axis against the false positive rate (FP= 1 – TN) on x-axis. In this way, it is possible to trace a curve which points represent the test performances for every cut off value.

Example: For example, we consider the following data obtained by a virtual study about nuchal translucency (NT). In fact we suppose to have 280 patients for which we have evaluated whether the translucency value was a good predictor for fetal abnormalities.

 Translucency value (mm) Normal karyotype Abnormal karyotype < 1 5 0 1.0 - 1.5 58 2 1.5 - 2.0 101 6 2.0 - 2.5 43 2 2.5 - 3.0 24 3 3.0 - 4.0 13 3 4.0 - 5.0 4 3 5.0 - 10.0 2 7 > 10.0 0 4 Total 250 30

Plotting these data, we obtain the following diagram: Where an NT value ~3.0 is a good cut off for screening chromosomal abnormalities, in which the sensitivity is about 0.67 and the specificity is about 0.83.

 Translucency value (mm) Sensitivity Specificity < 1 1.0 0.0 1.0 -1.5 1.0 0.02 1.5 - 2.0 0.93 0.25 2.0 - 2.5 0.73 0.66 2.3 - 3.0 0.67 0.83 3.0 - 4.0 0.57 0.92 4.0 - 5.0 0.43 0.98 5.0 - 10.0 0.36 0.99 >10.0 0.13 1.0

Plotting sensitivity versus (1 – specificity) we obtain the ROC curve bellow (Figure 3) The diagonal red line from (0,0) point in the lower left hand corner to (1,1) point in the upper right hand corner is typical of a test without a true discriminating power; otherwise, the line connecting  (1,0) point  to (0,1) point is typical of a test for which the sensitivity is equal to specificity (see the next section).
Like we can observe, ROC curve always start from (0,0) point, at  which point none of the patients are sick, to end to (1,1) point, where all the patients are sick.

Comparing ROC curves

(0,1) represents the ideal test point (when TP = TN), and the more a ROC curve will arch towards this point the better the test will be (Figure 4 ). The faster the ROC curve climbs towards the upper left hand corner of the graph more the test will be useful. The point (0,1) represents the ideal test point (when TP = TN), and the more a ROC curve will arch towards this point the better the test will be.
(Figure 4 ).

Moreover, the tangent line slope to a cut off will give us the likelihood ratio (LR) for that test value.
The likelihood rate is a number that tells us:

• How much a positive test is predictive for the real presence of disease (likelihood ratio of a positive test).
• How much a negative test is predictive for the real absence of disease (likelihood ratio of a negative test).
• It is a ratio of these two possibilities: the probablility of a test result among people with a disease divided by the probability of the test result among people without the disease

In other words, in terms of sensitivity and specificity, we can write:

• + LR = sensitivity/(1-specificity)
• - LR = (1- sensitivity)/specificity

The LR range can go from 0 to ∞ values, in particular:

• for values ~1 it doesn’t add additional information to disease likelihood;
• for values > 1 it increases disease likelihood
• for values < 1 it decreases disease likelihood

In our study case, the +LR value corresponding to cut off point is ~3.9. By increasing NT value from 4.0 to 10.0, +LR increases from ~7.1 to ∞, showing that the higher the NT value (over 3 mm), the more likely we have disease likelihood.
Finally, when the ROC curve follows a diagonal path from the lower left hand corner to the upper right hand corner, we will have a test that is not the ideal one. This means that every improvement in false positive rate is matched by a corresponding decline in the false negative rate.

Measuring the area under the curve

Measuring the area under the ROC curve, we can obtain the accuracy of the test. The larger area, the better the diagnostic test is. If the area is 1.0, we have an ideal test because test achieves 100% sensitivity and 100% specificity. If the area is 0.5, we have a test which has effectively 50% sensitivity and 50% specificity. In a few words, the area measures the ability of the test to correctly classify those with and without the disease. And to calculate the area, you can use the following formula: Where t = 1 - specificity (false positive rate) and ROC (t) is sensitivity (true positive rate).
So, we can establish the following classification for the test:

 0.9 < AUC < 1.0 excellent 0.8 < AUC < 0.9 good 0.7 < AUC < 0.8 worthless 0.6 < AUC 0.7 not good 0.5 < AUC < 0.6 failed

REFERENCES

1. Acacio G. L., Barini R., Pinto W. J., Silveira Ximenes R. L., Pettersen H, Faria M., 2001: “Nuchal translucency: an ultrasound marker for fetal chromosomal abnormalities”, Sao Paulo Med J/Rev Paul Med, 119 (1):19-23.
2. Crichton Nichola, 2002: “Receiver Operating  Characteristic (ROC) curves”, Journal of Clinical Nursing, 11, 134-136.
3. Tape G. T., “Introduction to Likelihood ratios”,
http://gim.unmc.edu/dxtests/introduc.htm.
4. Tape G. T., “Introduction to Roc Curves”,
http://gim.unmc.edu/dxtests/ROC1.htm.
5. Tape G. T., “The area under an ROC curves”,
http://gim.unmc.edu/dxtests/ROC3.htm.
6. Zoccali C., “Le basi statistiche ed epidemiologiche della medicina clinica”, cap.4.

Help Support TheFetus.net :