View source for Receiver operating characteristic

{{Short description|Diagnostic plot of binary classifier ability}}
{{More citations needed|date=September 2023}}
[[File:Roccurves.png|thumb|ROC curve of three predictors of peptide cleaving in the [[proteasome]].]]
A '''receiver operating characteristic curve''', or '''ROC curve''', is a [[graph of a function|graphical plot]] that illustrates the performance of a [[binary classifier]] model (can be used for multi class classification as well) at varying threshold values.

The ROC curve is the plot of the [[true positive rate]] (TPR) against the [[false positive rate]] (FPR) at each threshold setting.

The ROC can also be thought of as a plot of the [[statistical power]] as a function of the [[Type I Error]] of the decision rule (when the performance is calculated from just a sample of the population, it can be thought of as estimators of these quantities). The ROC curve is thus the sensitivity as a function of [[false positive rate]].

Given that the [[probability distribution]]s for both true positive and false positive are known, the ROC curve is obtained as the [[cumulative distribution function]] (CDF, area under the probability distribution from <math>-\infty</math> to the discrimination threshold) of the detection probability in the ''y''-axis versus the CDF of the false positive probability on the ''x''-axis.

ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. ROC analysis is related in a direct and natural way to the cost/benefit analysis of diagnostic [[decision making]].

== Terminology ==

The true-positive rate is also known as [[Sensitivity (tests)|sensitivity]] or ''probability of detection''.<ref name="matlab">{{cite web |title=Detector Performance Analysis Using ROC Curves - MATLAB & Simulink Example |url=http://www.mathworks.com/help/phased/examples/detector-performance-analysis-using-roc-curves.html |access-date=11 August 2016 |website=www.mathworks.com}}</ref> The false-positive rate is also known as the ''probability of false alarm''<ref name="matlab" /> and equals (1 − [[Specificity (tests)|specificity]]). 
The ROC is also known as a relative operating characteristic curve, because it is a comparison of two operating characteristics (TPR and FPR) as the criterion changes.<ref name="Swets1996">Swets, John A.; [https://www.questia.com/PM.qst?a=o&d=91082370 ''Signal detection theory and ROC analysis in psychology and diagnostics : collected papers''], Lawrence Erlbaum Associates, Mahwah, NJ, 1996</ref>

== History ==
The ROC curve was first developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battlefields, starting in 1941, which led to its name ("receiver operating characteristic").<ref>{{Cite journal |date=May 3, 2024 |title= ROC Solid: Receiver Operator Characteristic (ROC) Curves as a Foundation for Better Diagnostic Tests|pmc= 6022965|last1= Junge|first1= M. R.|last2= Dettori|first2= J. R.|journal= Global Spine Journal|volume= 8|issue= 4|pages= 424–429|doi= 10.1177/2192568218778294|pmid= 29977728}}</ref>

It was soon introduced to [[psychology]] to account for the perceptual detection of stimuli. ROC analysis has been used in [[medicine]], [[radiology]], [[biometrics]], [[forecasting]] of [[natural hazard]]s,<ref>{{Cite journal|last1=Peres|first1=D. J.|last2=Cancelliere|first2=A.|date=2014-12-08|title=Derivation and evaluation of landslide-triggering thresholds by a Monte Carlo approach|journal=Hydrol. Earth Syst. Sci.|volume=18|issue=12|pages=4913–4931|doi=10.5194/hess-18-4913-2014|issn=1607-7938|bibcode=2014HESS...18.4913P|doi-access=free}}</ref> [[meteorology]],<ref>{{Cite journal|last=Murphy|first=Allan H.|date=1996-03-01|title=The Finley Affair: A Signal Event in the History of Forecast Verification|journal=Weather and Forecasting|volume=11|issue=1|pages=3–20|doi=10.1175/1520-0434(1996)011<0003:tfaase>2.0.co;2|issn=0882-8156|bibcode=1996WtFor..11....3M|doi-access=free}}</ref> model performance assessment,<ref>{{Cite journal|last1=Peres|first1=D. J.|last2=Iuppa|first2=C.|last3=Cavallaro|first3=L.|last4=Cancelliere|first4=A.|last5=Foti|first5=E.|date=2015-10-01|title=Significant wave height record extension by neural networks and reanalysis wind data|journal=Ocean Modelling|volume=94|pages=128–140|doi=10.1016/j.ocemod.2015.08.002|bibcode=2015OcMod..94..128P}}</ref> and other areas for many decades and is increasingly used in [[machine learning]] and [[data mining]] research.

==Basic concept==
{{See also|Type I and type II errors|Sensitivity and specificity}}
A classification model ([[Classifier (mathematics)|classifier]] or [[medical diagnosis|diagnosis]]<ref name="Sushkova">{{cite journal |last1=Sushkova |first1=Olga |last2=Morozov |first2=Alexei |last3=Gabova |first3=Alexandra |last4=Karabanov |first4=Alexei |last5=Illarioshkin |first5 =Sergey |journal=Sensors |number=14 |pages=4700 |title=A Statistical Method for Exploratory Data Analysis Based on 2D and 3D Area under Curve Diagrams: Parkinson's Disease Investigation |volume=21 |year=2021 |pmid=34300440 |doi=10.3390/s21144700| pmc=8309570 |bibcode=2021Senso..21.4700S |doi-access=free }}</ref>) is a [[Mapping (mathematics)|mapping]] of instances between certain classes/groups. Because the classifier or diagnosis result can be an arbitrary [[Real number|real value]] (continuous output), the classifier boundary between classes must be determined by a threshold value (for instance, to determine whether a person has [[hypertension]] based on a [[blood pressure]] measure). Or it can be a [[Countable set|discrete]] class label, indicating one of the classes.

Consider a two-class prediction problem ([[binary classification]]), in which the outcomes are labeled either as positive (''p'') or negative (''n''). There are four possible outcomes from a binary classifier. If the outcome from a prediction is ''p'' and the actual value is also ''p'', then it is called a ''true positive'' (TP); however if the actual value is ''n'' then it is said to be a ''false positive'' (FP). Conversely, a ''true negative'' (TN) has occurred when both the prediction outcome and the actual value are ''n'', and a ''false negative'' (FN) is when the prediction outcome is ''n'' while the actual value is ''p''.

To get an appropriate example in a real-world problem, consider a diagnostic test that seeks to determine whether a person has a certain disease. A false positive in this case occurs when the person tests positive, but does not actually have the disease. A false negative, on the other hand, occurs when the person tests negative, suggesting they are healthy, when they actually do have the disease.

Consider an experiment from '''P''' positive instances and '''N''' negative instances for some condition. The four outcomes can be formulated in a 2×2 ''[[contingency table]]'' or ''[[confusion matrix]]'', as follows:

{{diagnostic testing diagram}}
<!-- 
{| align=center
|-
! colspan=2 | &nbsp;
! colspan=2 align=center | actual value
|-
! colspan=2 | &nbsp; !! ''p'' !! ''n'' !! style="padding-left:1em;" | total
|-
! rowspan=2 valign=middle | prediction<br 
/>outcome
! valign=middle style="padding-right:1em;" | ''p'''
| style="border:thin solid; padding:1em;" | True<br />Positive
| style="border:thin solid; padding:1em;" | False<br />Positive
| style="padding-left:1em;" | P'
|-
! valign=middle style="padding-right:1em;" | ''n'''
| style="border:thin solid; padding:1em;" | False<br />Negative
| style="border:thin solid; padding:1em;" | True<br />Negative
| style="padding-left:1em;" | N'
|-
! colspan=2 align=right style="padding-right:1em;" | total
| align=center| P || align=center | N
|}.
-->

==ROC space==
[[Image:ROC space-2.png|thumb|right|The ROC space and plots of the four prediction examples.]]
[[Image:roc_curve.svg|thumb|right|The ROC space for a "better" and "worse" classifier.]]

The contingency table can derive several evaluation "metrics" (see infobox). To draw a ROC curve, only the true positive rate (TPR) and false positive rate (FPR) are needed (as functions of some classifier parameter). The TPR defines how many correct positive results occur among all positive samples available during the test. FPR, on the other hand, defines how many incorrect positive results occur among all negative samples available during the test.

A ROC space is defined by FPR and TPR as ''x'' and ''y'' axes, respectively, which depicts relative trade-offs between true positive (benefits) and false positive (costs). Since TPR is equivalent to sensitivity and FPR is equal to 1 − [[specificity (tests)|specificity]], the ROC graph is sometimes called the sensitivity vs (1 − specificity) plot. Each prediction result or instance of a [[confusion matrix]] represents one point in the ROC space.

The best possible prediction method would yield a point in the upper left corner or coordinate (0,1) of the ROC space, representing 100% sensitivity (no false negatives) and 100% [[specificity (tests)|specificity]] (no false positives). The (0,1) point is also called a ''perfect classification''. A random guess would give a point along a diagonal line (the so-called ''line of no-discrimination'') from the bottom left to the top right corners (regardless of the positive and negative [[base rate]]s).<ref>{{Cite web|title=classification - AUC-ROC of a random classifier|url=https://datascience.stackexchange.com/a/31877/108151|access-date=2020-11-30|website=Data Science Stack Exchange}}</ref> An intuitive example of random guessing is a decision by flipping coins. As the size of the sample increases, a random classifier's ROC point tends towards the diagonal line. In the case of a balanced coin, it will tend to the point (0.5, 0.5).

The diagonal divides the ROC space. Points above the diagonal represent good classification results (better than random); points below the line represent bad results (worse than random). Note that the output of a consistently bad predictor could simply be inverted to obtain a good predictor.

Consider four prediction results from 100 positive and 100 negative instances:

{| class="wikitable"
|+
! A !! B !! C !! C′
|-
|
{| style="text-align:center;"
| style=" border:thin solid; padding:1em;" | TP&nbsp;=&nbsp;63 || style=" border:thin solid; padding:1em;" | FN = 37 || 100
|-
| style=" border:thin solid; padding:1em;" | FP&nbsp;=&nbsp;28 || style=" border:thin solid; padding:1em;" | TN = 72 || 100
|-
| 91 || 109 || 200
|}
| style="padding-left:1em;" |
{| style="text-align:center;"
| style=" border:thin solid; padding:1em;" | TP&nbsp;=&nbsp;77 || style=" border:thin solid; padding:1em;" | FN = 23 || 100
|-
| style=" border:thin solid; padding:1em;" | FP&nbsp;=&nbsp;77 || style=" border:thin solid; padding:1em;" | TN = 23 || 100
|-
| 154 || 46 || 200
|}
| style="padding-left:1em;" |
{| style="text-align:center;"
| style=" border:thin solid; padding:1em;" | TP&nbsp;=&nbsp;24 || style=" border:thin solid; padding:1em;" | FN = 76 || 100
|-
| style=" border:thin solid; padding:1em;" | FP&nbsp;=&nbsp;88 || style=" border:thin solid; padding:1em;" | TN = 12 || 100
|-
| 112 || 88 || 200
|}
| style="padding-left:1em;" |
{| style="text-align:center;"
| style=" border:thin solid; padding:1em;" | TP&nbsp;=&nbsp;76 || style=" border:thin solid; padding:1em;" | FN = 24 || 100
|-
| style=" border:thin solid; padding:1em;" | FP&nbsp;=&nbsp;12 || style=" border:thin solid; padding:1em;" | TN = 88 || 100
|-
| 88 || 112 || 200
|}
|-
| style="padding-left:1em;" | TPR = 0.63 || style="padding-left:2em;" | TPR = 0.77 || style="padding-left:2em;" | TPR = 0.24 || style="padding-left:2em;" | TPR = 0.76
|-
| style="padding-left:1em;" | FPR = 0.28 || style="padding-left:2em;" | FPR = 0.77 || style="padding-left:2em;" | FPR = 0.88 || style="padding-left:2em;" | FPR = 0.12
|-
| style="padding-left:1em;" | PPV = 0.69 || style="padding-left:2em;" | PPV = 0.50 || style="padding-left:2em;" | PPV = 0.21 || style="padding-left:2em;" | PPV = 0.86
|-
| style="padding-left:1em;" | F1 = 0.66 || style="padding-left:2em;" | F1 = 0.61 || style="padding-left:2em;" | F1 = 0.23 || style="padding-left:2em;" | F1 = 0.81
|-
| style="padding-left:1em;" | ACC = 0.68 || style="padding-left:2em;" | ACC = 0.50 || style="padding-left:2em;" | ACC = 0.18 || style="padding-left:2em;" | ACC = 0.82
|}

Plots of the four results above in the ROC space are given in the figure. The result of method '''A''' clearly shows the best predictive power among '''A''', '''B''', and '''C'''. The result of '''B''' lies on the random guess line (the diagonal line), and it can be seen in the table that the [[Evaluation of binary classifiers#Single metrics|accuracy]] of '''B''' is 50%. However, when '''C''' is mirrored across the center point (0.5,0.5), the resulting method '''C′''' is even better than '''A'''. This mirrored method simply reverses the predictions of whatever method or test produced the '''C''' contingency table. Although the original '''C''' method has negative predictive power, simply reversing its decisions leads to a new predictive method '''C′''' which has positive predictive power. When the '''C''' method predicts '''p''' or '''n''', the '''C′''' method would predict '''n''' or '''p''', respectively. In this manner, the '''C′''' test would perform the best. The closer a result from a contingency table is to the upper left corner, the better it predicts, but the distance from the random guess line in either direction is the best indicator of how much predictive power a method has. If the result is below the line (i.e. the method is worse than a random guess), all of the method's predictions must be reversed in order to utilize its power, thereby moving the result above the random guess line.

==Curves in ROC space==
[[File:ROC curves.svg|thumb|right|300px]]
In binary classification, the class prediction for each instance is often made based on a [[Continuous probability distribution|continuous random variable]] <math> X </math>, which is a "score" computed for the instance (e.g. the estimated probability in logistic regression). Given a threshold parameter <math> T </math>, the instance is classified as "positive" if <math> X>T </math>, and "negative" otherwise. <math> X </math> follows a probability density <math> f_1 (x) </math> if the instance actually belongs to class "positive", and <math> f_0 (x) </math> if otherwise. Therefore, the true positive rate is given by <math> \mbox{TPR}(T)= \int_{T}^\infty f_1(x) \, dx </math> and the false positive rate is given by <math> \mbox{FPR}(T)= \int_{T}^\infty f_0(x) \, dx </math>. 
The ROC curve plots parametrically <math>\mbox{TPR}(T)</math> versus <math>\mbox{FPR}(T)</math> with <math>T</math> as the varying parameter.

For example, imagine that the blood protein levels in diseased people and healthy people are [[normal distribution|normally distributed]] with means of 2 [[gram|g]]/[[decilitre|dL]] and 1 g/dL respectively. A medical test might measure the level of a certain protein in a blood sample and classify any number above a certain threshold as indicating disease. The experimenter can adjust the threshold (green vertical line in the figure), which will in turn change the false positive rate. Increasing the threshold would result in fewer false positives (and more false negatives), corresponding to a leftward movement on the curve. The actual shape of the curve is determined by how much overlap the two distributions have.

==Criticisms==
[[File:ROC curve example highlighting sub-area with low sensitivity and low specificity.png|thumb|Example of receiver operating characteristic (ROC) curve highlighting the area under the curve (AUC) sub-area with low sensitivity and low specificity in red and the sub-area with high or sufficient sensitivity and specificity in green.<ref name="Chicco Jurman 2023 p. "/>]]

Several studies criticize certain applications of the ROC curve and its area under the curve as measurements for assessing binary classifications when they do not capture the information relevant to the application.<ref name="Muschelli 2019 pp. 696–708">{{cite journal | last=Muschelli | first=John | title=ROC and AUC with a binary predictor: a potentially misleading metric | journal=Journal of Classification | publisher=Springer Science and Business Media LLC | volume=37 | issue=3 | date=2019-12-23 | issn=0176-4268 | doi=10.1007/s00357-019-09345-1 | pages=696–708| pmid=33250548 | pmc=7695228 }}</ref><ref name="Chicco Jurman 2023 p. ">{{cite journal | last1=Chicco | first1=Davide | last2=Jurman | first2=Giuseppe | title=The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification | journal=BioData Mining | publisher=Springer Science and Business Media LLC | volume=16 | issue=1 | date=2023-02-17 | issn=1756-0381 | doi=10.1186/s13040-023-00322-4 | page=4| doi-access=free | pmid=36800973 | pmc=9938573 | hdl=10281/430042 | hdl-access=free }}</ref><ref name="Lobo Jiménez-Valverde Real 2008 pp. 145–151">{{cite journal | last1=Lobo | first1=Jorge M. | last2=Jiménez-Valverde | first2=Alberto | last3=Real | first3=Raimundo | title=AUC: a misleading measure of the performance of predictive distribution models | journal=Global Ecology and Biogeography | publisher=Wiley | volume=17 | issue=2 | year=2008 | issn=1466-822X | doi=10.1111/j.1466-8238.2007.00358.x | pages=145–151| bibcode=2008GloEB..17..145L }}</ref><ref name="Halligan Altman Mallett 2015 pp. 932–939">{{cite journal | last1=Halligan | first1=Steve | last2=Altman | first2=Douglas G. | last3=Mallett | first3=Susan | title=Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach | journal=European Radiology | publisher=Springer Science and Business Media LLC | volume=25 | issue=4 | date=2015-01-20 | issn=0938-7994 | doi=10.1007/s00330-014-3487-0 | pages=932–939| doi-access=free | pmid=25599932 | pmc=4356897 }}</ref><ref name="Berrar Flach 2011 pp. 83–97">{{cite journal | last1=Berrar | first1=D. | last2=Flach | first2=P. | title=Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them) | journal=Briefings in Bioinformatics | publisher=Oxford University Press (OUP) | volume=13 | issue=1 | date=2011-03-21 | issn=1467-5463 | doi=10.1093/bib/bbr008 | pages=83–97| doi-access=free | pmid=21422066 }}</ref>

The main criticism to the ROC curve described in these studies regards the incorporation of areas with low sensitivity and low [[specificity (tests)|specificity]] (both lower than 0.5) for the calculation of the total area under the curve (AUC).,<ref name="Lobo Jiménez-Valverde Real 2008 pp. 145–151" /> as described in the plot on the right.

According to the authors of these studies, that portion of area under the curve (with low sensitivity and low specificity) regards confusion matrices where binary predictions obtain bad results, and therefore should not be included for the assessment of the overall performance.
Moreover, that portion of AUC indicates a space with high or low confusion matrix threshold which is rarely of interest for scientists performing a binary classification in any field.<ref name="Lobo Jiménez-Valverde Real 2008 pp. 145–151" />

Another criticism to the ROC and its area under the curve is that they say nothing about precision and negative predictive value.<ref name="Chicco Jurman 2023 p. "/>

A high ROC AUC, such as 0.9 for example, might correspond to low values of precision and negative predictive value, such as 0.2 and 0.1 in the [0, 1] range.
If one performed a binary classification, obtained an ROC AUC of 0.9 and decided to focus only on this metric, they might overoptimistically believe their binary test was excellent. However, if this person took a look at the values of precision and negative predictive value, they might discover their values are low.

The ROC AUC summarizes sensitivity and specificity, but does not inform regarding precision and negative predictive value.<ref name="Chicco Jurman 2023 p. "/>

==Further interpretations==

Sometimes, the ROC is used to generate a summary statistic. Common versions are:
* the intercept of the ROC curve with the line at 45 degrees orthogonal to the no-discrimination line - the balance point where [[Sensitivity and specificity|Sensitivity]] = [[Specificity (statistics)|Specificity]] 
* the intercept of the ROC curve with the tangent at 45 degrees parallel to the no-discrimination line that is closest to the error-free point (0,1) – also called [[Youden's J statistic]] and generalized as Informedness{{citation needed|date=November 2019}}
* the area between the ROC curve and the no-discrimination line multiplied by two is called the ''Gini coefficient'', especially in the context of [[credit scoring]].<ref>{{cite journal | vauthors=((Řezáč, M.)), ((Řezáč, F.)) | journal=Czech Journal of Economics and Finance (Finance a úvěr) | title=How to Measure the Quality of Credit Scoring Models | volume=61 | issue=5 | pages=486–507 | publisher=Charles University Prague, Faculty of Social Sciences | date= 2011}}
</ref>  It should not be confused with the [[Gini coefficient|measure of statistical dispersion also called Gini coefficient]].
* the area between the full ROC curve and the triangular ROC curve including only (0,0), (1,1) and one selected operating point <math>(tpr,fpr)</math> – Consistency<ref name="Powers2012a">{{cite conference |title=ROC-ConCert: ROC-Based Measurement of Consistency and Certainty|first=David MW |last=Powers|year=2012|book-title=Spring Congress on Engineering and Technology (SCET)|volume=2|pages=238–241|publisher=IEEE|url=http://www.academia.edu/download/31939951/201203-SCET30795-ROC-ConCert-PID1124774.pdf}}{{dead link|date=July 2022|bot=medic}}{{cbignore|bot=medic}}</ref>
* the area under the ROC curve, or "AUC" ("area under curve"), or A' (pronounced "a-prime"),<ref>{{cite conference |first1=James |last1=Fogarty |first2=Ryan S. |last2=Baker |first3=Scott E. |last3=Hudson |year=2005 |title=Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction |book-title=ACM International Conference Proceeding Series, Proceedings of Graphics Interface 2005 |publisher=Canadian Human-Computer Communications Society |location=Waterloo, ON |url=http://portal.acm.org/citation.cfm?id=1089530 }}</ref> or "c-statistic" ("concordance statistic").<ref>{{cite book |first1=Trevor |last1=Hastie|author-link1=Trevor Hastie|first2=Robert |last2=Tibshirani |author-link2=Robert Tibshirani|first3=Jerome H. |last3=Friedman |year=2009 |title=The elements of statistical learning: data mining, inference, and prediction |edition=2nd}}</ref>
* the [[sensitivity index|sensitivity index ''d′'']] (pronounced "d-prime"), the distance between the mean of the distribution of activity in the system under noise-alone conditions and its distribution under signal-alone conditions, divided by their [[standard deviation]], under the assumption that both these distributions are [[normal distribution|normal]] with the same standard deviation. Under these assumptions, the shape of the ROC is entirely determined by ''d′''.

However, any attempt to summarize the ROC curve into a single number loses information about the pattern of tradeoffs of the particular discriminator algorithm.

===Probabilistic interpretation===
The area under the curve (often referred to as simply the AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative').<ref name="fawcett">Fawcett, Tom (2006); ''[https://www.math.ucdavis.edu/~saito/data/roc/fawcett-roc.pdf An introduction to ROC analysis]'', Pattern Recognition Letters, 27, 861–874.</ref> In other words, when given one randomly selected positive instance and one randomly selected negative instance, AUC is the probability that the classifier will be able to tell which one is which.

This can be seen as follows: the area under the curve is given by (the integral boundaries are reversed as large threshold <math> T </math> has a lower value on the ''x''-axis)
:<math>\operatorname{TPR}(T): T \to y(x)</math>
:<math>\operatorname{FPR}(T): T \to x</math>
:<math>
\begin{align}
A & = \int_{x=0}^1 \mbox{TPR}(\mbox{FPR}^{-1}(x)) \, dx \\[5pt]
& = \int_{\infty}^{-\infty} \mbox{TPR}(T) \mbox{FPR}'(T) \, dT \\[5pt]
& = \int_{-\infty}^\infty \int_{-\infty}^\infty I(T' \ge T)f_1(T') f_0(T) \, dT' \, dT = P(X_1 \ge X_0)
\end{align}
</math>
where <math> X_1 </math> is the score for a positive instance and <math> X_0 </math> is the score for a negative instance, and <math>f_0</math> and <math>f_1</math> are probability densities as defined in previous section.

If <math> X_0 </math> and <math> X_1 </math> follows two Gaussian distributions, then <math> A = \Phi\left((\mu_1-\mu_0)/\sqrt{\sigma_1^2 + \sigma_0^2}\right) </math>.


===Area under the curve===
It can be shown that the AUC is closely related to the [[Mann–Whitney U]],<ref name="Hanley">{{cite journal |last1=Hanley |first1=James A. |last2=McNeil |first2=Barbara J. |s2cid=10511727 |author-link2=Barbara Joyce McNeil |journal=Radiology |number=1 |pages=29–36 |title=The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve |volume=143 |year=1982 |pmid=7063747 |doi=10.1148/radiology.143.1.7063747}}</ref><ref name="Mason">{{cite journal |last1=Mason |first1=Simon J. |last2=Graham |first2=Nicholas E. |year=2002 |title=Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation |journal=Quarterly Journal of the Royal Meteorological Society |volume=128 |issue=584 |pages=2145–2166 |url=http://www.inmet.gov.br/documentos/cursoI_INMET_IRI/Climate_Information_Course/References/Mason+Graham_2002.pdf |archive-url=https://web.archive.org/web/20081120134338/http://www.inmet.gov.br/documentos/cursoI_INMET_IRI/Climate_Information_Course/References/Mason%2BGraham_2002.pdf |url-status=dead |archive-date=2008-11-20 |doi=10.1256/003590002320603584 |citeseerx=10.1.1.458.8392 |bibcode=2002QJRMS.128.2145M |s2cid=121841664 }}</ref> which tests whether positives are ranked higher than negatives. For a predictor <math display="inline">f</math>, an unbiased estimator of its AUC can be expressed by the following ''Wilcoxon-Mann-Whitney'' statistic:<ref>{{Cite book|last1=Calders|first1=Toon|last2=Jaroszewicz|first2=Szymon|date=2007|editor-last=Kok|editor-first=Joost N.|editor2-last=Koronacki|editor2-first=Jacek|editor3-last=Lopez de Mantaras|editor3-first=Ramon|editor4-last=Matwin|editor4-first=Stan|editor5-last=Mladenič|editor5-first=Dunja|editor6-last=Skowron|editor6-first=Andrzej|chapter=Efficient AUC Optimization for Classification|title=Knowledge Discovery in Databases: PKDD 2007|series=Lecture Notes in Computer Science|volume=4702|language=en|location=Berlin, Heidelberg|publisher=Springer|pages=42–53|doi=10.1007/978-3-540-74976-9_8|isbn=978-3-540-74976-9|doi-access=free}}</ref>

: <math>\text{AUC}(f) = 
  \frac{\sum _{t_0 \in \mathcal{D}^0} \sum _{t_1 \in \mathcal{D}^1} 
  \textbf{1}[f(t_0) < f(t_1)]}{|\mathcal{D}^0| \cdot |\mathcal{D}^1|},
</math>

where <math display="inline">\textbf{1}[f(t_0) < f(t_1)]</math> denotes an ''indicator function'' which returns 1 if <math>f(t_0) < f(t_1)</math> otherwise return 0; <math>\mathcal{D}^0</math> is the set of negative examples, and <math>\mathcal{D}^1</math> is the set of positive examples.

In the context of [[credit scoring]], a rescaled version of AUC is often used:

<math>G_1 = 2 \operatorname{AUC} - 1</math>.

<math>G_1</math> is referred to as Gini index or Gini coefficient,<ref>Hand, David J.; and Till, Robert J. (2001); ''A simple generalization of the area under the ROC curve for multiple class classification problems'', Machine Learning, 45, 171–186.</ref> but it should not be confused with the [[Gini coefficient|measure of statistical dispersion that is also called Gini coefficient]]. <math>G_1</math> is a special case of [[Somers' D]].

It is also common to calculate the Area Under the ROC Convex Hull (ROC AUCH = ROCH AUC) as any point on the line segment between two prediction results can be achieved by randomly using one or the other system with probabilities proportional to the relative length of the opposite component of the segment.<ref>{{cite journal |first1=F. |last1=Provost |first2=T. |last2=Fawcett |title=Robust classification for imprecise environments. |journal=Machine Learning |volume=42 |issue=3 |pages=203–231 |year=2001 |doi=10.1023/a:1007601015854 |arxiv=cs/0009007 |s2cid=5415722 }}</ref> It is also possible to invert concavities – just as in the figure the worse solution can be reflected to become a better solution; concavities can be reflected in any line segment, but this more extreme form of fusion is much more likely to overfit the data.<ref name="FlachWu2005">{{cite conference |first1=P.A. |last1=Flach |first2=S. |last2=Wu |year=2005 |title= Repairing concavities in ROC curves. |book-title= 19th International Joint Conference on Artificial Intelligence (IJCAI'05) |pages= 702–707 |url=http://www.icml-2011.org/papers/385_icmlpaper.pdf }}</ref>

The [[machine learning]] community most often uses the ROC AUC statistic for model comparison.<ref>{{cite journal |issue=3 |pages=839–843 |last1=Hanley |first1=James A.| last2=McNeil |first2=Barbara J. |title=A method of comparing the areas under receiver operating characteristic curves derived from the same cases |journal=Radiology |date=1983-09-01 |volume=148 |pmid=6878708 |doi=10.1148/radiology.148.3.6878708|doi-access=free }}</ref> This practice has been questioned because AUC estimates are quite noisy and suffer from other problems.<ref name="Hanczar2010">{{cite journal | last1 = Hanczar | first1 = Blaise | last2 = Hua | first2 = Jianping | last3 = Sima | first3 = Chao | last4 = Weinstein | first4 = John | last5 = Bittner | first5 = Michael | last6 = Dougherty | first6 = Edward R | year = 2010 | title = Small-sample precision of ROC-related estimates | journal = Bioinformatics | volume = 26 | issue = 6| pages = 822–830 | doi=10.1093/bioinformatics/btq037| pmid = 20130029 | doi-access = free }}</ref><ref name="Lobo2008">{{cite journal | last1 = Lobo | first1 = Jorge M. | last2 = Jiménez-Valverde | first2 = Alberto | last3 = Real | first3 = Raimundo | s2cid = 15206363 | year = 2008 | title = AUC: a misleading measure of the performance of predictive distribution models | journal = Global Ecology and Biogeography | volume = 17 | issue = 2| pages = 145–151 | doi=10.1111/j.1466-8238.2007.00358.x| bibcode = 2008GloEB..17..145L }}</ref><ref name="Hand2009">{{cite journal | last1 = Hand | first1 = David J | year = 2009 | title = Measuring classifier performance: A coherent alternative to the area under the ROC curve | journal = Machine Learning | volume = 77 | pages = 103–123 | doi=10.1007/s10994-009-5119-5| doi-access = free | hdl = 10044/1/18420 | hdl-access = free }}</ref> Nonetheless, the coherence of AUC as a measure of aggregated classification performance has been vindicated, in terms of a uniform rate distribution,<ref name="Flachetal2011">{{cite conference |first1=P.A. |last1=Flach |first2=J. |last2=Hernandez-Orallo | first3=C. | last3=Ferri |year=2011 |title=A coherent interpretation of AUC as a measure of aggregated classification performance. |book-title=Proceedings of the 28th International Conference on Machine Learning (ICML-11) |pages=657–664|url=http://www.icml-2011.org/papers/385_icmlpaper.pdf}}</ref> and AUC has been linked to a number of other performance metrics such as the [[Brier score]].<ref name="hernandez2012unified ">{{cite journal |first1=J. |last1= Hernandez-Orallo| first2=P.A.| last2=Flach | first3=C. | last3=Ferri |year=2012 |title=A unified view of performance metrics: translating threshold choice into expected classification loss|journal=Journal of Machine Learning Research|volume=13 |pages=2813–2869 |url=http://jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf}}</ref>

Another problem with ROC AUC is that reducing the ROC Curve to a single number ignores the fact that it is about the tradeoffs between the different systems or performance points plotted and not the performance of an individual system, as well as ignoring the possibility of concavity repair, so that related alternative measures such as Informedness{{citation needed|date=November 2019}} or DeltaP are recommended.<ref name="Powers2012a"/><ref name="Powers2012b">{{cite conference |first=David M.W. |last=Powers |title=The Problem of Area Under the Curve |book-title=International Conference on Information Science and Technology |year=2012}}</ref> These measures are essentially equivalent to the Gini for a single prediction point with DeltaP' = Informedness = 2AUC-1, whilst DeltaP = Markedness represents the dual (viz. predicting the prediction from the real class) and their geometric mean is the [[Matthews correlation coefficient]].{{citation needed|date=November 2019}}

Whereas ROC AUC varies between 0 and 1 — with an uninformative classifier yielding 0.5 — the alternative measures known as [[Informedness]],{{citation needed|date=November 2019}} Certainty <ref name="Powers2012a"/> and Gini Coefficient (in the single parameterization or single system case){{citation needed|date=November 2019}} all have the advantage that 0 represents chance performance whilst 1 represents perfect performance, and −1 represents the "perverse" case of full informedness always giving the wrong response.<ref>{{cite conference|first=David M. W. |last=Powers |year=2003 |title= Recall and Precision versus the Bookmaker |book-title=Proceedings of the International Conference on Cognitive Science (ICSC-2003), Sydney Australia, 2003, pp.&nbsp;529–534. | url=http://dl.dropbox.com/u/27743223/200302-ICCS-Bookmaker.pdf }}</ref> Bringing chance performance to 0 allows these alternative scales to be interpreted as Kappa statistics. Informedness has been shown to have desirable characteristics for Machine Learning versus other common definitions of Kappa such as [[Cohen's kappa|Cohen Kappa]] and [[Fleiss' kappa|Fleiss Kappa]].{{citation needed|date=November 2019}}<ref>{{cite conference |first=David M. W. |last=Powers |year=2012 |url=http://dl.dropbox.com/u/27743223/201209-eacl2012-Kappa.pdf |title=The Problem with Kappa |book-title=Conference of the European Chapter of the Association for Computational Linguistics (EACL2012) Joint ROBUS-UNSUP Workshop |access-date=2012-07-20 |archive-url=http://arquivo.pt/wayback/20160518183306/http://dl.dropbox.com/u/27743223/201209-eacl2012-Kappa.pdf |archive-date=2016-05-18 |url-status=dead }}</ref>

Sometimes it can be more useful to look at a specific region of the ROC Curve rather than at the whole curve. It is possible to compute [[Partial Area Under the ROC Curve (pAUC)|partial AUC]].<ref>{{cite journal |doi=10.1177/0272989X8900900307 |volume=9 |issue=3 |pages=190–195 |last=McClish |first=Donna Katzman |s2cid=24442201 |title=Analyzing a Portion of the ROC Curve |journal=Medical Decision Making |date=1989-08-01 |pmid=2668680 }}</ref> For example, one could focus on the region of the curve with low false positive rate, which is often of prime interest for population screening tests.<ref>{{cite journal |doi=10.1111/1541-0420.00071 |volume=59 |issue=3 |pages=614–623 |last1=Dodd |first1=Lori E. | first2=Margaret S. |last2=Pepe |title=Partial AUC Estimation and Regression |journal=Biometrics |year=2003 |pmid=14601762 |s2cid=23054670 |url=http://biostats.bepress.com/cgi/viewcontent.cgi?article=1005&context=uwbiostat |doi-access=free }}</ref> Another common approach for classification problems in which P ≪ N (common in bioinformatics applications) is to use a logarithmic scale for the ''x''-axis.<ref>Karplus, Kevin (2011); [http://www.soe.ucsc.edu/~karplus/papers/better-than-chance-sep-07.pdf ''Better than Chance: the importance of null models''], University of California, Santa Cruz, in Proceedings of the First International Workshop on Pattern Recognition in Proteomics, Structural Biology and Bioinformatics (PR PS BB 2011)</ref>

The ROC area under the curve is also called '''c-statistic''' or '''c statistic'''.<ref>{{Cite web|url=https://www.statisticshowto.datasciencecentral.com/c-statistic/|title=C-Statistic: Definition, Examples, Weighting and Significance|date=August 28, 2016|website=Statistics How To}}</ref>

===Other measures===
[[File:TOC.png|thumb|right|TOC Curve]]
The [[Total Operating Characteristic]] (TOC) also characterizes diagnostic ability while revealing more information than the ROC. For each threshold, ROC reveals two ratios, TP/(TP + FN) and FP/(FP + TN). In other words, ROC reveals <math>\frac{\text{hits}}{\text{hits}+\text{misses}}</math> and <math>\frac{\text{false alarms}}{\text{false alarms} + \text{correct rejections}}</math>. On the other hand, TOC shows the total information in the contingency table for each threshold.<ref>{{cite journal|last1=Pontius|first1=Robert Gilmore|last2=Parmentier|first2=Benoit|s2cid=15924380|title=Recommendations for using the Relative Operating Characteristic (ROC)|journal=Landscape Ecology|date=2014|volume=29|issue=3|pages=367–382|doi=10.1007/s10980-013-9984-8|bibcode=2014LaEco..29..367P }}</ref> The TOC method reveals all of the information that the ROC method provides, plus additional important information that ROC does not reveal, i.e. the size of every entry in the contingency table for each threshold. TOC also provides the popular AUC of the ROC.<ref>{{cite journal|last1=Pontius|first1=Robert Gilmore|last2=Si|first2=Kangping|s2cid=29204880|title=The total operating characteristic to measure diagnostic ability for multiple thresholds|journal=International Journal of Geographical Information Science|date=2014|volume=28|issue=3|pages=570–583|doi=10.1080/13658816.2013.862623|bibcode=2014IJGIS..28..570P }}</ref>
[[File:ROC.png|thumb|right|ROC Curve]]
These figures are the TOC and ROC curves using the same data and thresholds.
Consider the point that corresponds to a threshold of 74. The TOC curve shows the number of hits, which is 3, and hence the number of misses, which is 7. Additionally, the TOC curve shows that the number of false alarms is 4 and the number of correct rejections is 16. At any given point in the ROC curve, it is possible to glean values for the ratios of <math>\frac{\text{false alarms}}{\text{false alarms} + \text{correct rejections}}</math> and <math>\frac{\text{hits}}{\text{hits}+\text{misses}}</math>. For example, at threshold 74, it is evident that the x coordinate is 0.2 and the y coordinate is 0.3. However, these two values are insufficient to construct all entries of the underlying two-by-two contingency table.
{{Clear}}

==Detection error tradeoff graph==
[[File:Example of DET curves.png|thumb|Example DET graph]]
An alternative to the ROC curve is the [[detection error tradeoff]] (DET) graph, which plots the false negative rate (missed detections) vs. the false positive rate (false alarms) on non-linearly transformed x- and y-axes. The transformation function is the quantile function of the normal distribution, i.e., the inverse of the cumulative normal distribution. It is, in fact, the same transformation as zROC, below, except that the complement of the hit rate, the miss rate or false negative rate, is used. This alternative spends more graph area on the region of interest. Most of the ROC area is of little interest; one primarily cares about the region tight against the ''y''-axis and the top left corner – which, because of using miss rate instead of its complement, the hit rate, is the lower left corner in a DET plot. Furthermore, DET graphs have the useful property of linearity and a linear threshold behavior for normal distributions.<ref>{{Cite book|last1=Navratil|first1=J.|last2=Klusacek|first2=D.|title=2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07 |chapter=On Linear DETs |s2cid=18173315|date=2007-04-01|volume=4|pages=IV–229–IV–232|doi=10.1109/ICASSP.2007.367205|isbn=978-1-4244-0727-9}}</ref> The DET plot is used extensively in the [[automatic speaker recognition]] community, where the name DET was first used. The analysis of the ROC performance in graphs with this warping of the axes was used by psychologists in perception studies halfway through the 20th century,{{Citation needed|date=July 2019}} where this was dubbed "double probability paper".<ref>{{cite book |title=Observer Performance Methods for Diagnostic Imaging: Foundations, Modeling, and Applications with R-Based Examples |author=Dev P. Chakraborty |publisher=CRC Press |date=December 14, 2017 |isbn=9781351230711 
 |page=214 |url=https://books.google.com/books?id=MwZDDwAAQBAJ&q="double+probability+paper"&pg=PT214 |access-date=July 11, 2019}}</ref>

==Z-score==

If a [[standard score]] is applied to the ROC curve, the curve will be transformed into a straight line.<ref>{{cite book |first1=Neil A. |last1=MacMillan |first2=C. Douglas |last2=Creelman |edition=2nd |year=2005 |title=Detection Theory: A User's Guide |publisher=Lawrence Erlbaum Associates |location=Mahwah, NJ | isbn=978-1-4106-1114-7 }}</ref>  This z-score is based on a normal distribution with a mean of zero and a standard deviation of one. In memory [[Recognition memory|strength theory]], one must assume that the zROC is not only linear, but has a slope of 1.0. The normal distributions of targets (studied objects that the subjects need to recall) and lures (non studied objects that the subjects attempt to recall) is the factor causing the zROC to be linear.

The linearity of the zROC curve depends on the standard deviations of the target and lure strength distributions. If the standard deviations are equal, the slope will be 1.0. If the standard deviation of the target strength distribution is larger than the standard deviation of the lure strength distribution, then the slope will be smaller than 1.0. In most studies, it has been found that the zROC curve slopes constantly fall below 1, usually between 0.5 and 0.9.<ref>{{cite journal |last1=Glanzer |first1=Murray |first2=Kim |last2=Kisok |last3=Hilford |first3=Andy |last4=Adams |first4=John K. |title=Slope of the receiver-operating characteristic in recognition memory |journal=Journal of Experimental Psychology: Learning, Memory, and Cognition |year=1999 |volume=25 |issue=2 |pages=500–513 |doi=10.1037/0278-7393.25.2.500}}</ref> Many experiments yielded a zROC slope of 0.8. A slope of 0.8 implies that the variability of the target strength distribution is 25% larger than the variability of the lure strength distribution.<ref>{{cite journal |last1=Ratcliff |first1=Roger |last2=McCoon |first2=Gail |last3=Tindall |first3=Michael |title=Empirical generality of data from recognition memory ROC functions and implications for GMMs |journal=Journal of Experimental Psychology: Learning, Memory, and Cognition |year=1994 |volume=20 |issue=4 |pages=763–785 |doi=10.1037/0278-7393.20.4.763|pmid=8064246 |citeseerx=10.1.1.410.2114 }}</ref>

Another variable used is&nbsp;[[D prime|''d<nowiki>'</nowiki>'' (d prime)]] (discussed above in "Other measures"), which can easily be expressed in terms of z-values. Although ''d''<nowiki>'</nowiki>&nbsp;is a commonly used parameter, it must be recognized that it is only relevant when strictly adhering to the very strong assumptions of strength theory made above.<ref>{{cite journal |last1=Zhang |first1=Jun |last2=Mueller |first2=Shane T. |s2cid=122355230 |title=A note on ROC analysis and non-parametric estimate of sensitivity |journal=Psychometrika |year=2005 |volume=70 |doi=10.1007/s11336-003-1119-8 |pages=203–212|citeseerx=10.1.1.162.1515 }}</ref>

The z-score of an ROC curve is always linear, as assumed, except in special situations. The Yonelinas familiarity-recollection model is a two-dimensional account of recognition memory. Instead of the subject simply answering yes or no to a specific input, the subject gives the input a feeling of familiarity, which operates like the original ROC curve. What changes, though, is a parameter for Recollection (R). Recollection is assumed to be all-or-none, and it trumps familiarity. If there were no recollection component, zROC would have a predicted slope of 1. However, when adding the recollection component, the zROC curve will be concave up, with a decreased slope. This difference in shape and slope result from an added element of variability due to some items being recollected. Patients with anterograde amnesia are unable to recollect, so their Yonelinas zROC curve would have a slope close to 1.0.<ref>{{cite journal |last1=Yonelinas |first1=Andrew P. |last2=Kroll |first2=Neal E. A. |last3=Dobbins |first3=Ian G. |last4=Lazzara |first4=Michele |last5=Knight |first5=Robert T. |title=Recollection and familiarity deficits in amnesia: Convergence of remember-know, process dissociation, and receiver operating characteristic data |journal=Neuropsychology |year=1998 |volume=12 |issue=3 |pages=323–339 |doi=10.1037/0894-4105.12.3.323|pmid=9673991 }}</ref>

==History==
The ROC curve was first used during [[World War II]] for the analysis of [[radar|radar signals]] before it was employed in [[signal detection theory]].<ref name="green66">{{cite book |first1=David M. |last1=Green |first2=John A. |last2=Swets |title=Signal detection theory and psychophysics |publisher=John Wiley and Sons Inc. |year=1966 |location=New York, NY |isbn=978-0-471-32420-1 }}</ref> Following the [[attack on Pearl Harbor]] in 1941, the United States military began new research to increase the prediction of correctly detected Japanese aircraft from their radar signals. For these purposes they measured the ability of a radar receiver operator to make these important distinctions, which was called the Receiver Operating Characteristic.<ref name="roc etymology">{{cite web|title=Using the Receiver Operating Characteristic (ROC) curve to analyze a classification model: A final note of historical interest|url=http://www.math.utah.edu/~gamez/files/ROC-Curves.pdf|url-status=live|archive-url=https://web.archive.org/web/20200822005346/http://www.math.utah.edu/~gamez/files/ROC-Curves.pdf |archive-date=2020-08-22 |access-date=May 25, 2017|website=Department of Mathematics, University of Utah}}</ref>

In the 1950s, ROC curves were employed in [[psychophysics]] to assess human (and occasionally non-human animal) detection of weak signals.<ref name="green66" /> In [[medicine]], ROC analysis has been extensively used in the evaluation of [[diagnostic test]]s.<ref>{{cite journal |first1=Mark H. |last1=Zweig |first2=Gregory |last2=Campbell |journal=Clinical Chemistry |pmid=8472349 |title=Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine |volume=39 |issue=8 |year=1993 |pages=561–577 |doi=10.1093/clinchem/39.4.561 |url=http://www.clinchem.org/content/39/4/561.full.pdf |doi-access=free }}</ref><ref>{{cite book |first=Margaret S. |last=Pepe |title=The statistical evaluation of medical tests for classification and prediction |location=New York, NY |publisher=Oxford |year=2003 |isbn=978-0-19-856582-6 }}</ref> ROC curves are also used extensively in [[epidemiology]] and [[medical research]] and are frequently mentioned in conjunction with [[evidence-based medicine]]. In [[radiology]], ROC analysis is a common technique to evaluate new radiology techniques.<ref>{{cite journal |first=Nancy A. |last=Obuchowski|author-link=Nancy Obuchowski |title=Receiver operating characteristic curves and their use in radiology |pmid=14519861 |journal=Radiology |volume=229 |issue=1 |year=2003 |pages=3–8 |doi=10.1148/radiol.2291010898 }}</ref> In the social sciences, ROC analysis is often called the ROC Accuracy Ratio, a common technique for judging the accuracy of default probability models.
ROC curves are widely used in laboratory medicine to assess the diagnostic accuracy of a test, to choose the optimal cut-off of a test and to compare diagnostic accuracy of several tests.

ROC curves also proved useful for the evaluation of [[machine learning]] techniques. The first application of ROC in machine learning was by Spackman who demonstrated the value of ROC curves in comparing and evaluating different classification [[algorithm]]s.<ref>{{cite conference |last=Spackman |first=Kent A. |year=1989 |title=Signal detection theory: Valuable tools for evaluating inductive learning |book-title=Proceedings of the Sixth International Workshop on Machine Learning |location=San Mateo, CA |pages=160–163 |publisher=[[Morgan Kaufmann]] }}</ref>

ROC curves are also used in verification of forecasts in meteorology.<ref>{{ cite journal |first=Viatcheslav |last=Kharin| title=On the ROC score of probability forecasts| journal=Journal of Climate| volume=16|number=24|  pages=4145–4150|year=2003|doi=10.1175/1520-0442(2003)016<4145:OTRSOP>2.0.CO;2
|bibcode=2003JCli...16.4145K|doi-access=free}}</ref>

=== Radar in detail ===
As mentioned ROC curves are critical to [[radar]] operation and theory. The signals received at a receiver station, as reflected by a target, are often of very low energy, in comparison to the [[noise floor]]. The ratio of [[Signal-to-noise ratio|signal to noise]] is an important metric when determining if a target will be detected. This signal to noise ratio is directly correlated to the receiver operating characteristics of the whole radar system, which is used to quantify the ability of a radar system.

Consider the development of a radar system. A specification for the abilities of the system may be provided in terms of probability of detect, <math>P_{D}</math>, with a certain tolerance for false alarms, <math>P_{FA}</math>. A simplified approximation of the required signal to noise ratio at the receiver station can be calculated by solving<ref>{{Citation |title=Fundamentals of Radar |date=2008-01-29 |url=http://dx.doi.org/10.1002/9780470377765.ch4 |work=Digital Signal Processing Techniques and Applications in Radar Image Processing |pages=93–115 |access-date=2023-05-20 |place=Hoboken, NJ, USA |publisher=John Wiley & Sons, Inc.|doi=10.1002/9780470377765.ch4 |isbn=9780470377765 }}</ref>

: <math>P_{D}=\frac{1}{2}\operatorname{erfc}\left(\operatorname{erfc}^{-1}\left(2P_{FA}\right)-\sqrt{\mathcal{X}}\right)</math>

for the signal to noise ratio <math>\mathcal{X}</math>. Here, <math>\mathcal{X}</math> is not in [[decibel]]s, as is common in many radar applications. Conversion to decibels is through <math>\mathcal{X}_{dB}=10\log_{10}\mathcal{X}</math>. From this figure, the common entries in the radar range equation (with noise factors) may be solved, to estimate the required [[effective radiated power]].

==ROC curves beyond binary classification==
The extension of ROC curves for classification problems with more than two classes is cumbersome.  Two common approaches for when there are multiple classes are (1) average over all pairwise AUC values<ref name="HandTill01">{{cite journal |last1=Till |first1=D.J. |last2=Hand |first2=R.J. |year=2001 |title=A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems |journal=Machine Learning |volume=45 |issue=2 |pages=171–186 |doi=10.1023/A:1010920819831 |doi-access=free}}</ref> and (2) compute the volume under surface (VUS).<ref name="Mossman99">{{cite journal |last=Mossman |first=D. |year=1999 |title=Three-way ROCs |journal=Medical Decision Making |volume=19 |issue=1 |pages=78–89 |doi=10.1177/0272989x9901900110 |pmid=9917023 |s2cid=24623127}}</ref><ref name="Ferri03">{{cite conference |last1=Ferri |first1=C. |last2=Hernandez-Orallo |first2=J. |last3=Salido |first3=M.A. |year=2003 |title=Volume under the ROC Surface for Multi-class Problems |pages=108–120 |book-title=Machine Learning: ECML 2003}}</ref> To average over all pairwise classes, one computes the AUC for each pair of classes, using only the examples from those two classes as if there were no other classes, and then averages these AUC values over all possible pairs. When there are {{math|''c''}} classes there will be {{math|''c''(''c'' − 1) / 2}} possible pairs of classes.

The volume under surface approach has one plot a hypersurface rather than a curve and then measure the hypervolume under that hypersurface. Every possible decision rule that one might use for a classifier for {{math|''c''}} classes can be described in terms of its true positive rates {{math|(TPR{{sub|1}},&nbsp;.&nbsp;.&nbsp;.&nbsp;,&nbsp;TPR{{sub|''c''}})}}. It is this set of rates that defines a point, and the set of all possible decision rules yields a cloud of points that define the hypersurface.  With this definition, the VUS is the probability that the classifier will be able to correctly label all {{math|''c''}} examples when it is given a set that has one randomly selected example from each class.  The implementation of a classifier that knows that its input set consists of one example from each class might first compute a goodness-of-fit score for each of the {{math|''c''{{sup|2}}}} possible pairings of an example to a class, and then employ the [[Hungarian algorithm]] to maximize the sum of the {{math|''c''}} selected scores over all {{math|''c''!}} possible ways to assign exactly one example to each class.

Given the success of ROC curves for the assessment of classification models, the extension of ROC curves for other supervised tasks has also been investigated. Notable proposals for regression problems are the so-called regression error characteristic (REC) Curves <ref name="bij2003regression">{{cite conference |last1=Bi |first1=J. |last2=Bennett |first2=K.P. |year=2003 |title=Regression error characteristic curves |url=https://www.aaai.org/Papers/ICML/2003/ICML03-009.pdf |book-title=Twentieth International Conference on Machine Learning (ICML-2003). Washington, DC.}}</ref> and the Regression ROC (RROC) curves.<ref name="hernandez2013rroc">{{cite journal |last=Hernandez-Orallo |first=J. |year=2013 |title=ROC curves for regression |journal=Pattern Recognition |volume=46 |issue=12 |pages=3395–3411 |doi=10.1016/j.patcog.2013.06.014 |bibcode=2013PatRe..46.3395H |hdl-access=free |hdl=10251/40252|s2cid=15651724 }}</ref> In the latter, RROC curves become extremely similar to ROC curves for classification, with the notions of asymmetry, dominance and convex hull. Also, the area under RROC curves is proportional to the error variance of the regression model.

==See also==
{{Portal|Mathematics}}
{{div col}}
* [[Brier score]]
* [[Coefficient of determination]]
* [[Constant false alarm rate]]
* [[Detection error tradeoff]]
* [[Detection theory]]
* [[F1 score]]
* [[False alarm]]
* [[Evaluation of binary classifiers#Single metrics|Hypothesis tests for accuracy]]
* [[Precision and recall]]
* [[ROCCET]]
* [[Sensitivity and specificity]]
* [[Total operating characteristic]]
{{div col end}}
{{clear}}

==References==
{{Reflist|30em}}

== External links ==
* [http://biosoft.erciyes.edu.tr/app/easyROC/ easy ROC calculator]
* [https://www.youtube.com/watch?v=RlJotioVi_0 Animated ROC demo]
* [https://kennis-research.shinyapps.io/ROC-Curves/ ROC demo]
* [http://www.navan.name/roc/ another ROC demo ]
* [https://www.youtube.com/watch?v=OAl6eAyP-yo ROC video explanation]
* [https://www.youtube.com/watch?v=KKVC3GT5EPw An Introduction to the Total Operating Characteristic: Utility in Land Change Model Evaluation]
* [https://www.youtube.com/watch?v=1JRwVOi0FSE How to run the TOC Package in R]
* [https://github.com/amsantac/TOC TOC R package on Github]
* [http://www2.clarku.edu/~rpontius/TOCexample2.xlsx Excel Workbook for generating TOC curves]

==Further reading==
{{Commons category|Receiver operating characteristic}}
* Balakrishnan, Narayanaswamy (1991); ''Handbook of the Logistic Distribution'', Marcel Dekker, Inc., {{ISBN|978-0-8247-8587-1}}
* {{cite journal | last1 = Brown | first1 = Christopher D. | last2 = Davis | first2 = Herbert T. | year = 2006 | title = Receiver operating characteristic curves and related decision measures: a tutorial | journal = Chemometrics and Intelligent Laboratory Systems | volume = 80 | pages = 24–38 | doi=10.1016/j.chemolab.2005.05.004}}
* {{cite journal | last1 = Rotello | first1 = Caren M. | last2 = Heit | first2 = Evan | last3 = Dubé | first3 = Chad | s2cid = 6046065 | year = 2014 | title = When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions | url = http://faculty.ucmerced.edu/sites/default/files/eheit/files/rotello_heit_dube_pbr.pdf | journal = Psychonomic Bulletin & Review | doi=10.3758/s13423-014-0759-2 | pmid = 25384892 | volume=22 | issue = 4 | pages=944–954| doi-access = free }}
* {{cite journal | last1 = Fawcett | first1 = Tom | year = 2004 | title = ROC Graphs: Notes and Practical Considerations for Researchers | url = http://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdf| journal = Pattern Recognition Letters | volume = 27 | issue = 8| pages = 882–891 | doi = 10.1016/j.patrec.2005.10.012 | citeseerx = 10.1.1.145.4649 }}
* Gonen, Mithat (2007); ''Analyzing Receiver Operating Characteristic Curves Using SAS'', SAS Press, {{ISBN|978-1-59994-298-8}}
* Green, William H., (2003) ''Econometric Analysis'', fifth edition, [[Prentice Hall]], {{ISBN|0-13-066189-9}}
* {{cite journal | last1 = Heagerty | first1 = Patrick J. | last2 = Lumley | first2 = Thomas | last3 = Pepe | first3 = Margaret S. | s2cid = 8822160 | year = 2000 | title = Time-dependent ROC Curves for Censored Survival Data and a Diagnostic Marker | journal = Biometrics | volume = 56 | issue = 2| pages = 337–344 | doi=10.1111/j.0006-341x.2000.00337.x| pmid = 10877287 }}
* Hosmer, David W.; and Lemeshow, Stanley (2000); ''Applied Logistic Regression'', 2nd ed., New York, NY: [[John Wiley & Sons|Wiley]], {{ISBN|0-471-35632-8}}
* {{cite journal | last1 = Lasko | first1 = Thomas A. | last2 = Bhagwat | first2 = Jui G. | last3 = Zou | first3 = Kelly H. | last4 = Ohno-Machado | first4 = Lucila | year = 2005 | title = The use of receiver operating characteristic curves in biomedical informatics | journal = Journal of Biomedical Informatics | volume = 38 | issue = 5| pages = 404–415 | doi=10.1016/j.jbi.2005.02.008| pmid = 16198999 | citeseerx = 10.1.1.97.9674 }}
* {{cite journal | last1 = Mas| first1 = Jean-François |last2 = Filho | first2 = Britaldo Soares |last3 = Pontius | first3 = Jr, Robert Gilmore | last4 = Gutiérrez| first4 = Michelle Farfán| last5 = Rodrigues | first5 = Hermann |year = 2013 | title = A suite of tools for ROC analysis of spatial models| journal = ISPRS International Journal of Geo-Information | volume = 2 | issue = 3| pages = 869–887| doi = 10.3390/ijgi2030869 | bibcode = 2013IJGI....2..869M | doi-access = free }}
* {{cite journal | last1 = Pontius | first1 = Jr, Robert Gilmore | last2 = Parmentier| first2 = Benoit| s2cid = 15924380 | year = 2014 | title = Recommendations for using the Relative Operating Characteristic (ROC)| url = https://www.researchgate.net/publication/263195341 | journal = Landscape Ecology | volume = 29 | issue = 3| pages = 367–382| doi = 10.1007/s10980-013-9984-8 | bibcode = 2014LaEco..29..367P }}
* {{cite journal | last1 = Pontius | first1 = Jr, Robert Gilmore | last2 = Pacheco| first2 = Pablo| s2cid = 155073463 | year = 2004 | title = Calibration and validation of a model of forest disturbance in the Western Ghats, India 1920–1990| url = https://www.researchgate.net/publication/226020087| journal = GeoJournal | volume = 61 | issue = 4| pages = 325–334| doi = 10.1007/s10708-004-5049-5 | bibcode = 2004GeoJo..61..325G }}
* {{cite journal | last1 = Pontius | first1 = Jr, Robert Gilmore | last2 = Batchu| first2 = Kiran| s2cid = 14452746 | year = 2003 | title = Using the relative operating characteristic to quantify certainty in prediction of location of land cover change in India| journal = Transactions in GIS| volume = 7 | issue = 4| pages = 467–484| doi = 10.1111/1467-9671.00159 | bibcode = 2003TrGIS...7..467P }}
* {{cite journal | last1 = Pontius | first1 = Jr, Robert Gilmore | last2 = Schneider| first2 = Laura| year = 2001 | title = Land-use change model validation by a ROC method for the Ipswich watershed, Massachusetts, USA| url = https://www.academia.edu/19474075| journal = Agriculture, Ecosystems & Environment| volume = 85 | issue = 1–3| pages = 239–248| doi = 10.1016/S0167-8809(01)00187-6 }}
* {{cite journal | last1 = Stephan | first1 = Carsten | last2 = Wesseling | first2 = Sebastian | last3 = Schink | first3 = Tania | last4 = Jung | first4 = Klaus | year = 2003 | title = Comparison of Eight Computer Programs for Receiver-Operating Characteristic Analysis | journal = Clinical Chemistry | volume = 49 | issue = 3| pages = 433–439 | doi=10.1373/49.3.433| pmid = 12600955 | doi-access = free }}
* Swets, John A.; Dawes, Robyn M.; and Monahan, John (2000); ''Better Decisions through Science'', [[Scientific American]], October, pp.&nbsp;82–87
* {{cite journal | last1 = Zou | first1 = Kelly H. | last2 = O'Malley | first2 = A. James | last3 = Mauri | first3 = Laura | year = 2007 | title = Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models | journal = Circulation | volume = 115 | issue = 5| pages = 654–7 | doi=10.1161/circulationaha.105.594929| pmid = 17283280 | doi-access = free }}
* {{cite book |first1=Xiao-Hua |last1=Zhou |first2=Nancy A. |last2=Obuchowski|author2-link=Nancy Obuchowski |first3=Donna K. |last3=McClish |year=2002 |title=Statistical Methods in Diagnostic Medicine |publisher=Wiley & Sons |location=New York, NY| isbn=978-0-471-34772-9 }}
* {{cite journal |author = Chicco D. |author2 = Jurman G. | title = The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification | journal = BioData Mining | date = 2023 | volume = 16 | issue = 1 |page = 4 | doi = 10.1186/s13040-023-00322-4  |pmid = 36800973 | pmc = 9938573 |doi-access = free}}

{{Statistics|state=expanded}}
{{Machine learning evaluation metrics}}
{{Public health}}
{{Authority control}}

{{DEFAULTSORT:Receiver Operating Characteristic}}
[[Category:Detection theory]]
[[Category:Data mining]]
[[Category:Biostatistics]]
[[Category:Statistical classification]]
[[Category:Summary statistics for contingency tables]]