View source for Total operating characteristic

{{Short description|Statistical evaluation method}}{{Promotional|date=November 2024}}

The '''total operating characteristic''' ('''TOC''') is a [[statistical method]] to compare a [[Boolean variable]] versus a [[Rank statistics|rank variable]]. TOC can measure the ability of an index variable to diagnose either presence or absence of a characteristic. The diagnosis of presence or absence depends on whether the value of the index is above a threshold. TOC considers multiple possible thresholds. Each threshold generates a two-by-two [[contingency table]], which contains four entries: hits, misses, false alarms, and correct rejections.<ref name="Si">{{cite journal|last1=Pontius|first1=Robert Gilmore|last2=Si|first2=Kangping|title=The total operating characteristic to measure diagnostic ability for multiple thresholds| journal=International Journal of Geographical Information Science|volume=28|issue=3|year=2014|pages=570–583|doi=10.1080/13658816.2013.862623|bibcode=2014IJGIS..28..570P |s2cid=29204880 }}</ref>

The [[receiver operating characteristic]] (ROC) also characterizes diagnostic ability, although ROC reveals less information than the TOC. For each threshold, ROC reveals two ratios, hits/(hits + misses) and false alarms/(false alarms + correct rejections), while TOC shows the total information in the contingency table for each threshold.<ref>{{cite journal|last1=Pontius|first1=Robert Gilmore|last2=Parmentier|first2=Benoit|title=Recommendations for using the Relative Operating Characteristic (ROC)|journal=Landscape Ecology|date=2014|volume=29 |issue=3 |pages=367–382 |doi=10.1007/s10980-013-9984-8 |bibcode=2014LaEco..29..367P |s2cid=254740981 }}</ref> The TOC method reveals all of the information that the ROC method provides, plus additional important information that ROC does not reveal, i.e. the size of every entry in the contingency table for each threshold. TOC also provides the popular [[Receiver operating characteristic|area under the curve]] (AUC) of the ROC.

TOC is applicable to measure diagnostic ability in many fields including but not limited to: land change science, [[medical imaging]], [[weather forecasting]], [[remote sensing]], and [[materials testing]].

__TOC__

==Basic concept==
{{See also|Receiver operating characteristic}}
The procedure to construct the TOC curve compares the Boolean variable to the index variable by diagnosing each observation as either presence or absence, depending on how the index relates to various thresholds. If an observation's index is greater than or equal to a threshold, then the observation is diagnosed as presence, otherwise the observation is diagnosed as absence. The contingency table that results from the comparison between the Boolean variable and the diagnosis for a single threshold has four central entries. The four central entries are hits (''H''), misses (''M''), false alarms (''F''), and correct rejections (''C''). The total number of observations is&nbsp;''P''&nbsp;+&nbsp;''Q''. The terms “true positives”, “false negatives”, “false positives” and “true negatives” are equivalent to hits, misses, false alarms and correct rejections, respectively. The entries can be formulated in a two-by-two contingency table or confusion matrix, as follows:

{| class="wikitable" style="background:transparent; border:none;"
! {{diagonal split header|<br />Boolean|Diagnosis}}
! Presence
! Absence
! style="background:transparent; border:none;"|Boolean total
|-
! Presence
| Hits (''H'')
| Misses (''M'')
| style="background:transparent; border:none;"|''H'' + ''M'' = ''P''
|-
! Absence
| False alarms (''F'')
| Correct rejections (''C'')
| style="background:transparent; border:none;"|''F'' + ''C'' = ''Q''
|-
! style="background:transparent; border:none;"|Diagnosis total
| style="background:transparent; border:none;"|''H'' + ''F''
| style="background:transparent; border:none;"|''M'' + ''C''
| style="background:transparent; border:none;"|''P'' + ''Q''
|}

{{-}}

Four bits of information determine all the entries in the contingency table, including its marginal totals. For example, if we know ''H'', ''M'', ''F'', and ''C'', then we can compute all the marginal totals for any threshold. Alternatively, if we know ''H''/''P'', ''F''/''Q'', ''P'', and ''Q'', then we can compute all the entries in the table.<ref name= "Si"/> Two bits of information are not sufficient to complete the contingency table. For example, if we know only ''H''/''P'' and ''F''/''Q'', which is what ROC shows, then it is impossible to know all the entries in the table.<ref name="Si" />

==TOC space==
[[File:TOC labeled.png|thumb|left|TOC labeled]]
The TOC curve with four boxes indicates how a point on the TOC curve reveals the hits, misses, false alarms, and correct rejections. The TOC curve is an effective way to show the total information in the contingency table for all thresholds. The data used to create this TOC curve is available for download [http://www2.clarku.edu/~rpontius/TOCexample2.xlsx here]. This dataset has 30 observations, each of which consists of values for a Boolean variable and an index variable. The observations are ranked from the greatest to the least value of the index. There are 31 thresholds, consisting of the 30 values of the index and one additional threshold that is greater than all the index values, which creates the point at the origin (0,0). Each point is labeled to indicate the value of each threshold. The horizontal axes ranges from 0 to 30 which is the number of observations in the dataset (''P''&nbsp;+&nbsp;''Q''). The vertical axis ranges from 0 to 10, which is the Boolean variable's number of presence observations ''P'' (i.e. hits&nbsp;+&nbsp;misses). TOC curves also show the threshold at which the diagnosed amount of presence matches the Boolean amount of presence, which is the threshold point that lies directly under the point where the maximum line meets the hits&nbsp;+&nbsp;misses line, as the TOC curve on the left illustrates.<ref name="Si" />

The following four pieces of information are the central entries in the contingency table for each threshold:

# The number of hits at each threshold is the distance between the threshold's point and the horizontal axis.
# The number of misses at each threshold is the distance between the threshold's point and the hits&nbsp;+&nbsp;misses horizontal line across the top of the graph.
# The number of false alarms at each threshold is the distance between threshold's point and the blue dashed maximum line that bounds the left side of the TOC space.
# The number of correct rejections at each threshold is the distance between the threshold's point and the purple dashed minimum line that bounds the right side of the TOC space.

==TOC vs. ROC curves==
[[File:TOC.png|thumb|left|TOC curve]]
[[File:ROC.png|thumb|left|ROC curve]]
These figures are the TOC and ROC curves using the same data and thresholds. Consider the point that corresponds to a threshold of 74. The TOC curve shows the number of hits, which is 3, and hence the number of misses, which is 7. Additionally, the TOC curve shows that the number of false alarms is 4 and the number of correct rejections is 16. At any given point in the ROC curve, it is possible to glean values for the ratios of false alarms/(false alarms+correct rejections) and hits/(hits+misses). For example, at threshold 74, it is evident that the x coordinate is 0.2 and the y coordinate is 0.3. However, these two values are insufficient to construct all entries of the underlying two-by-two contingency table.

{{-}}

==Interpreting TOC curves==

It is common to report the area under the curve (AUC) to summarize a TOC or ROC curve. However, condensing diagnostic ability into a single number fails to appreciate the shape of the curve. The following three TOC curves are TOC curves that have an AUC of 0.75 but have different shapes.{{cn|date=August 2023}}

[[File:TOC1.png|thumb|left|TOC curve with higher accuracy at high thresholds.]]
This TOC curve on the left exemplifies an instance in which the index variable has a high diagnostic ability at high thresholds near the origin, but random diagnostic ability at low thresholds near the upper right of the curve. The curve shows accurate diagnosis of presence until the curve reaches a threshold of 86. The curve then levels off and predicts around the random line.{{cn|date=August 2023}}

{{-}}

[[File:TOC2.png|thumb|left|TOC curve with medium accuracy at all thresholds.]]
This TOC curve exemplifies an instance in which the index variable has a medium diagnostic ability at all thresholds. The curve is consistently above the random line.

{{-}}

[[File:TOC3.png|thumb|left|TOC curve with higher accuracy at lower thresholds.]]
This TOC curve exemplifies an instance in which the index variable has random diagnostic ability at high thresholds and high diagnostic ability at low thresholds. The curve follows the random line at the highest thresholds near the origin, then the index variable diagnoses absence correctly as thresholds decrease near the upper right corner.

{{-}}

==Area under the curve==

When measuring diagnostic ability, a commonly reported measure is the area under the curve (AUC). The AUC is calculable from the TOC and the ROC. The value of the AUC is consistent for the same data whether you are calculating the area under the curve for a TOC curve or a ROC curve. The AUC indicates the probability that the diagnosis ranks a randomly chosen observation of Boolean presence higher than a randomly chosen observation of Boolean absence.<ref name="Halligan2015" />
The AUC is appealing to many researchers because AUC summarizes diagnostic ability in a single number, however, the AUC has come under critique as a potentially misleading measure, especially for spatially explicit analyses.<ref name="Halligan2015" /><ref name="Powers2012" />
Some features of the AUC that draw criticism include the fact that 1) AUC ignores the thresholds; 2) AUC summarizes the test performance over regions of the TOC or ROC space in which one would rarely operate; 3) AUC weighs omission and commission errors equally; 4) AUC does not give information about the spatial distribution of model errors; and, 5) the selection of spatial extent highly influences the rate of accurately diagnosed absences and the AUC scores.<ref name="Lobo2007" />
However, most of those criticisms apply to many other metrics.

When using normalized units, the area under the curve (often referred to as simply the AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative').<ref>Fawcett, Tom (2006); ''[https://www.math.ucdavis.edu/~saito/data/roc/fawcett-roc.pdf An introduction to ROC analysis]'', Pattern Recognition Letters, 27, 861–874.</ref> This can be seen as follows: the area under the curve is given by (the integral boundaries are reversed as large T has a lower value on the x-axis)
:<math>TPR(T): T \rightarrow y(x)</math>
:<math>FPR(T): T \rightarrow x</math>
:<math> A =  \int_{x=0}^{1} \mbox{TPR}(\mbox{FPR}^{-1}(x)) \, dx = \int_{\infty}^{-\infty} \mbox{TPR}(T) \mbox{FPR}'(T) \, dT = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} I(T'>T)f_1(T') f_0(T) \, dT' \, dT = P(X_1 > X_0) </math>
where <math> X_1 </math> is the score for a positive instance and <math> X_0 </math> is the score for a negative instance, and <math>f_0</math> and <math>f_1</math> are probability densities as defined in previous section.

It can further be shown that the AUC is closely related to the [[Mann–Whitney U]],<ref name="Hanley">{{cite journal |last1=Hanley |first1=James A. |last2=McNeil |first2=Barbara J. |author-link2=Barbara Joyce McNeil |journal=Radiology |number=1 |pages=29–36 |title=The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve |volume=143 |year=1982 |pmid=7063747 |doi=10.1148/radiology.143.1.7063747|s2cid=10511727 }}</ref><ref name="Mason">{{cite journal |last1=Mason |first1=Simon J. |last2=Graham |first2=Nicholas E. |year=2002 |title=Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation |journal=Quarterly Journal of the Royal Meteorological Society |volume=128 |issue=584 |pages=2145–2166 |url=http://www.inmet.gov.br/documentos/cursoI_INMET_IRI/Climate_Information_Course/References/Mason+Graham_2002.pdf |archive-url=https://web.archive.org/web/20081120134338/http://www.inmet.gov.br/documentos/cursoI_INMET_IRI/Climate_Information_Course/References/Mason%2BGraham_2002.pdf |url-status=dead |archive-date=2008-11-20 |doi=10.1256/003590002320603584 |citeseerx=10.1.1.458.8392 |bibcode=2002QJRMS.128.2145M |s2cid=121841664 }}</ref> which tests whether positives are ranked higher than negatives. It is also equivalent to the [[Wilcoxon signed-rank test|Wilcoxon test of ranks]].<ref name="Mason" /> The AUC is related to the [[Gini coefficient]] (<math>G_1</math>) by the formula <math>G_1 = 2 \mbox{AUC} - 1</math>, where:

:<math>G_1 = 1 - \sum_{k=1}^n (X_{k} - X_{k-1}) (Y_k + Y_{k-1})</math><ref>Hand, David J.; and Till, Robert J. (2001); ''A simple generalization of the area under the ROC curve for multiple class classification problems'', Machine Learning, 45, 171–186.</ref>

In this way, it is possible to calculate the AUC by using an average of a number of trapezoidal approximations.

It is also common to calculate the area under the TOC convex hull (ROC AUCH = ROCH AUC) as any point on the line segment between two prediction results can be achieved by randomly using one or the other system with probabilities proportional to the relative length of the opposite component of the segment.<ref>{{cite journal |first1=F. |last1=Provost |first2=T. |last2=Fawcett |title=Robust classification for imprecise environments. |journal=Machine Learning |volume=42 |issue=3 |pages=203–231 |year=2001 |doi=10.1023/a:1007601015854 |arxiv=cs/0009007 |s2cid=5415722 }}</ref> It is also possible to invert concavities – just as in the figure the worse solution can be reflected to become a better solution; concavities can be reflected in any line segment, but this more extreme form of fusion is much more likely to overfit the data.<ref name="FlachWu2005">{{cite conference |first1=P.A. |last1=Flach |first2=S. |last2=Wu |year=2005 |title= Repairing concavities in ROC curves. |book-title= 19th International Joint Conference on Artificial Intelligence (IJCAI'05) |pages= 702–707 |url=http://www.icml-2011.org/papers/385_icmlpaper.pdf }}</ref>

Another problem with TOC AUC is that reducing the TOC Curve to a single number ignores the fact that it is about the tradeoffs between the different systems or performance points plotted and not the performance of an individual system, as well as ignoring the possibility of concavity repair, so that related alternative measures such as informedness{{cn|date=November 2019}} or DeltaP are recommended.<ref name="Powers2012a">{{cite conference |title=ROC-ConCert: ROC-Based Measurement of Consistency and Certainty|first=David MW |last=Powers|year=2012|book-title=Spring Congress on Engineering and Technology (SCET)|volume=2|pages=238–241|publisher=IEEE|url=http://www.academia.edu/download/31939951/201203-SCET30795-ROC-ConCert-PID1124774.pdf}}{{dead link|date=July 2022|bot=medic}}{{cbignore|bot=medic}}</ref><ref name="Powers2012b">{{cite conference |first=David M.W. |last=Powers |title=The Problem of Area Under the Curve |book-title=International Conference on Information Science and Technology |year=2012}}</ref> These measures are essentially equivalent to the Gini for a single prediction point with DeltaP' = informedness = 2AUC-1, whilst DeltaP = markedness represents the dual (viz. predicting the prediction from the real class) and their geometric mean is the [[Matthews correlation coefficient]].{{cn|date=November 2019}}

Whereas TOC AUC varies between 0 and 1 — with an uninformative classifier yielding 0.5 — the alternative measures known as [[informedness]],{{cn|date=November 2019}} Certainty <ref name="Powers2012a"/> and [[Gini coefficient]] (in the single parameterization or single system case){{cn|date=November 2019}} all have the advantage that 0 represents chance performance whilst 1 represents perfect performance, and −1 represents the "perverse" case of full informedness always giving the wrong response.<ref>{{cite conference |first=David M. W. |last=Powers |year=2003 |title=Recall and Precision versus the Bookmaker |book-title=Proceedings of the International Conference on Cognitive Science (ICSC-2003), Sydney Australia, 2003, pp.&nbsp;529–534. |url=http://dl.dropbox.com/u/27743223/200302-ICCS-Bookmaker.pdf }}{{Dead link|date=October 2023 |bot=InternetArchiveBot |fix-attempted=yes }}</ref> Bringing chance performance to 0 allows these alternative scales to be interpreted as Kappa statistics. Informedness has been shown to have desirable characteristics for machine learning versus other common definitions of Kappa such as [[Cohen's kappa|Cohen kappa]] and [[Fleiss' kappa|Fleiss kappa]].{{cn|date=November 2019}}<ref>{{cite conference |first=David M. W. |last=Powers |year=2012 |url=http://dl.dropbox.com/u/27743223/201209-eacl2012-Kappa.pdf |title=The Problem with Kappa |book-title=Conference of the European Chapter of the Association for Computational Linguistics (EACL2012) Joint ROBUS-UNSUP Workshop |access-date=2012-07-20 |archive-url=http://arquivo.pt/wayback/20160518183306/http://dl.dropbox.com/u/27743223/201209-eacl2012-Kappa.pdf |archive-date=2016-05-18 |url-status=dead }}</ref>

Sometimes it can be more useful to look at a specific region of the TOC curve rather than at the whole curve. It is possible to compute partial AUC.<ref>{{cite journal |doi=10.1177/0272989X8900900307 |volume=9 |issue=3 |pages=190–195 |last=McClish |first=Donna Katzman |title=Analyzing a Portion of the ROC Curve |journal=Medical Decision Making |date=1989-08-01 |pmid=2668680 |s2cid=24442201 }}</ref> For example, one could focus on the region of the curve with low false positive rate, which is often of prime interest for population screening tests.<ref>{{cite journal |doi=10.1111/1541-0420.00071 |volume=59 |issue=3 |pages=614–623 |last1=Dodd |first1=Lori E. | first2=Margaret S. |last2=Pepe |title=Partial AUC Estimation and Regression |journal=Biometrics |year=2003 |pmid=14601762 |s2cid=23054670 |url=http://biostats.bepress.com/cgi/viewcontent.cgi?article=1005&context=uwbiostat |doi-access=free }}</ref> Another common approach for classification problems in which P ≪ N (common in bioinformatics applications) is to use a logarithmic scale for the x-axis.<ref>Karplus, Kevin (2011); [http://www.soe.ucsc.edu/~karplus/papers/better-than-chance-sep-07.pdf ''Better than Chance: the importance of null models''], University of California, Santa Cruz, in Proceedings of the First International Workshop on Pattern Recognition in Proteomics, Structural Biology and Bioinformatics (PR PS BB 2011)</ref>

==See also==
* [[Brier score]]
* [[Coefficient of determination]]
* [[Constant false alarm rate]]
* [[Detection error tradeoff]]
* [[Detection theory]]
* [[F1 score]]
* [[False alarm]]
* [[Precision and recall]]
* [[ROCCET]]
* [[Receiver operating characteristic]]

==References==
{{reflist|refs=
<ref name="Halligan2015">{{cite journal |last1=Halligan |first1=Steve |last2=Altman |first2=Douglas G. |last3=Mallett |first3=Susan |title=Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach |journal=European Radiology |date=2015 |volume=25 |issue=4 |pages=932–939 |doi=10.1007/s00330-014-3487-0|pmc=4356897 |pmid=25599932}}</ref>
<ref name="Lobo2007">{{cite journal |last1=Lobo |first1=Jorge M. |last2=Jiménez-Valverde |first2=Alberto |last3=Real |first3=Raimundo |title=AUC: a misleading measure of the performance of predictive distribution models |journal=Global Ecology and Biogeography |date=2008 |volume=17 |issue=2 |pages=145–151 |doi=10.1111/j.1466-8238.2007.00358.x|bibcode=2008GloEB..17..145L }}</ref>
<ref name="Powers2012">{{cite book |last1=Powers |first1=David Martin Ward |title=2012 IEEE International Conference on Information Science and Technology |chapter=The problem of Area Under the Curve |date=2012 |pages=567–573 |doi=10.1109/ICIST.2012.6221710|isbn=978-1-4577-0345-4 |s2cid=11072457 }}</ref>
}}

{{Commons category|Receiver operating characteristic|position=left}}

==Further reading==
* {{cite journal | last1 = Pontius Jr | first1 = Robert Gilmore | last2 = Si | first2 = Kangping | year = 2014 | title = The total operating characteristic to measure diagnostic ability for multiple thresholds | journal = International Journal of Geographical Information Science | volume = 28 | issue = 3| pages = 570–583 | doi = 10.1080/13658816.2013.862623 | bibcode = 2014IJGIS..28..570P | s2cid = 29204880 }}
* {{cite journal | last1 = Pontius Jr | first1 = Robert Gilmore | last2 = Parmentier | first2 = Benoit | year = 2014 | title = Recommendations for using the Relative Operating Characteristic (ROC) | journal = Landscape Ecology | volume = 29 | issue = 3| pages = 367–382 | doi = 10.1007/s10980-013-9984-8 | bibcode = 2014LaEco..29..367P | s2cid = 254740981 }}
* {{cite journal | last1 = Mas | first1 = Jean-François | last2 = Filho | first2 = Britaldo Soares | last3 = Pontius Jr | first3 = Robert Gilmore | last4 = Gutiérrez | first4 = Michelle Farfán | last5 = Rodrigues | first5 = Hermann | year = 2013 | title = A suite of tools for ROC analysis of spatial models | journal = ISPRS International Journal of Geo-Information | volume = 2 | issue = 3| pages = 869–887 | doi = 10.3390/ijgi2030869 | bibcode = 2013IJGI....2..869M | doi-access = free }}
* {{cite journal | last1 = Pontius Jr | first1 = Robert Gilmore | last2 = Pacheco | first2 = Pablo | year = 2004 | title = Calibration and validation of a model of forest disturbance in the Western Ghats, India 1920–1990 | journal = GeoJournal | volume = 61 | issue = 4| pages = 325–334 | doi = 10.1007/s10708-004-5049-5 | bibcode = 2004GeoJo..61..325G | s2cid = 155073463 }}
* {{cite journal | last1 = Pontius Jr | first1 = Robert Gilmore | last2 = Batchu | first2 = Kiran | year = 2003 | title = Using the relative operating characteristic to quantify certainty in prediction of location of land cover change in India | journal = Transactions in GIS | volume = 7 | issue = 4| pages = 467–484 | doi = 10.1111/1467-9671.00159 | bibcode = 2003TrGIS...7..467P | s2cid = 14452746 }}
* {{cite journal | last1 = Pontius Jr | first1 = Robert Gilmore | last2 = Schneider | first2 = Laura | year = 2001 | title = Land-use change model validation by a ROC method for the Ipswich watershed, Massachusetts, USA | journal = Agriculture, Ecosystems & Environment | volume = 85 | issue = 1–3| pages = 239–248 | doi = 10.1016/s0167-8809(01)00187-6 }}


{{Portal|Mathematics}}

==External links==
* [https://www.youtube.com/watch?v=KKVC3GT5EPw An Introduction to the Total Operating Characteristic: Utility in Land Change Model Evaluation]
* [https://storymaps.arcgis.com/stories/243b2cf9bc194892badf8ab66bd39807 TOC utilization in Wildfire Risk]
* [https://www.youtube.com/watch?v=1JRwVOi0FSE How to run the TOC Package in R]
* [https://github.com/amsantac/TOC TOC R package on Github]
* [http://www2.clarku.edu/~rpontius/TOCexample2.xlsx Excel Workbook for generating TOC curves]
* [https://smithdavid529.users.earthengine.app/view/toc Google Earth Engine TOC Curve Tutorial]
* [https://code.earthengine.google.com/?scriptPath=users%2Fsmithdavid529%2FTOC_Curve_project%3ATOC_Curve_Explorer_11.29.21 Google Earth Engine TOC Curve Source Code]
{{Statistics|state=expanded}}
{{Public health}}

[[Category:Boolean algebra]]
[[Category:Detection theory]]
[[Category:Data mining]]
[[Category:Biostatistics]]
[[Category:Statistical classification]]
[[Category:Summary statistics for contingency tables]]