View source for Fleiss' kappa

{{Short description|Statistical measure}}
'''Fleiss' kappa''' (named after [[Joseph L. Fleiss]]) is a [[statistical measure]] for assessing the [[inter-rater reliability|reliability of agreement]] between a fixed number of [[rater]]s when assigning [[categorical rating]]s to a number of items or classifying items. This contrasts with other kappas such as [[Cohen's kappa]], which only work when assessing the agreement between not more than two raters or the intra-rater reliability (for one appraiser versus themself). The measure calculates the degree of agreement in classification over that which would be expected by chance.

Fleiss' kappa can be used with binary or [[Nominal data|nominal-scale]]. It can also be applied to [[ordinal data]] (ranked data): the MiniTab online documentation {{r|MiniTab}} gives an example. However, this document notes: "When you have ordinal ratings, such as defect severity ratings on a scale of 1–5, [[Kendall rank correlation coefficient|Kendall's coefficients]], which account for ordering, are usually more appropriate statistics to determine association than kappa alone." Keep in mind however, that Kendall rank coefficients are only appropriate for rank data.

==Introduction==
Fleiss' kappa is a generalisation of [[Scott's pi]] statistic,{{r|Scott1955}} a [[statistical]] measure of [[inter-rater reliability]].{{r|Fleiss1971}} It is also related to Cohen's kappa statistic and [[Youden's J statistic]] which may be more appropriate in certain instances.{{r|Powers2012}} Whereas Scott's pi and Cohen's kappa work for only two raters, Fleiss' kappa works for any number of raters giving categorical ratings, to a fixed number of items, at the condition that for each item raters are randomly sampled. It can be interpreted as expressing the extent to which the observed amount of agreement among raters exceeds what would be expected if all raters made their ratings completely randomly. It is important to note that whereas Cohen's kappa assumes the same two raters have rated a set of items, Fleiss' kappa specifically allows that although there are a fixed number of raters (e.g., three), different items may be rated by different individuals.{{r|Fleiss1971}} That is, Item 1 is rated by Raters A, B, and C; but Item 2 could be rated by Raters D, E, and F. The condition of random sampling among raters makes Fleiss' kappa not suited for cases where all raters rate all patients.{{r|Hallgren2012}}

Agreement can be thought of as follows, if a fixed number of people assign numerical ratings to a number of items then the kappa will give a measure for how consistent the ratings are. The kappa, <math>\kappa\,</math>, can be defined as,

<span style="float: right">(1)</span>
<math display="block"> \kappa = \frac{\bar{P} - \bar{P_e}}{1 - \bar{P_e}} </math>

The factor <math>1 - \bar{P_e}</math> gives the degree of agreement that is attainable above chance, and, <math>\bar{P} - \bar{P_e}</math> gives the degree of agreement actually achieved above chance. If the raters are in complete agreement then <math>\kappa = 1~</math>. If there is no agreement among the raters (other than what would be expected by chance) then <math>\kappa \le 0</math>.

An example of using Fleiss' kappa may be the following: consider several psychiatrists who are asked to look at ten patients. For each patient, 14 psychiatrists give one of possibly five diagnoses. These are compiled into a matrix, and Fleiss' kappa can be computed from this [[Matrix (mathematics)|matrix]] (see [[#Worked example|example below]]) to show the degree of agreement between the psychiatrists above the level of agreement expected by chance.

==Definition==
Let {{mvar|N}} be the total number of elements, let {{mvar|n}} be the number of ratings per element, and let {{mvar|k}} be the number of categories into which assignments are made. The elements are indexed by {{math|1=''i'' = 1, ..., ''N''}} and the categories are indexed by {{math|1=''j'' = 1, ..., ''k''}}. Let {{math|''n''<sub>''ij''</sub>}} represent the number of raters who assigned the {{mvar|i}}-th element to the {{mvar|j}}-th category.

First calculate {{math|''p''<sub>''j''</sub>}}, the proportion of all assignments which were to the {{mvar|j}}-th category:

<span style="float: right">(2)</span>
<math display="block"> p_{j} = \frac{1}{N n} \sum_{i=1}^N n_{i j},\quad\quad 1 = \sum_{j=1}^k p_{j} </math>

Now calculate <math>P_{i}\,</math>, the extent to which raters agree for the {{mvar|i}}-th element (i.e., compute how many rater-rater pairs are in agreement, relative to the number of all possible rater-rater pairs):

<span style="float: right">(3)</span>
{{bi|left=1.6|1=<math> \begin{align}
 P_i &= \frac{1}{n(n - 1)} \sum_{j=1}^k n_{i j} (n_{i j} - 1) \\
 &= \frac{1}{n(n - 1)} \sum_{j=1}^k (n_{i j}^2 - n_{i j}) \\
 &= \frac{1}{n(n - 1)} \biggl[ \sum_{j=1}^k \bigl(n_{i j}^2 \bigr) - n\biggr]
\end{align} </math>
}}

Note that <math>P_i</math> is bound between {{math|0}}, when ratings are assigned equally over all categories, and {{math|1}}, when all ratings are assigned to a single category.

Now compute <math>\bar{P}</math>, the mean of the <math>P_i</math>'s, and <math>\bar{P_e}</math>, which go into the formula for <math>\kappa</math>:

<span style="float: right">(4)</span>
<math display="block"> \begin{align}
 \bar{P} &= \frac{1}{N} \sum_{i=1}^N P_{i} \\ 
 &= \frac{1}{N n (n - 1)} \biggl[\sum_{i=1}^N \sum_{j=1}^k \bigl(n_{i j}^2\bigr) - N n\biggr]
\end{align} </math>

<span style="float: right">(5)</span>
<math display="block"> \bar{P_e} = \sum_{j=1}^k p_j^2</math>

==Worked example==
<div style="float: right">
{|class="wikitable"
|+ Table of values for computing the worked example
! <math>n_{ij}</math>  ||	1  ||	2  ||	3  ||	4  ||	5   ||  <math>P_i\,</math>
|-
|'''1''' ||	0  ||	0  ||	0  ||	0  ||	14   || 1.000
|-
|'''2''' ||	0  ||	2  ||	6  ||	4  ||	2   || 0.253
|-
|'''3''' ||	0  ||	0  ||	3  ||	5  ||	6   || 0.308
|-
|'''4''' ||	0  ||	3  ||	9  ||	2  ||	0   || 0.440
|-
|'''5''' || 	2  ||	2  ||	8  ||	1  ||	1   || 0.330
|-
|'''6''' ||	7  ||	7  ||	0  ||	0  ||	0   || 0.462
|-
|'''7''' ||	3  ||	2  ||	6  ||	3  ||	0   || 0.242
|-
|'''8''' ||	2  ||	5  ||	3  ||	2  ||	2   || 0.176
|-
|'''9''' ||	6  ||	5  ||	2  ||	1  ||	0   || 0.286
|-
|'''10''' ||	0  ||	2  ||	2  ||	3  ||	7   || 0.286
|-
|'''Total''' || 20 ||  28  ||  39  ||   21 ||   32
|-
| '''<math>p_j\,</math>''' || 0.143 ||0.200 ||0.279 || 0.150 || 0.229
|}</div>

In the following example, for each of ten "subjects" (<math>N</math>) fourteen raters (<math>n</math>), sampled from a larger group, assign a total of five categories (<math>k</math>). The categories are presented in the columns, while the subjects are presented in the rows. Each cell lists the number of raters who assigned the indicated (row) subject to the indicated (column) category.

In the following table, given that <math> N = 10 </math>, <math> n = 14 </math>, and <math> k = 5 </math>. The value <math> p_j </math> is the proportion of all assignments that were made to the <math>j</math>th category. For example, taking the first column
<math display="block"> p_1 = \frac{ 0+0+0+0+2+7+3+2+6+0 }{140} = 0.143,</math>
and taking the second row,
<math display="block"> P_2 = \frac{1}{14(14 - 1)} \left(0^2 + 2^2 + 6^2 + 4^2 + 2^2 - 14\right) = 0.253.</math>

In order to calculate <math>\bar{P}</math>, we need to know the sum of <math>P_i</math>,
<math display="block">\sum_{i=1}^N P_{i}= 1.000 + 0.253 + \cdots + 0.286 + 0.286 = 3.780.</math>

Over the whole sheet,
{{bi|left=1.6|1=<math> \begin{align}
 \bar{P} &= \frac{1}{(10)} (3.780) = 0.378 \\
 \bar{P}_{e} &= 0.143^2 + 0.200^2 + 0.279^2 + 0.150^2 + 0.229^2 = 0.213 \\
 \kappa &= \frac{0.378 - 0.213}{1 - 0.213} = 0.210
\end{align} </math>
}}

== Interpretation ==
{{harvtxt|Landis|Koch|1977}} gave the following table for interpreting  <math>\kappa</math> values for a 2-annotator 2-class example.{{r|Landis1977}} This table is however ''by no means'' universally accepted.  They supplied no evidence to support it, basing it instead on personal opinion. It has been noted that these guidelines may be more harmful than helpful,{{r|Gwet2010}} as the number of categories and subjects will affect the magnitude of the value. For example, the kappa is higher when there are fewer categories.{{r|Sim2005}}

{|class=wikitable style="margin-left: auto; margin-right: auto; border: none;"
! Condition !! <math>\kappa</math> !! Interpretation
|-
|rowspan=6 align=center| Subjective example: <br/> '''only''' for two annotators, <br/>on two classes.{{r|Landis1977}}
|align=center| < 0               || Poor agreement
|-
|align=center| 0.01 &ndash; 0.20  || Slight agreement
|-
|align=center| 0.21 &ndash; 0.40 || Fair agreement
|-
|align=center| 0.41 &ndash; 0.60 || Moderate agreement
|-
|align=center| 0.61 &ndash; 0.80 || Substantial agreement
|-
|align=center| 0.81 &ndash; 1.00 || Almost perfect agreement
|-
|}

== Tests of significance ==
Statistical packages can calculate a [[standard score]] (Z-score) for [[Cohen's kappa]] or Fleiss's Kappa, which can be converted into a [[P-value]]. However, even when the P value reaches the threshold of statistical significance (typically less than 0.05), it only indicates that the agreement between raters is significantly better than would be expected by chance. The p-value does not tell, by itself, whether the agreement is good enough to have high predictive value.

== See also ==
*[[Pearson product-moment correlation coefficient]]
*[[Matthews correlation coefficient]]
*[[Krippendorff's alpha]]

== References ==
{{reflist|refs=

<ref name="Fleiss1971">{{citation
 | last = Fleiss | first = J. L.
 | year = 1971
 | title = Measuring nominal scale agreement among many raters
 | journal = Psychological Bulletin
 | volume = 76
 | issue = 5
 | pages = 378&ndash;382
 | doi = 10.1037/h0031619
}}.</ref>

<ref name="Gwet2010">{{citation
 | last = Gwet | first = K. L.
 | year = 2014
 | title = Handbook of Inter-Rater Reliability
 | edition = 4th
 | chapter = Chapter 6. (Gaithersburg: Advanced Analytics, LLC)
 | publisher = Advanced Analytics, LLC
 | isbn = 978-0970806284
 | url = http://www.agreestat.com/book4/9780970806284_chap2.pdf
}}.</ref>

<ref name="Hallgren2012">{{citation
 | last = Hallgren | first = Kevin A.
 | year = 2012
 | title = Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
 | journal = Tutorials in Quantitative Methods for Psychology
 | volume = 8
 | issue = 1
 | pages = 3&ndash;34
| doi = 10.20982/tqmp.08.1.p023
 | pmid = 22833776
 }}.</ref>

<ref name="Landis1977">{{citation 
 | last1 = Landis | first1 = J. R.
 | last2 = Koch | first2 = G. G.
 | year = 1977
 | title = The measurement of observer agreement for categorical data
 | journal = Biometrics
 | volume = 33
 | issue = 1
 | pages = 159&ndash;174
 | doi = 10.2307/2529310
 | jstor = 2529310
| pmid = 843571
 | url = https://repositorio.unal.edu.co/handle/unal/80119
 }}.</ref>

<ref name="MiniTab">{{citation
 | title = Kappa statistics for Attribute Agreement Analysis, MiniTab Inc
 | url = https://support.minitab.com/en-us/minitab/18/help-and-how-to/quality-and-process-improvement/measurement-system-analysis/how-to/attribute-agreement-analysis/attribute-agreement-analysis/interpret-the-results/all-statistics-and-graphs/kappa-statistics/
 | access-date = Jan 22, 2019
}}.</ref>

<ref name="Powers2012">{{citation
 | last = Powers | first = David M. W.
 | year = 2012
 | title = The Problem with Kappa
 | volume = Conference of the European Chapter of the Association for Computational Linguistics (EACL2012) Joint ROBUS-UNSUP Workshop.
 | publisher = Association for Computational Linguistics
}}.</ref>

<ref name="Scott1955">{{citation
 | last = Scott | first = W.
 | year = 1955
 | title = Reliability of content analysis: The case of nominal scale coding
 | journal = Public Opinion Quarterly
 | volume = 19
 | issue = 3
 | pages = 321&ndash;325
 | doi = 10.1086/266577
 | jstor = 2746450
}}.</ref>

<ref name="Sim2005">{{citation
 | last1 = Sim | first1 = J.
 | last2 = Wright | first2 = C. C.
 | year = 2005
 | title = The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements
 | journal = Physical Therapy
 | volume = 85
 | issue = 3
 | pages = 257&ndash;268
 | doi = 10.1093/ptj/85.3.257
}}.</ref>

}}

==Further reading==
* {{citation
 | last1 = Fleiss | first1 = J. L.
 | last2 = Cohen | first2 = J.
 | year = 1973
 | title = The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability
 | journal = Educational and Psychological Measurement
 | volume = 33
 | issue = 3
 | pages = 613&ndash;619
| doi = 10.1177/001316447303300309
 | s2cid = 145183399
 }}.
* {{citation
 | last = Fleiss | first = J. L.
 | year = 1981
 | title = Statistical methods for rates and proportions
 | edition = 2nd 
 | publisher = [[John Wiley & Sons]]
 | location = New York
 | pages = 38&ndash;46
}}.
* {{citation
 | last = Gwet
 | first = K. L.
 | year = 2008
 | url = http://www.agreestat.com/research_papers/bjmsp2008_interrater.pdf
 | title = Computing inter-rater reliability and its variance in the presence of high agreement
 | journal = British Journal of Mathematical and Statistical Psychology
 | volume = 61
 | issue = Pt 1
 | pages = 29&ndash;48
 | doi = 10.1348/000711006X126600
 | pmid = 18482474
 | s2cid = 13915043
 | access-date = 2010-06-16
 | archive-date = 2016-03-03
 | archive-url = https://web.archive.org/web/20160303181406/http://www.agreestat.com/research_papers/bjmsp2008_interrater.pdf
 | url-status = dead
 }}.

==External links==
{{Wikibooks|Algorithm implementation|Statistics/Fleiss' kappa|Fleiss' kappa}}
*[https://agreestat360.com/ Cloud-based inter-rater reliability analysis, Cohen's kappa, Gwet's AC1/AC2, Krippendorff's alpha, Brennan-Prediger, Fleiss generalized kappa, intraclass correlation coefficients]
*[http://www.john-uebersax.com/stat/kappa.htm Kappa: Pros and Cons] – contains a good bibliography of articles about the coefficient
*[http://justus.randolph.name/kappa Online Kappa Calculator] {{Webarchive|url=https://web.archive.org/web/20090228090813/http://justus.randolph.name/kappa |date=2009-02-28 }} – calculates a variation of Fleiss' kappa

{{good article}}

[[Category:Categorical variable interactions]]
[[Category:Inter-rater reliability]]
[[Category:Summary statistics for contingency tables]]