View source for Base rate fallacy

{{Short description|Logic error due to ignoring the base rate}}
{{Lead rewrite|date=July 2023}}
[[File:Base rate fallacy with vaccines.svg|thumb|upright=1.3|A hospital receiving more vaccinated [[COVID-19]] patients than unvaccinated ones might suggest that the vaccine is ineffective, but such an imbalance is to be expected within a highly vaccinated population.<ref>{{cite web |date=2023-01-18 |title=COVID-19 Cases, Hospitalizations, and Deaths by Vaccination Status |url=https://doh.wa.gov/sites/default/files/2022-02/421-010-CasesInNotFullyVaccinated.pdf |url-status=live |archive-url=https://web.archive.org/web/20230126022307/https://doh.wa.gov/sites/default/files/2022-02/421-010-CasesInNotFullyVaccinated.pdf |archive-date=2023-01-26 |access-date=2023-02-14 |publisher=Washington State Department of Health |quote=If the exposure to COVID-19 stays the same, as more individuals are vaccinated, more cases, hospitalizations, and deaths will be in vaccinated individuals, as they will continue to make up more and more of the population. For example, if 100% of the population was vaccinated, 100% of cases would be among vaccinated people.}}</ref>]]
The '''base rate fallacy''', also called '''base rate neglect'''<ref>{{Cite journal |last1=Welsh |first1=Matthew B. |last2=Navarro |first2=Daniel J. |date=2012 |title=Seeing is believing: Priors, trust, and base rate neglect |url=http://dx.doi.org/10.1016/j.obhdp.2012.04.001 |journal=Organizational Behavior and Human Decision Processes |volume=119 |issue=1 |pages=1–14 |doi=10.1016/j.obhdp.2012.04.001 |issn=0749-5978 |hdl-access=free |hdl=2440/41190}}</ref> or '''base rate bias''', is a type of [[fallacy]] in which people tend to ignore the [[base rate]] (e.g., general [[prevalence]]) in favor of the individuating information (i.e., information pertaining only to a specific case).<ref>{{cite web |title=Logical Fallacy: The Base Rate Fallacy |url=http://www.fallacyfiles.org/baserate.html |access-date=2013-06-15 |publisher=Fallacyfiles.org}}</ref> For example, if someone hears that a friend is very shy and quiet, they might think the friend is more likely to be a librarian than a salesperson. However, there are far more salespeople than librarians overall—hence making it more likely that their friend is actually a salesperson, even if a greater proportion of librarians fit the description of being shy and quiet. Base rate neglect is a specific form of the more general [[extension neglect]].

It is also called the '''prosecutor's fallacy''' or '''defense attorney's fallacy''' when applied to the results of statistical tests (such as DNA tests) in the context of law proceedings. These terms were introduced by William C. Thompson and Edward Schumann in 1987,<ref name="TS">{{cite journal |last1=Thompson |first1=W.C. |last2=Schumann |first2=E.L. |year=1987 |title=Interpretation of Statistical Evidence in Criminal Trials: The Prosecutor's Fallacy and the Defense Attorney's Fallacy |journal=Law and Human Behavior |volume=11 |issue=3 |page=167 |doi=10.1007/BF01044641 |jstor=1393631 |s2cid=147472915}}</ref><ref>{{cite web |last1=Fountain |first1=John |last2=Gunby |first2=Philip |date=February 2010 |title=Ambiguity, the Certainty Illusion, and Gigerenzer's Natural Frequency Approach to Reasoning with Inverse Probabilities |url=http://uctv.canterbury.ac.nz/viewfile.php/4/sharing/55/74/74/NZEPVersionofImpreciseProbabilitiespaperVersi.pdf |publisher=[[University of Canterbury]] |page=6}}{{Dead link|date=May 2020|bot=InternetArchiveBot|fix-attempted=yes}}</ref> although it has been argued that their definition of the prosecutor's fallacy extends to many additional invalid imputations of guilt or liability that are not analyzable as errors in base rates or [[Bayes's theorem]].<ref>{{Cite journal |last=Suss |first=Richard A. |date=October 4, 2023 |title=The Prosecutor's Fallacy Framed as a Sample Space Substitution |journal=OSF Preprints |language=en |doi=10.31219/osf.io/cs248}}</ref>

==False positive paradox==
An example of the base rate fallacy is the '''false positive paradox''' (also known as '''accuracy paradox'''). This paradox describes situations where there are more [[false positive]] test results than true positives (this means the classifier has a low [[Precision and recall|precision]]). For example, if a facial recognition camera can identify wanted criminals 99% accurately, but analyzes 10,000 people a day, the high accuracy is outweighed by the number of tests; because of this, the program's list of criminals will likely have far more innocents (false positives) than criminals (true positives) because there are far more innocents than criminals overall. The probability of a positive test result is determined not only by the accuracy of the test but also by the characteristics of the sampled population.<ref>{{cite book |last1=Rheinfurth |first1=M. H. |url=https://ntrs.nasa.gov/citations/19980045313 |title=Probability and Statistics in Aerospace Engineering |last2=Howell |first2=L. W. |date=March 1998 |publisher=[[NASA]] |page=16 |quote=MESSAGE: False positive tests are more probable than true positive tests when the overall population has a low prevalence of the disease. This is called the false-positive paradox.}}</ref> The fundamental issue is that the far higher prevalence of true negatives means that the pool of people testing positively will be dominated by false positives, given that even a small fraction of the much larger [negative] group will produce a larger number of indicated positives than the larger fraction of the much smaller [positive] group.

When the prevalence, the proportion of those who have a given condition, is lower than the test's [[false positive rate]], even tests that have a very low risk of giving a false positive ''in an individual case'' will give more false than true positives ''overall''.<ref name="Vacher">{{cite journal |last=Vacher |first=H. L. |date=May 2003 |title=Quantitative literacy - drug testing, cancer screening, and the identification of igneous rocks |url=http://findarticles.com/p/articles/mi_qa4089/is_200305/ai_n9252796/pg_2/ |journal=Journal of Geoscience Education |page=2 |quote=At first glance, this seems perverse: the less the students as a whole use [[steroids]], the more likely a student identified as a user will be a non-user. This has been called the False Positive Paradox}} - Citing: {{cite book |last1=Gonick |first1=L. |title=The cartoon guide to statistics |last2=Smith |first2=W. |publisher=Harper Collins |year=1993 |location=New York |page=49}}</ref>

It is especially counter-intuitive when interpreting a positive result in a test on a low-prevalence [[population (statistics)|population]] after having dealt with positive results drawn from a high-prevalence population.<ref name="Vacher" /> If the false positive rate of the test is higher than the proportion of the ''new'' population with the condition, then a test administrator whose experience has been drawn from testing in a high-prevalence population may [[rule of thumb|conclude from experience]] that a positive test result usually indicates a positive subject, when in fact a false positive is far more likely to have occurred.

==Examples==
===Example 1: Disease===
==== High-prevalence population ====
{| class="wikitable floatright" style="text-align:right;"
! Number<br />of people !! Infected !! Uninfected !! Total
|-
! Test<br />positive
| 400<br />(true positive) || 30<br />(false positive)
| 430
|-
! Test<br />negative
| 0<br />(false negative) || 570<br />(true negative)
| 570
|-
! Total 
| 400 || 600 
! 1000
|}
Imagine running an infectious disease test on a population ''A'' of 1,000 persons, of which 40% are infected. The test has a false positive rate of 5% (0.05) and a false negative rate of zero. The [[expected value|expected outcome]] of the 1,000 tests on population ''A'' would be:

:Infected and test indicates disease ([[true positive]])
::1000 × {{sfrac|40|100}} = 400 people would receive a true positive
:Uninfected and test indicates disease (false positive)
::1000 × {{sfrac|100 – 40|100}} × 0.05 = 30 people would receive a false positive
:The remaining 570 tests are correctly negative.

So, in population ''A'', a person receiving a positive test could be over 93% confident ({{sfrac|400|30 + 400}}) that it correctly indicates infection.

====Low-prevalence population====
{| class="wikitable floatright" style="text-align:right;"
! Number<br />of people !! Infected !! Uninfected !! Total
|-
! Test<br />positive
| 20<br />(true positive) || 49<br />(false positive)
| 69
|-
! Test<br />negative
| 0<br />(false negative) || 931<br />(true negative)
| 931
|-
! Total 
| 20 || 980 
! 1000
|}
Now consider the same test applied to population ''B'', of which only 2% are infected. The expected outcome of 1000 tests on population ''B'' would be:

:Infected and test indicates disease (true positive)
::1000 × {{sfrac|2|100}} = 20 people would receive a true positive
:Uninfected and test indicates disease (false positive)
::1000 × {{sfrac|100 – 2|100}} × 0.05 = 49 people would receive a false positive
:The remaining 931 tests are correctly negative.

In population ''B'', only 20 of the 69 total people with a positive test result are actually infected. So, the probability of actually being infected after one is told that one is infected is only 29% ({{sfrac|20|20 + 49}}) for a test that otherwise appears to be "95% accurate".

A tester with experience of group ''A'' might find it a paradox that in group ''B'', a result that had usually correctly indicated infection is now usually a false positive. The confusion of the [[posterior probability]] of infection with the [[prior probability]] of receiving a false positive is a natural [[fallacy|error]] after receiving a health-threatening test result.

===Example 2: Drunk drivers===
Imagine that a group of police officers have [[breathalyzer]]s displaying false drunkenness in 5% of the cases in which the driver is sober. However, the breathalyzers never fail to detect a truly drunk person. One in a thousand drivers is driving drunk. Suppose the police officers then stop a driver at random to administer a breathalyzer test. It indicates that the driver is drunk. No other information is known about them.

Many would estimate the probability that the driver is drunk as high as 95%, but the correct probability is about 2%.

An explanation for this is as follows: on average, for every 1,000 drivers tested,
* 1 driver is drunk, and it is 100% certain that for that driver there is a ''true'' positive test result, so there is 1 ''true'' positive test result
* 999 drivers are not drunk, and among those drivers there are 5% ''false'' positive test results, so there are 49.95 ''false'' positive test results
Therefore, the probability that any given driver among the 1 + 49.95 = 50.95 positive test results really is drunk is <math>1/50.95 \approx 0.019627</math>.

The validity of this result does, however, hinge on the validity of the initial assumption that the police officer stopped the driver truly at random, and not because of bad driving. If that or another non-arbitrary reason for stopping the driver was present, then the calculation also involves the probability of a drunk driver driving competently and a non-drunk driver driving (in-)competently.

More formally, the same probability of roughly 0.02 can be established using [[Bayes' theorem]]. The goal is to find the probability that the driver is drunk given that the breathalyzer indicated they are drunk, which can be represented as
:<math>p(\mathrm{drunk}\mid D)</math>
where ''D'' means that the breathalyzer indicates that the driver is drunk. Using Bayes's theorem,
:<math>p(\mathrm{drunk}\mid D) = \frac{p(D \mid \mathrm{drunk})\, p(\mathrm{drunk})}{p(D)}.</math>
The following information is known in this scenario: 
:<math>p(\mathrm{drunk}) = 0.001,</math>
:<math>p(\mathrm{sober}) = 0.999,</math>
:<math>p(D\mid\mathrm{drunk}) = 1.00,</math> and
:<math>p(D\mid\mathrm{sober}) = 0.05.</math>
As can be seen from the formula, one needs ''p''(''D'') for Bayes' theorem, which can be computed from the preceding values using the [[law of total probability]]:
:<math>p(D) = p(D \mid \mathrm{drunk})\,p(\mathrm{drunk})+p(D\mid\mathrm{sober})\,p(\mathrm{sober})</math>
which gives
:<math>p(D)= (1.00 \times 0.001) + (0.05 \times 0.999) = 0.05095.</math>
Plugging these numbers into Bayes' theorem, one finds that
:<math>p(\mathrm{drunk}\mid D) = \frac{1.00 \times 0.001}{0.05095} \approx 0.019627,</math>
which is the precision of the test.

===Example 3: Terrorist identification===
In a city of 1 million inhabitants, let there be 100 terrorists and 999,900 non-terrorists. To simplify the example, it is assumed that all people present in the city are inhabitants. Thus, the base rate probability of a randomly selected inhabitant of the city being a terrorist is 0.0001, and the base rate probability of that same inhabitant being a non-terrorist is 0.9999. In an attempt to catch the terrorists, the city installs an alarm system with a surveillance camera and automatic [[facial recognition software]].

The software has two failure rates of 1%:
* The false negative rate: If the camera scans a terrorist, a bell will ring 99% of the time, and it will fail to ring 1% of the time.
* The false positive rate: If the camera scans a non-terrorist, a bell will not ring 99% of the time, but it will ring 1% of the time.

Suppose now that an inhabitant triggers the alarm. Someone making the base rate fallacy would infer that there is a 99% probability that the detected person is a terrorist. Although the inference seems to make sense, it is actually bad reasoning, and a calculation below will show that the probability of a terrorist is actually near 1%, not near 99%.

The fallacy arises from confusing the natures of two different failure rates. The 'number of non-bells per 100 terrorists' (P(¬B | T), or the probability that the bell fails to ring given the inhabitant is a terrorist) and the 'number of non-terrorists per 100 bells' (P(¬T | B), or the probability that the inhabitant is a non-terrorist given the bell rings) are unrelated quantities; one does not necessarily equal—or even be close to—the other. To show this, consider what happens if an identical alarm system were set up in a second city with no terrorists at all. As in the first city, the alarm sounds for 1 out of every 100 non-terrorist inhabitants detected, but unlike in the first city, the alarm never sounds for a terrorist. Therefore, 100% of all occasions of the alarm sounding are for non-terrorists, but a false negative rate cannot even be calculated. The 'number of non-terrorists per 100 bells' in that city is 100, yet P(T | B) = 0%. There is zero chance that a terrorist has been detected given the ringing of the bell.

Imagine that the first city's entire population of one million people pass in front of the camera. About 99 of the 100 terrorists will trigger the alarm—and so will about 9,999 of the 999,900 non-terrorists. Therefore, about 10,098 people will trigger the alarm, among which about 99 will be terrorists. The probability that a person triggering the alarm actually is a terrorist is only about 99 in 10,098, which is less than 1% and very, very far below the initial guess of 99%.

The base rate fallacy is so misleading in this example because there are many more non-terrorists than terrorists, and the number of false positives (non-terrorists scanned as terrorists) is so much larger than the true positives (terrorists scanned as terrorists).

Multiple practitioners have argued that as the base rate of terrorism is extremely low, using [[data mining]] and predictive algorithms to identify terrorists cannot feasibly work due to the false positive paradox.<ref name=":0">{{Cite journal |last=Munk |first=Timme Bisgaard |date=1 September 2017 |title=100,000 false positives for every real terrorist: Why anti-terror algorithms don't work |url=https://firstmonday.org/ojs/index.php/fm/article/view/7126 |journal=First Monday |volume=22 |issue=9 |doi=10.5210/fm.v22i9.7126 |doi-access=free}}</ref><ref name=":1">{{Cite magazine |last=Schneier |first=Bruce |author-link=Bruce Schneier |title=Why Data Mining Won't Stop Terror |url=https://www.wired.com/2006/03/why-data-mining-wont-stop-terror-2/ |access-date=2022-08-30 |magazine=Wired |language=en-US |issn=1059-1028}}</ref><ref name=":2">{{Cite web |last1=Jonas |first1=Jeff |last2=Harper |first2=Jim |date=2006-12-11 |title=Effective Counterterrorism and the Limited Role of Predictive Data Mining |url=https://www.cato.org/policy-analysis/effective-counterterrorism-limited-role-predictive-data-mining# |access-date=2022-08-30 |website=[[CATO Institute]]}}</ref><ref name=":3">{{Cite journal |last=Sageman |first=Marc |author-link=Marc Sageman |date=2021-02-17 |title=The Implication of Terrorism's Extremely Low Base Rate |url=https://doi.org/10.1080/09546553.2021.1880226 |journal=Terrorism and Political Violence |volume=33 |issue=2 |pages=302–311 |doi=10.1080/09546553.2021.1880226 |issn=0954-6553 |s2cid=232341781}}</ref> Estimates of the number of false positives for each accurate result vary from over ten thousand<ref name=":3" /> to one billion;<ref name=":1" /> consequently, investigating each lead would be cost- and time-prohibitive.<ref name=":0" /><ref name=":2" /> The level of accuracy required to make these models viable is likely unachievable. Foremost, the low base rate of terrorism also means there is a lack of data with which to make an accurate algorithm.<ref name=":2" /> Further, in the context of detecting terrorism false negatives are highly undesirable and thus must be minimised as much as possible; however, this requires [[Sensitivity and specificity|increasing sensitivity at the cost of specificity]], increasing false positives.<ref name=":3" /> It is also questionable whether the use of such models by law enforcement would meet the requisite [[Burden of proof (law)|burden of proof]] given that over 99% of results would be false positives.<ref name=":3" />

===Example 4: biological testing of a suspect===
A crime is committed. Forensic analysis determines that the perpetrator has a certain blood type shared by 10% of the population. A suspect is arrested, and found to have that same blood type.

A prosecutor might charge the suspect with the crime on that basis alone, and claim at trial that the probability that the defendant is guilty is 90%.

However, this conclusion is only close to correct if the defendant was selected as the main suspect based on robust evidence discovered prior to the blood test and unrelated to it. Otherwise, the reasoning presented is flawed, as it overlooks the high [[prior probability]] (that is, prior to the blood test) that he is a random innocent person. Assume, for instance, that 1000 people live in the town where the crime occurred. This means that 100 people live there who have the perpetrator's blood type, of whom only one is the true perpetrator; therefore, the true probability that the defendant is guilty – based only on the fact that his blood type matches that of the killer – is only 1%, far less than the 90% argued by the prosecutor.

The prosecutor's fallacy involves assuming that the prior probability of a random match is equal to the probability that the defendant is innocent. When using it, a prosecutor questioning an expert witness may ask: "The odds of finding this evidence on an innocent man are so small that the jury can safely disregard the possibility that this defendant is innocent, correct?"<ref>{{Cite journal |last1=Fenton |first1=Norman |last2=Neil |first2=Martin |last3=Berger |first3=Daniel |date=June 2016 |title=Bayes and the Law |journal=[[Annual Review of Statistics and Its Application]] |volume=3 |issue=1 |pages=51–77 |bibcode=2016AnRSA...3...51F |doi=10.1146/annurev-statistics-041715-033428 |pmc=4934658 |pmid=27398389}}</ref> The claim assumes that the probability that evidence is found on an innocent man is the same as the probability that a man is innocent given that evidence was found on him, which is not true. Whilst the former is usually small (10% in the previous example) due to good [[forensic evidence]] procedures, the latter (99% in that example) does not directly relate to it and will often be much higher, since, in fact, it depends on the likely quite high [[prior odds]] of the defendant being a random innocent person.

==Examples in law==
{{See also|People v. Collins}}

===O. J. Simpson trial===
{{see also|O. J. Simpson murder trial}}

[[O. J. Simpson]] was tried and acquitted in 1995 for the murders of his ex-wife Nicole Brown Simpson and her friend Ronald Goldman.

Crime scene blood matched Simpson's with characteristics shared by 1 in 400 people. However, the defense argued that the number of people from Los Angeles matching the sample could fill a football stadium and that the figure of 1 in 400 was useless.<ref>Robertson, B., & Vignaux, G. A. (1995). '' Interpreting evidence: Evaluating forensic evidence in the courtroom. '' Chichester: John Wiley and Sons.</ref><ref>Rossmo, D. Kim (2009). '' Criminal Investigative Failures. '' CRC Press Taylor & Francis Group.</ref> It would have been incorrect, and an example of prosecutor's fallacy, to rely solely on the "1 in 400" figure to deduce that a given person matching the sample would be likely to be the culprit.

[[File:OJ Simpson case frequency tree.svg|thumb|Frequency tree of 100 000 battered American women showing the base rate fallacy made by the defense in the [[O. J. Simpson murder trial]]]]

In the same trial, the prosecution presented evidence that Simpson had been violent toward his wife. The defense argued that there was only one woman murdered for every 2500 women who were subjected to spousal abuse, and that any history of Simpson being violent toward his wife was irrelevant to the trial. However, the reasoning behind the defense's calculation was fallacious. According to author [[Gerd Gigerenzer]], the correct probability requires additional context: Simpson's wife had not only been subjected to domestic violence, but rather subjected to domestic violence (by Simpson) {{em|and}} killed (by someone). Gigerenzer writes "the chances that a batterer actually murdered his partner, given that she has been killed, is about 8 in 9 or approximately 90%".<ref>Gigerenzer, G., Reckoning with Risk: Learning to Live with Uncertainty, Penguin, (2003)</ref> While most cases of spousal abuse do not end in murder, most cases of murder where there is a history of spousal abuse were committed by the spouse.

===Sally Clark case===
[[Sally Clark]], a British woman, was accused in 1998 of having killed her first child at 11 weeks of age and then her second child at 8 weeks of age. The prosecution had [[expert witness]] Sir [[Roy Meadow]], a professor and consultant paediatrician,<ref>{{Cite magazine |date=30 November 1998 |title=Resolution adopted by the Senate (21 October 1998) on the retirement of Professor Sir Roy Meadow |url=http://reporter.leeds.ac.uk/428/mead.htm |url-status=dead |archive-url=https://web.archive.org/web/20160416125642/http://reporter.leeds.ac.uk/428/mead.htm |archive-date=2016-04-16 |access-date=2015-10-17 |magazine=Reporter |publisher=[[University of Leeds]] |number=428}}</ref> testify that the probability of two children in the same family dying from [[SIDS]] is about 1 in 73 million. That was much less frequent than the actual rate measured in [[Epidemiology|historical data]]{{snd}} Meadow estimated it from single-SIDS death data, and the assumption that the probability of such deaths should be [[uncorrelated]] between infants.<ref>The population-wide probability of a SIDS fatality was about 1 in 1,303; Meadow generated his 1-in-73 million estimate from the lesser probability of SIDS death in the Clark household, which had lower risk factors (e.g. non-smoking). In this sub-population he estimated the probability of a single death at 1 in 8,500. See: {{cite web |last=Joyce |first=H. |date=September 2002 |title=Beyond reasonable doubt |url=http://plus.maths.org/issue21/features/clark/ |access-date=2010-06-12 |publisher=plus.maths.org |format=pdf}}. Professor Ray Hill questioned even this first step (1/8,500 vs 1/1,300) in two ways: firstly, on the grounds that it was [[bias]]ed, excluding those factors that increased risk (especially that both children were boys) and (more importantly) because reductions in SIDS risk factors will proportionately reduce murder risk factors, so that the relative frequencies of [[Münchausen syndrome by proxy]] and SIDS will remain in the same ratio as in the general population: {{cite web |last=Hill |first=Ray |year=2002 |title=Cot Death or Murder? – Weighing the Probabilities |url=http://www.mrbartonmaths.com/resources/a%20level/s1/Beyond%20reasonable%20doubt.doc |quote=it is patently unfair to use the characteristics which basically make her a good, clean-living, mother as factors which count against her.  Yes, we can agree that such factors make a natural death less likely – but those same characteristics also make murder less likely.}}
</ref>

Meadow acknowledged that 1-in-73&nbsp;million is not an impossibility, but argued that such accidents would happen "once every hundred years" and that, in a country of 15 million 2-child families, it is vastly more likely that the double-deaths are due to [[Münchausen syndrome by proxy]] than to such a rare accident. However, there is good reason to suppose that the likelihood of a death from SIDS in a family is significantly greater if a previous child has already died in these circumstances, (a [[genetic predisposition]] to SIDS is likely to invalidate that assumed [[statistical independence]]<ref>{{Cite news |last1=Sweeney |first1=John |last2=Law |first2=Bill |date=July 15, 2001 |title=Gene find casts doubt on double 'cot death' murders |url=http://observer.guardian.co.uk/print/0,,4221973-102285,00.html |archive-url=https://archive.today/20120711221623/http://observer.guardian.co.uk/print/0,,4221973-102285,00.html |archive-date=2012-07-11 |work=[[The Observer]]}}</ref>) making some families more susceptible to SIDS and the error an outcome of the [[ecological fallacy]].<ref>{{cite web |author=Vincent Scheurer |title=Convicted on Statistics? |url=http://understandinguncertainty.org/node/545#notes |access-date=2010-05-21}}</ref> The likelihood of two SIDS deaths in the same family cannot be soundly [[Estimation theory|estimated]] by squaring the likelihood of a single such death in all otherwise similar families.<ref>{{cite journal |last=Hill |first=R. |year=2004 |title=Multiple sudden infant deaths – coincidence or beyond coincidence? |url=http://www.cse.salford.ac.uk/staff/RHill/ppe_5601.pdf |url-status=dead |journal=Paediatric and Perinatal Epidemiology |volume=18 |issue=5 |page=321 |doi=10.1111/j.1365-3016.2004.00560.x |pmid=15367318 |archive-url=https://web.archive.org/web/20120830145646/http://www.cse.salford.ac.uk/staff/RHill/ppe_5601.pdf |archive-date=2012-08-30 |access-date=2010-06-13}}</ref>

The 1-in-73&nbsp;million figure greatly underestimated the chance of two successive accidents, but even if that assessment were accurate, the court seems to have missed the fact that the 1-in-73&nbsp;million number meant nothing on its own. As an ''[[a priori]]'' probability, it should have been weighed against the ''a priori'' probabilities of the alternatives. Given that two deaths had occurred, one of the following explanations must be true, and all of them are ''a priori'' extremely improbable:
# Two successive deaths in the same family, both by SIDS
# Double homicide (the prosecution's case)
# Other possibilities (including one homicide and one case of SIDS)

It is unclear whether an estimate of the probability for the second possibility was ever proposed during the trial, or whether the comparison of the first two probabilities was understood to be the key estimate to make in the statistical analysis assessing the prosecution's case against the case for innocence.

Clark was convicted in 1999, resulting in a press release by the [[Royal Statistical Society]] which pointed out the mistakes.<ref>{{cite web |date=23 October 2001 |title=Royal Statistical Society concerned by issues raised in Sally Clark case |url=http://www.rss.org.uk/uploadedfiles/documentlibrary/744.pdf |url-status=dead |archive-url=https://web.archive.org/web/20110824151124/http://www.rss.org.uk/uploadedfiles/documentlibrary/744.pdf |archive-date=24 August 2011 |quote=Society does not tolerate doctors making serious clinical errors because it is widely understood that such errors could mean the difference between life and death. The case of R v. Sally Clark is one example of a medical expert witness making a serious statistical error, one which may have had a profound effect on the outcome of the case}}</ref>

In 2002, Ray Hill (a mathematics professor at [[University of Salford|Salford]]) attempted to accurately compare the chances of these two possible explanations; he concluded that successive accidents are between 4.5 and 9 times more likely than are successive murders, so that the ''a priori'' [[odds]] of Clark's guilt were between 4.5 to 1 and 9 to 1 against.<ref>The uncertainty in this range is mainly driven by uncertainty in the likelihood of killing a second child, having killed a first, see: {{cite journal |last=Hill |first=R. |year=2004 |title=Multiple sudden infant deaths – coincidence or beyond coincidence? |url=http://www.cse.salford.ac.uk/staff/RHill/ppe_5601.pdf |url-status=dead |journal=Paediatric and Perinatal Epidemiology |volume=18 |issue=5 |pages=322–323 |doi=10.1111/j.1365-3016.2004.00560.x |pmid=15367318 |archive-url=https://web.archive.org/web/20120830145646/http://www.cse.salford.ac.uk/staff/RHill/ppe_5601.pdf |archive-date=2012-08-30 |access-date=2010-06-13}}</ref>

After the court found that the forensic pathologist who had examined both babies had withheld [[exculpatory evidence]], a higher court later quashed Clark's conviction, on 29 January 2003.<ref>{{cite web |title=R v Clark. [2003] EWCA Crim 1020 (11 April 2003) |url=http://www.bailii.org/ew/cases/EWCA/Crim/2003/1020.html |website=www.bailii.org}}</ref>

==Findings in psychology==
In experiments, people have been found to prefer individuating information over general information when the former is available.<ref>{{cite journal |last=Bar-Hillel |first=Maya |author-link=Maya Bar-Hillel |year=1980 |title=The base-rate fallacy in probability judgments |url=http://ratio.huji.ac.il/sites/default/files/publications/dp732.pdf |journal=Acta Psychologica |volume=44 |issue=3 |pages=211–233 |doi=10.1016/0001-6918(80)90046-3}}</ref><ref name="kv1">{{cite journal |last=Kahneman |first=Daniel |author2=Amos Tversky |year=1973 |title=On the psychology of prediction |journal=Psychological Review |volume=80 |issue=4 |pages=237–251 |doi=10.1037/h0034747 |s2cid=17786757}}</ref><ref>{{cite journal |last1=Tversky |first1=Amos |last2=Kahneman |first2=Daniel |date=1974-09-27 |title=Judgment under uncertainty: Heuristics and biases |journal=Science |volume=185 |issue=4157 |pages=1124–1131 |bibcode=1974Sci...185.1124T |doi=10.1126/science.185.4157.1124 |pmid=17835457 |s2cid=143452957}}</ref>

In some experiments, students were asked to estimate the [[grade point average]]s (GPAs) of hypothetical students. When given relevant statistics about GPA distribution, students tended to ignore them if given descriptive information about the particular student even if the new descriptive information was obviously of little or no relevance to school performance.<ref name="kv1" /> This finding has been used to argue that interviews are an unnecessary part of the [[college admission]]s process because interviewers are unable to pick successful candidates better than basic statistics.

Psychologists [[Daniel Kahneman]] and [[Amos Tversky]] attempted to explain this finding in terms of a [[heuristics in judgment and decision making|simple rule or "heuristic"]] called [[representativeness heuristic|representativeness]]. They argued that many judgments relating to likelihood, or to cause and effect, are based on how representative one thing is of another, or of a category.<ref name="kv1" /> Kahneman considers base rate neglect to be a specific form of [[extension neglect]].<ref>{{cite book |last=Kahneman |first=Daniel |title=Choices, Values and Frames |year=2000 |isbn=0-521-62749-4 |editor=Daniel Kahneman and Amos Tversky |chapter=Evaluation by moments, past and future}}</ref> [[Richard Nisbett]] has argued that some [[attributional bias]]es like the [[fundamental attribution error]] are instances of the base rate fallacy: people do not use the "consensus information" (the "base rate") about how others behaved in similar situations and instead prefer simpler [[dispositional attribution]]s.<ref>{{cite book |last=Nisbett |first=Richard E. |title=Cognition and social behavior |author2=E. Borgida |author3=R. Crandall |author4=H. Reed |publisher=John Wiley & Sons, Incorporated |year=1976 |isbn=0-470-99007-4 |editor=J. S. Carroll & J. W. Payne |volume=2 |pages=227–236 |chapter=Popular induction: Information is not always informative}}</ref>

There is considerable debate in psychology on the conditions under which people do or do not appreciate base rate information.<ref name="Koehler1996">{{Cite journal |last1=Koehler |first1=J. J. |year=2010 |title=The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges |journal=Behavioral and Brain Sciences |volume=19 |pages=1–17 |doi=10.1017/S0140525X00041157 |s2cid=53343238}}</ref><ref name="BarbeySloman2007">{{Cite journal |last1=Barbey |first1=A. K. |last2=Sloman |first2=S. A. |year=2007 |title=Base-rate respect: From ecological rationality to dual processes |journal=Behavioral and Brain Sciences |volume=30 |issue=3 |pages=241–254; discussion 255–297 |doi=10.1017/S0140525X07001653 |pmid=17963533 |s2cid=31741077}}</ref> Researchers in the heuristics-and-biases program have stressed empirical findings showing that people tend to ignore base rates and make inferences that violate certain norms of probabilistic reasoning, such as [[Bayes' theorem]]. The conclusion drawn from this line of research was that human probabilistic thinking is fundamentally flawed and error-prone.<ref name="TverskyKahneman1974">{{Cite journal |last1=Tversky |first1=A. |last2=Kahneman |first2=D. |year=1974 |title=Judgment under Uncertainty: Heuristics and Biases |journal=Science |volume=185 |issue=4157 |pages=1124–1131 |bibcode=1974Sci...185.1124T |doi=10.1126/science.185.4157.1124 |pmid=17835457 |s2cid=143452957}}</ref> Other researchers have emphasized the link between cognitive processes and information formats, arguing that such conclusions are not generally warranted.<ref>{{cite journal |last=Cosmides |first=Leda |author2=John Tooby |year=1996 |title=Are humans good intuitive statisticians after all? Rethinking some conclusions of the literature on judgment under uncertainty |journal=Cognition |volume=58 |pages=1–73 |citeseerx=10.1.1.131.8290 |doi=10.1016/0010-0277(95)00664-8 |s2cid=18631755}}</ref><ref name="GigerenzerHoffrage1995">{{Cite journal |last1=Gigerenzer |first1=G. |last2=Hoffrage |first2=U. |year=1995 |title=How to improve Bayesian reasoning without instruction: Frequency formats |journal=Psychological Review |volume=102 |issue=4 |page=684 |citeseerx=10.1.1.128.3201 |doi=10.1037/0033-295X.102.4.684 |s2cid=16281385}}</ref>

Consider again Example 2 from above. The required inference is to estimate the (posterior) probability that a (randomly picked) driver is drunk, given that the breathalyzer test is positive. Formally, this probability can be calculated using Bayes' theorem, as shown above. However, there are different ways of presenting the relevant information. Consider the following, formally equivalent variant of the problem:

: &nbsp;1 out of 1000 drivers are driving drunk. The breathalyzers never fail to detect a truly drunk person. For 50 out of the 999 drivers who are not drunk the breathalyzer falsely displays drunkenness. Suppose the policemen then stop a driver at random, and force them to take a breathalyzer test. It indicates that they are drunk. No other information is known about them. Estimate the probability the driver is really drunk.

In this case, the relevant numerical information—''p''(drunk), ''p''(''D'' | drunk), ''p''(''D'' | sober)—is presented in terms of natural frequencies with respect to a certain reference class (see [[reference class problem]]). Empirical studies show that people's inferences correspond more closely to Bayes' rule when information is presented this way, helping to overcome base-rate neglect in laypeople<ref name="GigerenzerHoffrage1995" /> and experts.<ref name="Hoffrage2000">{{Cite journal |last1=Hoffrage |first1=U. |last2=Lindsey |first2=S. |last3=Hertwig |first3=R. |last4=Gigerenzer |first4=G. |year=2000 |title=Medicine: Communicating Statistical Information |journal=Science |volume=290 |issue=5500 |pages=2261–2262 |doi=10.1126/science.290.5500.2261 |pmid=11188724 |s2cid=33050943 |hdl-access=free |hdl=11858/00-001M-0000-0025-9B18-3}}</ref> As a consequence, organizations like the [[Cochrane Collaboration]] recommend using this kind of format for communicating health statistics.<ref name="Cochrane2011">{{Cite journal |last1=Akl |first1=E. A. |last2=Oxman |first2=A. D. |last3=Herrin |first3=J. |last4=Vist |first4=G. E. |last5=Terrenato |first5=I. |last6=Sperati |first6=F. |last7=Costiniuk |first7=C. |last8=Blank |first8=D. |last9=Schünemann |first9=H. |year=2011 |editor1-last=Schünemann |editor1-first=Holger |title=Using alternative statistical formats for presenting risks and risk reductions |journal=The Cochrane Database of Systematic Reviews |volume=2011 |issue=3 |pages=CD006776 |doi=10.1002/14651858.CD006776.pub2 |pmc=6464912 |pmid=21412897}}</ref> Teaching people to translate these kinds of Bayesian reasoning problems into natural frequency formats is more effective than merely teaching them to plug probabilities (or percentages) into Bayes' theorem.<ref name="SedlmeierGigerenzer2002">{{Cite journal |last1=Sedlmeier |first1=P. |last2=Gigerenzer |first2=G. |year=2001 |title=Teaching Bayesian reasoning in less than two hours |url=http://edoc.mpg.de/175640 |journal=Journal of Experimental Psychology: General |volume=130 |issue=3 |pages=380–400 |doi=10.1037/0096-3445.130.3.380 |pmid=11561916 |s2cid=11147078 |hdl-access=free |hdl=11858/00-001M-0000-0025-9504-E}}</ref> It has also been shown that graphical representations of natural frequencies (e.g., icon arrays, hypothetical outcome plots) help people to make better inferences.<ref name="SedlmeierGigerenzer2002" /><ref name="Brase2008">{{Cite journal |last1=Brase |first1=G. L. |year=2009 |title=Pictorial representations in statistical reasoning |journal=Applied Cognitive Psychology |volume=23 |issue=3 |pages=369–381 |doi=10.1002/acp.1460 |s2cid=18817707}}</ref><ref name="Edwards2002">{{Cite journal |last1=Edwards |first1=A. |last2=Elwyn |first2=G. |last3=Mulley |first3=A. |year=2002 |title=Explaining risks: Turning numerical data into meaningful pictures |journal=BMJ |volume=324 |issue=7341 |pages=827–830 |doi=10.1136/bmj.324.7341.827 |pmc=1122766 |pmid=11934777}}</ref><ref>{{cite book |last1=Kim |first1=Yea-Seul |title=Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems |last2=Walls |first2=Logan A. |last3=Krafft |first3=Peter |last4=Hullman |first4=Jessica |date=2 May 2019 |isbn=9781450359702 |pages=1–14 |chapter=A Bayesian Cognition Approach to Improve Data Visualization |doi=10.1145/3290605.3300912 |chapter-url=https://idl.cs.washington.edu/papers/bayesian-cognition-vis/ |s2cid=57761146 |arxiv=1901.02949}}</ref>

One important reason why natural frequency formats are helpful is that this information format facilitates the required inference because it simplifies the necessary calculations. This can be seen when using an alternative way of computing the required probability ''p''(drunk|''D''):

:<math>p(\mathrm{drunk}\mid D) = \frac{N(\mathrm{drunk} \cap D)}{N(D)} = \frac{1}{51} = 0.0196</math>

where ''N''(drunk &cap; ''D'') denotes the number of drivers that are drunk and get a positive breathalyzer result, and ''N''(''D'') denotes the total number of cases with a positive breathalyzer result. The equivalence of this equation to the above one follows from the axioms of probability theory, according to which ''N''(drunk &cap; ''D'') = ''N'' × ''p'' (''D'' | drunk) × ''p'' (drunk). Importantly, although this equation is formally equivalent to Bayes' rule, it is not psychologically equivalent. Using natural frequencies simplifies the inference because the required mathematical operation can be performed on natural numbers, instead of normalized fractions (i.e., probabilities), because it makes the high number of false positives more transparent, and because natural frequencies exhibit a "nested-set structure".<ref name="Girotto2001">{{Cite journal |last1=Girotto |first1=V. |last2=Gonzalez |first2=M. |year=2001 |title=Solving probabilistic and statistical problems: A matter of information structure and question form |journal=Cognition |volume=78 |issue=3 |pages=247–276 |doi=10.1016/S0010-0277(00)00133-5 |pmid=11124351 |s2cid=8588451}}</ref><ref name="Hoffrage2002">{{Cite journal |last1=Hoffrage |first1=U. |last2=Gigerenzer |first2=G. |last3=Krauss |first3=S. |last4=Martignon |first4=L. |year=2002 |title=Representation facilitates reasoning: What natural frequencies are and what they are not |journal=Cognition |volume=84 |issue=3 |pages=343–352 |doi=10.1016/S0010-0277(02)00050-1 |pmid=12044739 |s2cid=9595672}}</ref>

Not every frequency format facilitates Bayesian reasoning.<ref name="Hoffrage2002" /><ref name="GigerenzerHoffrage1999">{{Cite journal |last1=Gigerenzer |first1=G. |last2=Hoffrage |first2=U. |year=1999 |title=Overcoming difficulties in Bayesian reasoning: A reply to Lewis and Keren (1999) and Mellers and McGraw (1999) |url=http://edoc.mpg.de/2936 |journal=Psychological Review |volume=106 |issue=2 |pages=425 |doi=10.1037/0033-295X.106.2.425 |hdl-access=free |hdl=11858/00-001M-0000-0025-9CB4-8}}</ref> Natural frequencies refer to frequency information that results from ''natural sampling'',<ref name="Kleiter1994">{{Cite book |last1=Kleiter |first1=G. D. |title=Contributions to Mathematical Psychology, Psychometrics, and Methodology |year=1994 |isbn=978-0-387-94169-1 |series=Recent Research in Psychology |pages=375–388 |chapter=Natural Sampling: Rationality without Base Rates |doi=10.1007/978-1-4612-4308-3_27}}</ref> which preserves base rate information (e.g., number of drunken drivers when taking a random sample of drivers). This is different from ''systematic sampling'', in which base rates are fixed ''a priori'' (e.g., in scientific experiments). In the latter case it is not possible to infer the posterior probability ''p''(drunk | positive test) from comparing the number of drivers who are drunk and test positive compared to the total number of people who get a positive breathalyzer result, because base rate information is not preserved and must be explicitly re-introduced using Bayes' theorem.

== See also ==
* [[Precision and recall]]
* {{annotated link|Data dredging}}
* {{annotated link|Evidence under Bayes' theorem}}
* {{annotated link|Inductive argument}}
* {{annotated link|List of cognitive biases}}
* {{annotated link|List of paradoxes}}
* {{annotated link|Misleading vividness}}
* {{annotated link|Prevention paradox}}
* {{annotated link|Simpson's paradox}}
* {{annotated link|Intuitive statistics}}
* [[R v Adams]]

==References==
{{Reflist}}

==External links==
* [http://www.fallacyfiles.org/baserate.html The Base Rate Fallacy] The Fallacy Files

{{Biases}}
{{Fallacies}}

[[Category:Relevance fallacies]]
[[Category:Cognitive biases]]
[[Category:Behavioral finance]]
[[Category:Probability fallacies]]
[[Category:Statistical paradoxes]]