View source for Misuse of statistics

{{Short description|Use of statistical arguments to assert falsehoods}}
{{Other uses|Manipulation (disambiguation){{!}}Manipulation}}
{{Cleanup|reason=this article is badly written, often unclear|date=November 2014}}
[[Statistics]], when used in a misleading fashion, can trick the casual observer into believing something other than what the [[data]] shows. That is, a '''misuse of statistics''' occurs when
a statistical argument asserts a [[falsehood]]. In some cases, the misuse may be accidental. In others, it is purposeful and for the gain of the perpetrator. When the statistical reason involved is false or misapplied, this constitutes a '''statistical [[fallacy]]'''.

The consequences of such misinterpretations can be quite severe.  For example, in medical science, correcting a falsehood may take decades and cost lives.

Misuses can be easy to fall into. Professional scientists, mathematicians and even professional statisticians, can be fooled by even some simple methods, even if they are careful to check everything.  Scientists have been known to fool themselves with statistics due to lack of knowledge of [[probability theory]] and lack of [[standardization]] of their [[statistical test|tests]].

==Definition, limitations and context==
One usable definition is: "Misuse of Statistics: Using numbers in such a manner that – either by intent or through ignorance or carelessness – the conclusions are unjustified or incorrect."{{sfn | Spirer | Spirer| Jaffe | 1998 |p=1}} The "numbers" include [[misleading graph]]ics discussed in other sources.  The term is not commonly encountered in statistics texts and there is no single authoritative definition. It is a generalization of [[How to Lie with Statistics|lying with statistics]] which was richly described by examples from statisticians 60 years ago.

The definition confronts some problems (some are addressed by the source):<ref>{{cite journal
 | last1 = Gardenier | first1 = John |first2=David|last2= Resnik
 | title = The misuse of statistics: concepts, tools, and a research agenda
 | journal = Accountability in Research: Policies and Quality Assurance | volume = 9|issue=2 | pages = 65–74|pmid=12625352
 | year = 2002 | doi=10.1080/08989620212968| s2cid = 24167609 }}</ref>

# Statistics usually produces probabilities; conclusions are provisional
# The provisional conclusions have errors and error rates.  Commonly 5% of the provisional conclusions of significance testing are wrong
# Statisticians are not in complete agreement on ideal methods
# Statistical methods are based on assumptions which are seldom fully met
# Data gathering is usually limited by ethical, practical and financial constraints.

''[[#CITEREFHuff1954|How to Lie with Statistics]]'' acknowledges that statistics can ''legitimately'' take many forms.  Whether the statistics show that a product is "light and economical" or "flimsy and cheap" can be debated whatever the numbers.  Some object to the substitution of statistical correctness for moral leadership (for example) as an objective.  Assigning blame for misuses is often difficult because scientists, pollsters, statisticians and reporters are often employees or consultants.

An insidious misuse of statistics is completed by the listener, observer, audience, or juror.  The supplier provides the "statistics" as numbers or graphics (or before/after photographs), allowing the consumer to draw conclusions that may be unjustified or incorrect. The poor state of public [[statistical literacy]] and the non-statistical nature of human intuition make it possible to mislead without explicitly producing faulty conclusion.  The definition is weak on the responsibility of the consumer of statistics.

A historian listed over 100 fallacies in a dozen categories including those of generalization and those of causation.<ref>{{cite book | last = Fischer | first = David | title = Historians' fallacies: toward a logic of historical thought | publisher = Harper & Row | location = New York | year = 1979 | isbn = 978-0060904982 | pages = 337–338 }}</ref>  A few of the fallacies are explicitly or potentially statistical including sampling, statistical nonsense, statistical probability, false extrapolation, false interpolation and insidious generalization.  All of the technical/mathematical problems of applied probability would fit in the single listed fallacy of statistical probability.  Many of the fallacies could be coupled to statistical analysis, allowing the possibility of a false conclusion flowing from a statistically sound analysis.

An example use of statistics is in the analysis of medical research.  The process includes<ref>{{cite journal
 | last = Strasak | first = Alexander M. |author2=Qamruz Zaman |author3=Karl P. Pfeiffer |author4=Georg Göbel |author5=Hanno Ulmer
  | title = Statistical errors in the medical research-a review of common pitfalls
 | journal = [[Swiss Medical Weekly]] | volume = 137 | issue = 3–4 | pages = 44–49 |pmid=17299669
 | year = 2007 | doi = 10.4414/smw.2007.11587 }}  In this article anything less than the best statistical practice is equated to the potential misuse of statistics.  In a few pages 47 potential statistical errors are discussed; errors in study design, data analysis, documentation, presentation and interpretation.  "[S]tatisticians should be involved early in study design, as mistakes at this point can have major repercussions, negatively affecting all subsequent stages of medical research."</ref><ref name="Indrayan2007">{{cite journal|last1=Indrayan|first1=Abhaya|title=Statistical fallacies in orthopedic research|journal=Indian Journal of Orthopaedics|volume=41|issue=1|year=2007|pages=37–46|doi=10.4103/0019-5413.30524|doi-broken-date=1 November 2024|doi-access=free|pmid=21124681|pmc=2981893}} Contains a rich list of medical misuses of statistics of all types.</ref> experimental planning, the conduct of the experiment, data analysis, drawing the logical conclusions and presentation/reporting.  The report is summarized by the popular press and by advertisers.  Misuses of statistics can result from problems at any step in the process.  The statistical standards ideally imposed on the scientific report are much different than those imposed on the popular press and advertisers; however, cases exist of advertising disguised as science, such as [[Australasian Journal of Bone & Joint Medicine]].  The definition of the misuse of statistics is weak on the required completeness of statistical reporting.  The opinion is expressed that newspapers must provide at least the source for the statistics reported.

==Simple causes==
Many misuses of statistics occur because
* The source is a subject matter expert, not a statistics expert.{{sfn | Spirer | Spirer| Jaffe | 1998 |loc=chapters 7 & 8}}  The source may incorrectly use a method or interpret a result.
* The source is a statistician, not a subject matter expert.{{sfn | Spirer | Spirer| Jaffe | 1998 |loc=chapter 3}}  An expert should know when the numbers being compared describe different things.  Numbers change, as reality does not, when legal definitions or political boundaries change.
* The subject being studied is not well defined,{{sfn | Spirer | Spirer| Jaffe | 1998 |loc=chapter 4}} or some of its aspects are easy to quantify while others hard to quantify or there is no known quantification method (see [[McNamara fallacy]]). For example:
** While [[Intelligence quotient#Criticism and views|IQ tests]] are available and numeric, it is difficult to define what they measure, as intelligence is an elusive concept.
** Publishing "impact" has the same problem.<ref>{{cite journal
 | last = Adler | first = Robert |author2=John Ewing |author3=Peter Taylor
 | title = Citation statistics | journal = Statistical Science
 | volume = 24 | issue = 1 | pages = 1–14 | year = 2009 | doi = 10.1214/09-STS285| doi-access = free }}</ref> Scientific papers and scholarly journals are often rated by "impact", quantified as the number of citations by later publications. Mathematicians and statisticians conclude that impact (while relatively objective) is not a very meaningful measure. "The sole reliance on citation data provides at best an incomplete and often shallow understanding of {{nowrap|research{{tsp}}{{mdash}}}}{{tsp}}an under{{shy}}standing that is valid only when reinforced by other judgments. Numbers are not inherently superior to sound judgments."
** A seemingly simple question about the number of words in the English language immediately encounters questions about archaic forms, accounting for prefixes and suffixes, multiple definitions of a word, variant spellings, dialects, fanciful creations (like ectoplastistics from ectoplasm and statistics),{{sfn | Spirer | Spirer| Jaffe | 1998 |loc=chapter title}} technical vocabulary, and so on.
* Data quality is poor.{{sfn | Spirer | Spirer| Jaffe | 1998 |loc=chapter 5}} Apparel provides an example.  People have a wide range of sizes and body shapes. It is obvious that apparel sizing must be multidimensional. Instead it is complex in unexpected ways. Some [[Clothing sizes#Women|apparel]] is sold by size only (with no explicit consideration of body shape), sizes vary by country and manufacturer and [[Vanity sizing|some sizes]] are deliberately misleading. While sizes are numeric, only the crudest of statistical analyses is possible using the size numbers with care.
* The popular press has limited expertise and mixed motives.<ref>{{Citation|last=Weatherburn |first=Don |title=Uses and abuses of crime statistics |journal=Crime and Justice Bulletin: Contemporary Issues in Crime and Justice |publisher=NSW Bureau of Crime Statistics and Research |issn=1030-1046 |volume=153 |isbn=9781921824357 |date=November 2011 |url=http://www.bocsar.nsw.gov.au/agdbasev7wr/bocsar/documents/pdf/cjb153.pdf |url-status=unfit |archive-url=https://web.archive.org/web/20140621205347/http://www.bocsar.nsw.gov.au/agdbasev7wr/bocsar/documents/pdf/cjb153.pdf |archive-date=June 21, 2014 }} This Australian report on crime statistics provides numerous examples of interpreting and misinterpreting the data. "The increase in media access to information about crime has not been matched by an increase in the quality of media reporting on crime. The misuse of crime statistics by the media has impeded rational debate about law and order." Among the alleged media abuses: selective use of data, selective reporting of facts, misleading commentary, misrepresentation of facts and misleading headlines.  Police and politicians also abused the statistics.</ref> If the facts are not "newsworthy" (which may require exaggeration) they may not be published. The motives of advertisers are even more mixed.
* "Politicians use statistics in the same way that a drunk uses lamp posts—for support rather than illumination" – Andrew Lang (WikiQuote)  "What do we learn from these two ways of looking at the same numbers?  We learn that a clever propagandist, right or left, can almost always find a way to present the data on economic growth that seems to support her case.  And we therefore also learn to take any statistical analysis from a strongly political source with handfuls of salt."<ref>{{cite book | last = Krugman | first = Paul | title = Peddling prosperity: economic sense and nonsense in the age of diminished expectations | publisher = W.W. Norton | location = New York | year = 1994 | isbn = 0-393-03602-2 | page = [https://archive.org/details/peddlingprosperi00krug/page/111 111] | url-access = registration | url = https://archive.org/details/peddlingprosperi00krug/page/111 }}</ref> The term statistics originates from numbers generated for and utilized by the state.  Good government may require accurate numbers, but popular government may require supportive numbers (not necessarily the same).  "The use and misuse of statistics by governments is an ancient art."{{sfn | Spirer | Spirer| Jaffe | 1998 }}

==Types of misuse==

===Discarding unfavorable observations===
{{See also|Publication bias}}

To promote a neutral (useless) product, a company must find or conduct, for example, 40 studies with a confidence level of 95%. If the product is useless, this would produce one study showing the product was beneficial, one study showing it was harmful, and thirty-eight inconclusive studies (38 is 95% of 40). This tactic becomes more effective when there are more studies available. Organizations that do not publish every study they carry out, such as tobacco companies denying a link between smoking and cancer, anti-smoking advocacy groups and media outlets trying to prove a link between smoking and various ailments, or miracle pill vendors, are likely to use this tactic.

[[Ronald Fisher]] considered this issue in his famous [[lady tasting tea]] example experiment (from his 1935 book, ''[[The Design of Experiments]]'').  Regarding repeated experiments, he said, "It would be illegitimate and would rob our calculation of its basis if unsuccessful results were not all brought into the account."

Another term related to this concept is [[Cherry picking (fallacy)|cherry picking]].

===Ignoring important features===
Multivariable datasets have two or more [[Feature (machine learning)|features/dimensions]]. If too few of these features are chosen for analysis (for example, if just one feature is chosen and [[simple linear regression]] is performed instead of [[Linear regression#Simple and multiple linear regression|multiple linear regression]]), the results can be misleading. This leaves the analyst vulnerable to any of various [[:Category:Statistical paradoxes|statistical paradoxes]], or in some (not all) cases false causality as below.

===Loaded questions===
{{Main|Loaded question}}
The answers to surveys can often be manipulated by wording the question in such a way as to induce a prevalence towards a certain answer from the respondent. For example, in polling support for a war, the questions:
* Do you support the attempt by the US to bring freedom and democracy to other places in the world?
* Do you support the unprovoked military action by the USA?
will likely result in data skewed in different directions, although they are both polling about the support for the war. A better way of wording the question could be "Do you support the current US military action abroad?" A still more nearly neutral way to put that question is "What is your view about the current US military action abroad?" The point should be that the person being asked has no way of guessing from the wording what the questioner might want to hear.

Another way to do this is to precede the question by information that supports the "desired" answer. For example, more people will likely answer "yes" to the question "Given the increasing burden of taxes on middle-class families, do you support cuts in income tax?" than to the question "Considering the rising federal budget deficit and the desperate need for more revenue, do you support cuts in income tax?"

The proper formulation of questions can be very subtle, but nonetheless can yield significant differences in results. Additionally, the responses to two questions can vary dramatically depending on the order in which they are asked.{{sfn|Kahneman|2013|p=102}} "A survey that asked about 'ownership of stock' found that most Texas ranchers owned stock, though probably not the kind traded on the New York Stock Exchange."{{sfn|Moore|Notz|2006|p= 59}}

===Overgeneralization===
[[Overgeneralization]] is a fallacy occurring when a statistic about a particular population is asserted to hold among members of a group for which the original population is not a representative sample.

For example, suppose 100% of apples are observed to be red in summer. The assertion "All apples are red" would be an instance of overgeneralization because the original statistic was true only of a specific subset of apples (those in summer), which is not expected to be representative of the population of apples as a whole.

A real-world example of the overgeneralization fallacy can be observed as an artifact of modern polling techniques, which prohibit calling cell phones for over-the-phone political polls. As young people are more likely than other demographic groups to lack a conventional "landline" phone, a telephone poll that exclusively surveys responders of calls landline phones, may cause the poll results to undersample the views of young people, if no other measures are taken to account for this skewing of the sampling.  Thus, a poll examining the voting preferences of young people using this technique may not be a perfectly accurate representation of young peoples' true voting preferences as a whole without overgeneralizing, because the sample used excludes young people that carry only cell phones, who may or may not have voting preferences that differ from the rest of the population.

Overgeneralization often occurs when information is passed through nontechnical sources, in particular mass media.

===Biased samples===
{{Main|Biased sample}}

Scientists have learned at great cost that gathering good experimental data for statistical analysis is difficult.  Example: The [[placebo]] effect (mind over body) is very powerful.  100% of subjects developed a rash when exposed to an inert substance that was falsely called poison ivy while few developed a rash to a "harmless" object that really was poison ivy.{{sfn|Moore|Notz|2006|p= 97}} Researchers combat this effect by double-blind randomized comparative [[experiment]]s.  Statisticians typically worry more about the validity of the data than the analysis.  This is reflected in a field of study within statistics known as the [[design of experiments]].

Pollsters have learned at great cost that gathering good survey data for statistical analysis is difficult.  The selective effect of cellular telephones on data collection (discussed in the Overgeneralization section) is one potential example; If young people with traditional telephones are not representative, the sample can be biased.  Sample surveys have many pitfalls and require great care in execution.{{sfn|Moore|McCabe|2003|pp=252–254}}  One effort required almost 3,000 telephone calls to get 1,000 answers.  The simple random sample of the population "isn't simple and may not be random."{{sfn|Moore|Notz|2006|p=53, Sample surveys in the real world}}

===Misreporting or misunderstanding of estimated error===
{{further|Opinion poll|Statistical survey}}
If a research team wants to know how 300 million people feel about a certain topic, it would be impractical to ask all of them.  However, if the team picks a random sample of about 1,000 people, they can be fairly certain that the results given by this group are representative of what the larger group would have said if they had all been asked.

This confidence can actually be quantified by the [[central limit theorem]] and other mathematical results. Confidence is expressed as a probability of the true result (for the larger group) being within a certain range of the estimate (the figure for the smaller group).  This is the "plus or minus" figure often quoted for statistical surveys.  The probability part of the confidence level is usually not mentioned; if so, it is assumed to be a standard number like 95%.

The two numbers are related.  If a survey has an estimated error of ±5% at 95% confidence, it also has an estimated error of ±6.6% at 99% confidence.  ±<math>x</math>% at 95% confidence is always ±<math>1.32x</math>% at 99% confidence for a normally distributed population.

The smaller the estimated error, the larger the required sample, at a given confidence level; for example, at [[68–95–99.7 rule|95.4%]] confidence:
* ±1% would require 10,000 people.
* ±2% would require 2,500 people.
* ±3% would require 1,111 people.
* ±4% would require 625 people.
* ±5% would require 400 people.
* ±10% would require 100 people. 
* ±20% would require 25 people. 
* ±25% would require 16 people. 
* ±50% would require 4 people.

People may assume, because the confidence figure is omitted, that there is a 100% certainty that the true result is within the estimated error.  This is not mathematically correct.

Many people may not realize that the randomness of the sample is very important.  In practice, many opinion polls are conducted by phone, which distorts the sample in several ways, including exclusion of people who do not have phones, favoring the inclusion of people who have more than one phone, favoring the inclusion of people who are willing to participate in a phone survey over those who refuse, etc.  Non-random sampling makes the estimated error unreliable.

On the other hand, people may consider that statistics are inherently unreliable because not everybody is called, or because they themselves are never polled. People may think that it is impossible to get data on the opinion of dozens of millions of people by just polling a few thousands. This is also inaccurate.{{efn|Some data on accuracy of polls is available.  Regarding one important poll by the U.S. government, "Relatively speaking, both [[sampling error]] and non-sampling [bias] error are tiny."{{sfn|  Freedman | Pisani | Purves | 1998 |loc= chapter 22: Measuring Employment and Unemployment, p. 405}}
The difference between the votes predicted by one private poll and the actually tally for American presidential elections is available for comparison at [http://www.presidency.ucsb.edu/data/preferences.php "Election Year Presidential Preferences: Gallup Poll Accuracy Record: 1936–2012"]. The predictions were typically calculated on the basis of less than 5,000 opinions by likely voters.{{sfn|  Freedman | Pisani | Purves | 1998 |pp= 389–390}}}} A poll with perfect unbiased sampling and truthful answers has a mathematically determined [[margin of error]], which only depends on the number of people polled.

However, often only one margin of error is reported for a survey. When results are reported for population subgroups, a larger margin of error will apply, but this may not be made clear.  For example, a survey of 1,000 people may contain 100 people from a certain ethnic or economic group.  The results focusing on that group will be much less reliable than results for the full population. If the margin of error for the full sample was 4%, say, then the margin of error for such a subgroup could be around 13%.

There are also many other measurement problems in population surveys.

The problems mentioned above apply to all statistical experiments, not just population surveys.

===False causality===
{{Main|Correlation does not imply causation}}

When a statistical test shows a correlation between A and B, there are usually six possibilities:

# A causes B.
# B causes A.
# A and B both partly cause each other.
# A and B are both caused by a third factor, C.
# B is caused by C which is correlated to A.
# The observed correlation was due purely to chance.

The sixth possibility can be quantified by statistical tests that can calculate the probability that the correlation observed would be as large as it is just by chance if, in fact, there is no relationship between the variables. However, even if that possibility has a small probability, there are still the five others.

If the number of people buying ice cream at the beach is statistically related to the number of people who drown at the beach, then nobody would claim ice cream causes drowning because it's obvious that it isn't so. (In this case, both drowning and ice cream buying are clearly related by a third factor: the number of people at the beach).

This fallacy can be used, for example, to prove that exposure to a chemical causes cancer. Replace "number of people buying ice cream" with "number of people exposed to chemical X", and "number of people who drown" with "number of people who get cancer", and many people will believe you. In such a situation, there may be a statistical correlation even if there is no real effect. For example, if there is a perception that a chemical site is "dangerous" (even if it really isn't) property values in the area will decrease, which will entice more low-income families to move to that area. If low-income families are more likely to get cancer than high-income families (due to a poorer diet, for example, or less access to medical care) then rates of cancer will go up, even though the chemical itself is not dangerous. It is believed<ref name=Farley2003>{{cite web|last=Farley|first=John W.|title=Power Lines and Cancer: Nothing to Fear |publisher=Quackwatch| editor-link=Stephen Barrett |editor-last=Barrett |editor-first=Stephen |url = http://www.quackwatch.org/01QuackeryRelatedTopics/emf.html |year=2003}}</ref> that this is exactly what happened with some of the early studies showing a link between EMF ([[electromagnetic field]]s) from power lines and [[Electromagnetic radiation and health#Leukemia and cancer|cancer]].<ref name=powerlines>{{cite news|last1=Vince |first1=Gaia |title=Large study links power lines to childhood cancer |url=http://www.newscientist.com/article/dn7460-large-study-links-power-lines-to-childhood-cancer.html |journal=New Scientist |date=2005-06-03 |url-status=unfit |archive-url=https://web.archive.org/web/20140816043956/http://www.newscientist.com/article/dn7460-large-study-links-power-lines-to-childhood-cancer.html |archive-date=August 16, 2014 }} Cites: {{cite journal|last1=Draper|first1=G.|title=Childhood cancer in relation to distance from high voltage power lines in England and Wales: a case-control study|journal=BMJ|volume=330|issue=7503|year=2005|pages=1290|pmc=558197|doi=10.1136/bmj.330.7503.1290|pmid=15933351}}</ref>

In well-designed studies, the effect of false causality can be eliminated by assigning some people into a "treatment group" and some people into a "control group" at random, and giving the treatment group the treatment and not giving the control group the treatment. In the above example, a researcher might expose one group of people to chemical X and leave a second group unexposed. If the first group had higher cancer rates, the researcher knows that there is no third factor that affected whether a person was exposed because he controlled who was exposed or not, and he assigned people to the exposed and non-exposed groups at random. However, in many applications, actually doing an experiment in this way is either prohibitively expensive, infeasible, unethical, illegal, or downright impossible. For example, it is highly unlikely that an [[Institutional Review Board|IRB]] would accept an experiment that involved intentionally exposing people to a dangerous substance in order to test its toxicity. The obvious ethical implications of such types of experiments limit researchers' ability to empirically test causation.

===Proof of the null hypothesis===
In a statistical test, the [[null hypothesis]] (<math>H_0</math>) is considered valid until enough data proves it wrong. Then <math>H_0</math> is rejected and the alternative hypothesis (<math>H_A</math>) is considered to be proven as correct. By chance this can happen, although <math>H_0</math> is true, with a probability denoted <math>\alpha</math> (the significance level). This can be compared to the judicial process, where the accused is considered innocent (<math>H_0</math>) until proven guilty (<math>H_A</math>) beyond reasonable doubt (<math>\alpha</math>).

But if data does not give us enough proof to reject that <math>H_0</math>, this does not automatically prove that <math>H_0</math> is correct. If, for example, a tobacco producer wishes to demonstrate that its products are safe, it can easily conduct a test with a small sample of smokers versus a small sample of non-smokers. It is unlikely that any of them will develop lung cancer (and even if they do, the difference between the groups has to be very big in order to reject <math>H_0</math>). Therefore, it is likely—even when smoking is dangerous—that our test will not reject <math>H_0</math>. If <math>H_0</math> is accepted, it does not automatically follow that smoking is proven harmless. The test has insufficient power to reject <math>H_0</math>, so the test is useless and the value of the "proof" of <math>H_0</math> is also null.

This can—using the judicial analogue above—be compared with the truly guilty defendant who is released just because the proof is not enough for a guilty verdict. This does not prove the defendant's innocence, but only that there is not proof enough for a guilty verdict.

"...the null hypothesis is never proved or established, but it is possibly disproved, in the course of experimentation.  Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis."  (Fisher in ''[[The Design of Experiments]]'')  Many reasons for confusion exist including the use of double negative logic and terminology resulting from the merger of Fisher's "significance testing" (where the null hypothesis is never accepted) with "hypothesis testing" (where some hypothesis is always accepted).

===Confusing statistical significance with practical significance===
Statistical significance is a measure of probability; practical significance is a measure of effect.{{sfn|Moore|McCabe|2003|pp=463}} A baldness cure is statistically significant if a sparse peach-fuzz usually covers the previously naked scalp.  The cure is practically significant when a hat is no longer required in cold weather and the barber asks how much to take off the top.  The bald want a cure that is both statistically and practically significant; It will probably work and if it does, it will have a big hairy effect.  Scientific publication often requires only statistical significance.  This has led to complaints (for the last 50 years) that statistical significance testing is a misuse of statistics.<ref name="Rozeboom1960">{{cite journal|last1=Rozeboom|first1=William W.|title=The fallacy of the null-hypothesis significance test|journal=Psychological Bulletin|volume=57|issue=5|year=1960|pages=416–428|doi=10.1037/h0042040|pmid=13744252}}</ref>

===Data dredging===
{{Main|Data dredging}}
[[Data dredging]] is an abuse of [[data mining]]. In data dredging, large compilations of data are examined in order to find a correlation, without any pre-defined choice of a [[hypothesis]] to be tested.  Since the required [[confidence interval]] to establish a relationship between two parameters is usually chosen to be 95% (meaning that there is a 95% chance that the relationship observed is not due to random chance), there is thus a 5% chance of finding a correlation between any two sets of completely random variables. Given that data dredging efforts typically examine large datasets with many variables, and hence even larger numbers of pairs of variables, spurious but apparently statistically significant results are almost certain to be found by any such study.

Note that data dredging is a valid way of ''finding'' a possible hypothesis but that hypothesis ''must'' then be tested with data not used in the original dredging.  The misuse comes in when that hypothesis is stated as fact without further validation.

"You cannot legitimately test a hypothesis on the same data that first suggested that hypothesis.  The remedy is clear.  Once you have a hypothesis, design a study to search specifically for the effect you now think is there.  If the result of this test is statistically significant, you have real evidence at last."{{sfn|Moore|McCabe|2003|p=466}}

===Data manipulation===
{{distinguish | text=[[Data processing]], [[Data preparation]], or [[Data wrangling]], overlapping terms which are often referred to generally as "data manipulation"}}

Informally called "fudging the data," this practice includes selective reporting (see also [[publication bias]]) and even simply making up false data.

Examples of selective reporting abound. The easiest and most common examples involve choosing a group of results that follow a pattern [[consistent]] with the preferred [[hypothesis]] while ignoring other results or "data runs" that contradict the hypothesis.

Scientists, in general, question the validity of study results that cannot be reproduced by other investigators. However, some scientists refuse to publish their data and methods.<ref>{{cite journal|last=Neylon |first=C |year=2009 |title=Scientists lead the push for open data sharing |journal=Research Information |publisher=Europa Science |volume=41 |pages=22–23 |url=http://www.researchinformation.info/features/feature.php?feature_id=214 |issn=1744-8026 |url-status=unfit |archive-url=https://web.archive.org/web/20131203050247/http://www.researchinformation.info/features/feature.php?feature_id=214 |archive-date=December 3, 2013 }}</ref>

Data manipulation is a serious issue/consideration in the most honest of statistical analyses.  Outliers, missing data and non-normality can all adversely affect the validity of statistical analysis.  It is appropriate to study the data and repair real problems before analysis begins.  "[I]n any scatter diagram there will be some points more or less detached from the main part of the cloud: these points should be rejected only for cause."<ref name=fpp3>{{harvnb|  Freedman | Pisani | Purves | 1998 |loc=chapter 9: More about correlations, §3: Some exceptional cases}}</ref>

===Other fallacies===
[[Pseudoreplication]] is a technical error associated with [[analysis of variance]].  Complexity hides the fact that statistical analysis is being attempted on a single sample (N=1).  For this degenerate case the variance cannot be calculated (division by zero). An (N=1) will always give the researcher the highest statistical correlation between intent bias and actual findings.

The [[gambler's fallacy]] assumes that an event for which a future likelihood can be measured had the same likelihood of happening once it has already occurred. Thus, if someone had already tossed 9 coins and each has come up heads, people tend to assume that the likelihood of a tenth toss also being heads is 1023 to 1 against (which it was before the first coin was tossed) when in fact the chance of the tenth head is 50% (assuming the coin is unbiased).

The [[prosecutor's fallacy]]<ref>{{cite book | last = Seife | first = Charles | title = Proofiness: how you're being fooled by the numbers | publisher = Penguin | location = New York | year = 2011 | isbn = 9780143120070|pages=203–205 and Appendix C}}  Discusses the notorious British case.</ref> assumes that the probability of an apparently criminal event being random chance is equal to the chance that the suspect is innocent. A prominent example in the UK is the wrongful conviction of [[Sally Clark]] for killing her two sons who appeared to have died of [[Sudden Infant Death Syndrome]] (SIDS). In his expert testimony, now discredited Professor Sir [[Roy Meadow]] claimed that due to the rarity of SIDS, the probability of Clark being innocent was 1 in 73 million. This was later questioned by the [[Royal Statistical Society]];<ref name=rss2>Royal Statistical Society (23 October 2001).  "{{cite web |url= http://www.rss.org.uk/uploadedfiles/documentlibrary/744.pdf |archive-url= https://web.archive.org/web/20110824151124/http://www.rss.org.uk/uploadedfiles/documentlibrary/744.pdf |url-status= dead |archive-date= 2011-08-24 |title= Royal Statistical Society concerned by issues raised in Sally Clark case }}&nbsp;{{small|(28.0&nbsp;KB)}}"</ref> assuming Meadows figure was accurate, one has to weigh up all the possible explanations against each other to make a conclusion on which most likely caused the unexplained death of the two children. Available data suggest that the odds would be in favour of double SIDS compared to double homicide by a factor of nine.<ref>{{cite journal|title=Multiple sudden infant deaths – coincidence or beyond coincidence?|journal = Paediatric and Perinatal Epidemiology|volume = 18|issue = 5|pages = 320–6|pmid = 15367318|year = 2004|last1 = Hill|first1 = R.|doi = 10.1111/j.1365-3016.2004.00560.x}}</ref> The 1 in 73 million figure was also misleading as it was reached by finding the probability of a baby from an affluent, non-smoking family dying from SIDS and [[Square (algebra)|squaring]] it: this erroneously treats each death as [[statistically independent]], assuming that there is no factor, such as genetics, that would make it more likely for two siblings to die from SIDS.<ref>{{Cite web |title=Beyond reasonable doubt |url=https://plus.maths.org/content/beyond-reasonable-doubt |access-date=2022-04-01 |website=Plus Maths |language=en}}</ref><ref name=":0">{{Cite journal |last=Watkins |first=Stephen J. |date=2000-01-01 |title=Conviction by mathematical error?: Doctors and lawyers should get probability theory right |url=https://www.bmj.com/content/320/7226/2 |journal=BMJ |language=en |volume=320 |issue=7226 |pages=2–3 |doi=10.1136/bmj.320.7226.2 |issn=0959-8138 |pmc=1117305 |pmid=10617504}}</ref> This is also an example of the [[ecological fallacy]] as it assumes the probability of SIDS in Clark's family was the same as the average of all affluent, non-smoking families; social class is a highly complex and multifaceted concept, with numerous other variables such as education, line of work, and many more. Assuming that an individual will have the same attributes as the rest of a given group fails to account for the effects of other variables which in turn can be misleading.<ref name=":0" /> The conviction of [[Sally Clark]] was eventually overturned and Meadow was struck from the  medical register.<ref>{{Cite journal |last=Dyer |first=Clare |date=2005-07-21 |title=Professor Roy Meadow struck off |url=https://www.bmj.com/content/331/7510/177.1 |journal=BMJ |language=en |volume=331 |issue=7510 |pages=177 |doi=10.1136/bmj.331.7510.177 |issn=0959-8138 |pmc=1179752 |pmid=16037430}}</ref>

The [[ludic fallacy]].  Probabilities are based on simple models that ignore real (if remote) possibilities.  Poker players do not consider that an opponent may draw a gun rather than a card.  The insured (and governments) assume that insurers will remain solvent, but see [[American International Group#Financial crisis|AIG]] and [[Systemic risk#Systemic risk and insurance|systemic risk]].

===Other types of misuse===
Other misuses include comparing [[apples and oranges]], using the wrong average,{{sfn|Huff|1954| loc=chapter 2}} [[regression toward the mean]],{{sfn|Kahneman|2013|loc=chapter 17}} and the umbrella phrase [[garbage in, garbage out]].{{sfn| Hooke | 1983 | loc=§50 }} Some statistics are simply irrelevant to an issue.{{sfn | Campbell | 1974  | loc=[https://books.google.com/books?id=GtSV7rG6Iu4C&pg=PA25 chapter 3: Meaningless statistics]}}

Certain advertising phrasing such as "[m]ore than 99 in 100," may be misinterpreted as 100%.<ref>{{cite web |last1=Mazer |first1=Robert |title=LABORATORY'S MARKETING MATERIALS MAY EXPOSE LAB TO LEGAL CLAIMS |url=https://www.linkedin.com/pulse/laboratorys-marketing-materials-may-expose-lab-legal-claims-mazer-punae?trk=article-ssr-frontend-pulse_more-articles_related-content-card |publisher=LinkedIn |access-date=10 April 2024 |language=en}}</ref>

[[Anscombe's quartet]] is a made-up dataset that exemplifies the shortcomings of simple [[descriptive statistics]] (and the value of [[Plot (graphics)|data plotting]] before numerical analysis).

==See also==
* [[Deception]]
* [[Ecological fallacy]]
* [[Ethics in mathematics]]
* [[Metascience (research)|Metascience]]
* [[Misconceptions about the normal distribution]]
* [[Misuse of p-values]]
* [[Misleading graph]]
* [[Post hoc analysis]]
* [[Simpson's paradox]]
* [[Statcheck]]

==References==

===Notes===
{{notelist}}

===Sources===
{{Reflist|30em}}

==Further reading==
{{Commons category|Misuse of statistics}}
{{wikiquote}}
* {{cite book | last = Campbell | first = Stephen | title = Flaws and fallacies in statistical thinking |publisher = Prentice Hall | year = 1974 | isbn = 0-486-43598-9 |url=https://books.google.com/books?id=GtSV7rG6Iu4C}}
* {{cite journal | last1 = Christensen | first1 = R. | last2 = Reichert | first2 = T. | year = 1976 | title = Unit Measure Violations in Pattern Recognition, Ambiguity and Irrelevancy | journal = Pattern Recognition | volume = 8 | issue = 4| pages = 239–245 | doi = 10.1016/0031-3203(76)90044-3 | bibcode = 1976PatRe...8..239C }}
*{{cite journal|vauthors=Ercan I, Yazici B, Yang Y, Ozkaya G, Cangur S, Ediz B, Kan I |year=2007 |title=Misusage of Statistics in Medical Researches |journal=European Journal of General Medicine |volume=4 |issue=3 |pages=127–133 |url=http://www.ejgm.org/upload/sayi/1016/EJGM-00166.pdf |url-status=unfit |archive-url=https://web.archive.org/web/20141113182754/http://www.ejgm.org/upload/sayi/1016/EJGM-00166.pdf |archive-date=November 13, 2014 |doi=10.29333/ejgm/82507 |doi-access=free }}
*{{cite journal|vauthors=Ercan I, Yazici B, Ocakoglu G, Sigirli D, Kan I |title=Review of Reliability and Factors Affecting the Reliability |journal=InterStat |year=2007 |url=http://biostat.home.uludag.edu.tr/Review%20Of%20Reliability%20And%20Factors%20Affecting%20The%20Reliability%20.pdf |url-status=unfit |archive-url=https://web.archive.org/web/20130828175105/http://biostat.home.uludag.edu.tr/Review%20Of%20Reliability%20And%20Factors%20Affecting%20The%20Reliability%20.pdf |archive-date=August 28, 2013 }}
*{{cite book|last1=Freedman|first1=David|last2=Pisani|first2=Robert|last3=Purves|first3=Roger|title=Statistics|url=https://archive.org/details/statistics00free|url-access=registration|edition=3rd|year=1998|publisher=W.W. Norton|isbn=978-0-393-97083-8}}
* {{cite book | last = Hooke | first = Robert | title = How to tell the liars from the statisticians | url = https://archive.org/details/howtotellliarsfr0000hook | url-access = registration | publisher = M. Dekker | location = New York | year = 1983 | isbn = 0-8247-1817-8 }}
* {{cite book|last=Huff|first=Darrell|title=How to Lie with Statistics|year=1954|publisher=W. W. Norton & Company|lccn=53013322 | ol=6138576M |title-link=How to Lie with Statistics}}
*{{cite book | last = Kahneman | first = Daniel | title = Thinking, fast and slow | publisher = Farrar, Straus and Giroux | location = New York | year = 2013 | isbn = 9780374533557 | url-access = registration | url = https://archive.org/details/thinkingfastslow0000kahn }}
*{{cite book | last1 = Moore | first1 = David |first2=George P. |last2= McCabe | title = Introduction to the practice of statistics | url = https://archive.org/details/isbn_9780716749127 | url-access = registration | edition = 4th | publisher = W.H. Freeman and Co | location = New York | year = 2003 | isbn = 0716796570 }}
*{{cite book | last1 = Moore | first1 = David |first2=William I. |last2=Notz | title = Statistics: concepts and controversies | publisher = W.H. Freeman | edition = 6th | location = New York | year = 2006 | isbn = 9780716786368 }}
<!--*{{cite book|last1=Jaffe|first1=Abram J.|last2=Spirer|first2=Herbert F.|title=Misused statistics: straight talk for twisted numbers|url=https://books.google.com/books?id=xC-2AAAAIAAJ|year=1987|publisher=M. Dekker|location = New York|isbn=978-0-8247-7631-2 }}-->
*{{cite book | last1 = Spirer | first1 = Herbert | first2= Louise |last2=Spirer| first3=A. J.| last3= Jaffe | title = Misused statistics | publisher = M. Dekker | location = New York | year = 1998 | edition = revised and expanded 2nd| isbn = 978-0824702113 |url=https://books.google.com/books?id=hT_ELDzB99gC  }} The book is based on several hundred examples of misuse.
* Oldberg, T. and R. Christensen (1995) "Erratic Measure" in ''NDE for the Energy Industry 1995'', The American Society of Mechanical Engineers. {{ISBN|0-7918-1298-7}} (pages 1–6) [http://www.ndt.net/article/v04n05/oldberg/oldberg.htm Republished on the Web by ndt.net]
* Oldberg, T. (2005) "An Ethical Problem in the Statistics of Defect Detection Test Reliability," Speech to the Golden Gate Chapter of the [[American Society for Nondestructive Testing]]. [http://www.ndt.net/article/v10n05/oldberg/oldberg.htm Published on the Web by ndt.net]
* Stone, M. (2009) ''Failing to Figure: Whitehall's Costly Neglect of Statistical Reasoning'', Civitas, London. {{ISBN|1-906837-07-4}}
* {{Cite journal
| last1 = Galbraith | first1 = J.
| last2 = Stone | first2 = M.
| doi = 10.1111/j.1467-985X.2010.00700.x
| title = The abuse of regression in the National Health Service allocation formulae: Response to the Department of Health's 2007 'resource allocation research paper'
| journal = [[Journal of the Royal Statistical Society, Series A]]
| volume = 174
| issue = 3
| pages = 517–528
| year = 2011
| s2cid = 118029429
| doi-access = free
}}

{{Misuse of statistics}}

{{DEFAULTSORT:Misuse Of Statistics}}
[[Category:Misuse of statistics| ]]
[[Category:Ethics and statistics]]