View source for Big data

{{Short description|Extremely large or complex datasets}}
{{About|large collections of data|the band|Big Data (band)|the practice of buying and selling of personal and consumer data|Surveillance capitalism}}

{{Use dmy dates|date=January 2020}}
[[File:Hilbert InfoGrowth.png|thumb|right|400px|Non-linear growth of digital global information-storage capacity and the waning of analog storage<ref>{{cite journal|url= http://www.martinhilbert.net/WorldInfoCapacity.html|title= The World's Technological Capacity to Store, Communicate, and Compute Information|volume= 332|issue= 6025|pages= 60–65|journal= Science |access-date= 13 April 2016|bibcode= 2011Sci...332...60H|last1= Hilbert|first1= Martin|last2= López|first2= Priscila |year= 2011|doi= 10.1126/science.1200970|pmid= 21310967|s2cid= 206531385 |archive-date= 14 April 2016|archive-url= https://web.archive.org/web/20160414233810/http://www.martinhilbert.net/WorldInfoCapacity.html/|url-status= live |doi-access= free}}</ref>{{Update inline|date=June 2024}}]]

'''Big data''' primarily refers to [[data set]]s that are too large or complex to be dealt with by traditional [[data processing|data-processing]] [[application software|software]]. Data with many entries (rows) offer greater [[statistical power]], while data with higher complexity (more attributes or columns) may lead to a higher [[false discovery rate]].<ref>{{Cite journal |last= Breur |first=Tom|date=July 2016|title=Statistical Power Analysis and the contemporary "crisis" in social sciences|journal=Journal of Marketing Analytics |publisher=[[Palgrave Macmillan]] |location= London, England |volume=4 |issue=2–3 |pages=61–65 |doi=10.1057/s41270-016-0001-3 |issn=2050-3318|doi-access=free}}</ref>

Big data analysis challenges include [[Automatic identification and data capture|capturing data]], [[Computer data storage|data storage]], [[data analysis]], search, [[Data sharing|sharing]], [[Data transmission|transfer]], [[Data visualization|visualization]], [[Query language|querying]], updating, [[information privacy]], and data source. Big data was originally associated with three key concepts: ''volume'', ''variety'', and ''velocity''.<ref>{{Cite journal |last1=McAfee |first1=Andrew |last2=Brynjolfsson |first2=Erik |date=2012-10-01 |title=Big data: the management revolution |url=https://pubmed.ncbi.nlm.nih.gov/23074865/ |journal=Harvard Business Review |volume=90 |issue=10 |pages=60–66, 68, 128 |issn=0017-8012 |pmid=23074865}}</ref> The analysis of big data presents challenges in sampling, and thus previously allowing for only observations and sampling. Thus a fourth concept, ''veracity,'' refers to the quality or insightfulness of the data.<ref name=":0" /> Without sufficient investment in expertise for big data veracity, the volume and variety of data can produce costs and risks that exceed an organization's capacity to create and capture ''value'' from big data''.''<ref>{{Cite journal |last1=Cappa |first1= Francesco |last2=Oriani |first2=Raffaele |last3=Peruffo |first3=Enzo |last4=McCarthy |first4=Ian |date=2021 |title=Big Data for Creating and Capturing Value in the Digitalized Environment: Unpacking the Effects of Volume, Variety, and Veracity on Firm Performance |url=https://onlinelibrary.wiley.com/doi/10.1111/jpim.12545 |journal=Journal of Product Innovation Management |language=en |volume=38 |issue=1 |pages=49–67 |doi=10.1111/jpim.12545 |s2cid=225209179 |issn=0737-6782}}</ref>

Current usage of the term ''big data'' tends to refer to the use of [[predictive analytics]], [[user behavior analytics]], or certain other advanced data analytics methods that extract [[Data valuation|value]] from big data, and seldom to a particular size of data set. "There is little doubt that the quantities of data now available are indeed large, but that's not the most relevant characteristic of this new data ecosystem."<ref>{{cite journal |last1=boyd |first1=dana |last2=Crawford |first2=Kate |title=Six Provocations for Big Data |journal=Social Science Research Network: A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society |date=21 September 2011 |doi=10.2139/ssrn.1926431 |s2cid=148610111 |url=http://osf.io/nrjhn/ |access-date=12 July 2019 |archive-date=28 February 2020 |archive-url=https://web.archive.org/web/20200228015558/https://osf.io/nrjhn/ |url-status=live }}</ref>
Analysis of data sets can find new correlations to "spot business trends, prevent diseases, combat crime and so on".{{r|Economist}} Scientists, business executives, medical practitioners, advertising and [[Government database|governments]] alike regularly meet difficulties with large data-sets in areas including [[Web search engine|Internet searches]], [[fintech]], healthcare analytics, geographic information systems, [[urban informatics]], and [[business informatics]]. Scientists encounter limitations in [[e-Science]] work, including [[meteorology]], [[genomics]],<ref>{{cite journal | title = Community cleverness required | journal = Nature | volume = 455 | issue = 7209 | pages = 1 | date = September 2008 | pmid = 18769385 | doi = 10.1038/455001a | bibcode = 2008Natur.455....1. | doi-access = free }}</ref> [[connectomics]], complex physics simulations, biology, and environmental research.<ref>{{cite journal | vauthors = Reichman OJ, Jones MB, Schildhauer MP | title = Challenges and opportunities of open data in ecology | journal = Science | volume = 331 | issue = 6018 | pages = 703–5 | date = February 2011 | pmid = 21311007 | doi = 10.1126/science.1197962 | bibcode = 2011Sci...331..703R | s2cid = 22686503 | url = https://escholarship.org/uc/item/7627s45z | access-date = 12 July 2019 | archive-date = 19 October 2020 | archive-url = https://web.archive.org/web/20201019060839/https://escholarship.org/uc/item/7627s45z | url-status = live }}</ref>

The size and number of available data sets have grown rapidly as data is collected by devices such as [[mobile device]]s, cheap and numerous information-sensing [[Internet of things]] devices, aerial ([[remote sensing]]) equipment, software logs, [[Digital camera|cameras]], microphones, [[radio-frequency identification]] (RFID) readers and [[wireless sensor networks]].<ref>{{cite web |author= Hellerstein, Joe |title= Parallel Programming in the Age of Big Data |date= 9 November 2008 |work= Gigaom Blog |url= http://gigaom.com/2008/11/09/mapreduce-leads-the-way-for-parallel-programming/ |access-date= 21 April 2010 |archive-date= 7 October 2012 |archive-url= https://web.archive.org/web/20121007033928/http://gigaom.com/2008/11/09/mapreduce-leads-the-way-for-parallel-programming/ |url-status= live }}</ref><ref>{{cite book |first1= Toby |last1= Segaran |first2= Jeff |last2= Hammerbacher |title= Beautiful Data: The Stories Behind Elegant Data Solutions |url= https://books.google.com/books?id=zxNglqU1FKgC |year= 2009 |publisher= O'Reilly Media |isbn= 978-0-596-15711-1 |page= 257 |access-date= 31 December 2015 |archive-date= 12 May 2016 |archive-url= https://web.archive.org/web/20160512110157/https://books.google.com/books?id=zxNglqU1FKgC |url-status= live }}</ref> The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s;<ref name="martinhilbert.net">{{cite journal | vauthors = Hilbert M, López P | title = The world's technological capacity to store, communicate, and compute information | journal = Science | volume = 332 | issue = 6025 | pages = 60–5 | date = April 2011 | pmid = 21310967 | doi = 10.1126/science.1200970 | url = http://www.uvm.edu/pdodds/files/papers/others/2011/hilbert2011a.pdf | bibcode = 2011Sci...332...60H | s2cid = 206531385 | access-date = 11 May 2019 | archive-date = 19 August 2019 | archive-url = https://web.archive.org/web/20190819122943/http://www.uvm.edu/pdodds/files/papers/others/2011/hilbert2011a.pdf | url-status = live }}</ref> {{As of|2012|lc=on}}, every day 2.5 [[exabyte]]s (2.17×2<sup>60</sup> bytes) of data are generated.<ref>{{cite web |url= http://www.ibm.com/big-data/us/en/ |title= IBM What is big data? – Bringing big data to the enterprise |publisher= ibm.com |access-date= 26 August 2013 |archive-date= 24 August 2013 |archive-url= https://web.archive.org/web/20130824213031/http://www.ibm.com/big-data/us/en/ |url-status= live }}</ref> Based on an [[International Data Corporation|IDC]] report prediction, the global data volume was predicted to grow exponentially from 4.4 [[zettabyte]]s to 44 zettabytes between 2013 and 2020. By 2025, IDC predicts there will be 163 zettabytes of data.<ref>{{Cite web|url=https://www.seagate.com/files/www-content/our-story/trends/files/Seagate-WP-DataAge2025-March-2017.pdf|title=Data Age 2025: The Evolution of Data to Life-Critical|last1=Reinsel|first1=David|last2=Gantz|first2=John|date=13 April 2017|website=seagate.com|publisher=[[International Data Corporation]]|location=Framingham, MA, US|access-date=2 November 2017|last3=Rydning|first3=John|archive-date=8 December 2017|archive-url=https://web.archive.org/web/20171208010603/https://www.seagate.com/files/www-content/our-story/trends/files/Seagate-WP-DataAge2025-March-2017.pdf|url-status=live}}</ref> According to IDC, global spending on big data and business analytics (BDA) solutions is estimated to reach $215.7 billion in 2021.<ref>{{cite web | url=https://www.idc.com/getdoc.jsp?containerId=prUS48165721 | title=Global Spending on Big Data and Analytics Solutions Will Reach $215.7 Billion in 2021, According to a New IDC Spending Guide | access-date=31 July 2022 | archive-date=23 July 2022 | archive-url=https://web.archive.org/web/20220723034825/https://www.idc.com/getdoc.jsp?containerId=prUS48165721 | url-status=dead }}</ref><ref>{{cite web | url=https://www.statista.com/statistics/551501/worldwide-big-data-business-analytics-revenue/ | title=Big data and business analytics revenue 2022 }}</ref> While [[Statista]] report, the global big data market is forecasted to grow to $103 billion by 2027.<ref>{{cite web | url=https://www.statista.com/statistics/254266/global-big-data-market-forecast/ | title=Global big data industry market size 2011-2027 }}</ref> In 2011 [[McKinsey & Company]] reported, if US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year.<ref name="McKinsey2011"/> In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data.<ref name="McKinsey2011"/> And users of services enabled by personal-location data could capture $600 billion in consumer surplus.<ref name="McKinsey2011">[https://www.mckinsey.com/~/media/mckinsey/business%20functions/mckinsey%20digital/our%20insights/big%20data%20the%20next%20frontier%20for%20innovation/mgi_big_data_exec_summary.pdf Big data: The next frontier for innovation, competition, and productivity] McKinsey Global Institute May 2011</ref> One question for large enterprises is determining who should own big-data initiatives that affect the entire organization.<ref>Oracle and FSN, [http://www.fsn.co.uk/channel_bi_bpm_cpm/mastering_big_data_cfo_strategies_to_transform_insight_into_opportunity "Mastering Big Data: CFO Strategies to Transform Insight into Opportunity"] {{Webarchive|url=https://web.archive.org/web/20130804062518/http://www.fsn.co.uk/channel_bi_bpm_cpm/mastering_big_data_cfo_strategies_to_transform_insight_into_opportunity |date=4 August 2013 }}, December 2012</ref>

[[Relational database management system]]s and desktop statistical software packages used to visualize data often have difficulty processing and analyzing big data. The processing and analysis of big data may require "massively parallel software running on tens, hundreds, or even thousands of servers".<ref>{{cite web |author= Jacobs, A. |title= The Pathologies of Big Data |date= 6 July 2009 |work= ACMQueue |url= http://queue.acm.org/detail.cfm?id=1563874 |access-date= 21 April 2010 |archive-date= 8 December 2015 |archive-url= https://web.archive.org/web/20151208072827/http://queue.acm.org/detail.cfm?id=1563874 |url-status= live }}</ref> What qualifies as "big data" varies depending on the capabilities of those analyzing it and their tools. Furthermore, expanding capabilities make big data a moving target. "For some organizations, facing hundreds of [[gigabyte]]s of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."<ref>{{cite journal|last1= Magoulas |first1= Roger |last2=Lorica|first2=Ben|date=February 2009|title=Introduction to Big Data|url=https://academics.uccs.edu/~ooluwada/courses/datamining/ExtraReading/BigData|journal=Release 2.0 |location= Sebastopol, CA|publisher=O'Reilly Media|issue=11|access-date=26 February 2021|archive-date=2 November 2021|archive-url=https://web.archive.org/web/20211102185002/https://academics.uccs.edu/~ooluwada/courses/datamining/ExtraReading/BigData|url-status=live}}</ref>

==Definition==
The term ''big data'' has been in use since the 1990s, with some giving credit to [[John Mashey]] for popularizing the term.<ref>{{Cite web |title= Big Data… and the Next Wave of InfraStress |author= John R. Mashey |date= 25 April 1998 |publisher= Usenix |work= Slides from invited talk |url= http://static.usenix.org/event/usenix99/invited_talks/mashey.pdf |access-date= 28 September 2016 |archive-date= 12 October 2016 |archive-url= https://web.archive.org/web/20161012095743/http://static.usenix.org/event/usenix99/invited_talks/mashey.pdf |url-status= live }}</ref><ref>{{cite news |title=The Origins of 'Big Data': An Etymological Detective Story |author=Steve Lohr |date=1 February 2013 |url=http://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story/ |work=[[The New York Times]] |access-date=28 September 2016 |archive-date=6 March 2016 |archive-url=https://web.archive.org/web/20160306042217/http://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story/ |url-status=live }}</ref>  Big data usually includes data sets with sizes beyond the ability of commonly used software tools to [[data acquisition|capture]], [[data curation|curate]], manage, and process data within a tolerable elapsed time.{{Sfn | Snijders | Matzat | Reips | 2012}}{{Page needed|date=December 2023}}  Big data philosophy encompasses unstructured, semi-structured and structured data; however, the main focus is on unstructured data.<ref name="Springer 2017">{{cite book |chapter=Towards Differentiating Business Intelligence, Big Data, Data Analytics and Knowledge Discovery |last1=Dedić |first1=N. |title= Innovations in Enterprise Information Systems Management and Engineering |last2=Stanier |first2=C. |issn=1865-1356 |oclc=909580101 |publisher=Springer International Publishing |location= Berlin; Heidelberg |year=2017 |volume=285 |pages= 114–22 |doi=10.1007/978-3-319-58801-8_10 |series=Lecture Notes in Business Information Processing |isbn=978-3-319-58800-1 |chapter-url= http://eprints.staffs.ac.uk/3551/1/Towards%20Differentiating%20Business%20Intelligence%20Big%20Data%20Data%20Analytics%20and%20Knowldge%20Discovery.docx |access-date=7 September 2019 |archive-date= 27 November 2020 |archive-url= https://web.archive.org/web/20201127065236/http://eprints.staffs.ac.uk/3551/1/Towards%20Differentiating%20Business%20Intelligence%20Big%20Data%20Data%20Analytics%20and%20Knowldge%20Discovery.docx |url-status=live }}</ref> Big data "size" is a constantly moving target; {{As of|2012|lc=on}} ranging from a few dozen terabytes to many [[zettabyte]]s of data.<ref name="Everts">{{cite magazine |last1=Everts |first1=Sarah |title=Information Overload |magazine= [[Distillations (magazine)|Distillations]] |date=2016 |volume=2 |issue=2 |pages=26–33 |url=https://www.sciencehistory.org/distillations/magazine/information-overload |access-date= 22 March 2018 |archive-date=3 April 2019 |archive-url=https://web.archive.org/web/20190403133511/https://www.sciencehistory.org/distillations/magazine/information-overload |url-status=live }}</ref>  Big data requires a set of techniques and technologies with new forms of [[data integration|integration]] to reveal insights from [[Data set|data-sets]] that are diverse, complex, and of a massive scale.<ref>{{cite journal | last1 = Ibrahim | last2 = Targio Hashem | first2 = Abaker | last3 = Yaqoob | first3 = Ibrar | last4 = Badrul Anuar | first4 = Nor | last5 = Mokhtar | first5 = Salimah | last6 = Gani | first6 = Abdullah | last7 = Ullah Khan | first7 = Samee | year = 2015 | title = big data" on cloud computing: Review and open research issues | journal = Information Systems | volume = 47 | pages = 98–115 | doi = 10.1016/j.is.2014.07.006 | s2cid = 205488005 }}</ref>

"Volume", "variety", "velocity", and various other "Vs" are added by some organizations to describe it, a revision challenged by some industry authorities.<ref>{{cite magazine|last=Grimes|first=Seth|title=Big Data: Avoid 'Wanna V' Confusion|url=http://www.informationweek.com/big-data/big-data-analytics/big-data-avoid-wanna-v-confusion/d/d-id/1111077|magazine=[[InformationWeek]]|access-date=5 January 2016|archive-date=23 December 2015|archive-url=https://web.archive.org/web/20151223185813/http://www.informationweek.com/big-data/big-data-analytics/big-data-avoid-wanna-v-confusion/d/d-id/1111077|url-status=live}}</ref> The Vs of big data were often referred to as the "three Vs", "four Vs", and "five Vs". They represented the qualities of big data in volume, variety, velocity, veracity, and value.<ref name=":0">{{Cite web|date=2016-09-17|title=The 5 V's of big data|url=https://www.ibm.com/blogs/watson-health/the-5-vs-of-big-data/|access-date=2021-01-20|website=Watson Health Perspectives|language=en-US|archive-date=18 January 2021|archive-url=https://web.archive.org/web/20210118085939/https://www.ibm.com/blogs/watson-health/the-5-vs-of-big-data/|url-status=live}}</ref> Variability is often included as an additional quality of big data.

A 2018 definition states "Big data is where [[parallel computing]] tools are needed to handle data", and notes, "This represents a distinct and clearly defined change in the computer science used, via parallel programming theories, and losses of some of the guarantees and capabilities made by [[Relational database|Codd's relational model]]."<ref>{{Cite book|last=Fox|first=Charles|date=25 March 2018|title=Data Science for Transport|url=https://www.springer.com/us/book/9783319729527|publisher=Springer|isbn=9783319729527|series=Springer Textbooks in Earth Sciences, Geography and Environment|access-date=31 March 2018|archive-date=1 April 2018|archive-url=https://web.archive.org/web/20180401144520/https://www.springer.com/us/book/9783319729527|url-status=live}}</ref>

In a comparative study of big datasets, [[Rob Kitchin|Kitchin]] and McArdle found that none of the commonly considered characteristics of big data appear consistently across all of the analyzed cases.<ref>{{cite journal | last1 = Kitchin | first1 = Rob | last2 = McArdle | first2 = Gavin | year = 2016 | title = What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets | journal = Big Data & Society | volume = 3 | pages = 1–10 | doi = 10.1177/2053951716631130 | s2cid = 55539845 | doi-access = free }}</ref> For this reason, other studies identified the redefinition of power dynamics in knowledge discovery as the defining trait.<ref>{{cite journal | last1 = Balazka | first1 = Dominik | last2 = Rodighiero | first2 = Dario | year = 2020 | title = Big Data and the Little Big Bang: An Epistemological (R)evolution | journal = Frontiers in Big Data | volume = 3 | page = 31 | doi = 10.3389/fdata.2020.00031 | pmid = 33693404 | pmc = 7931920 | hdl = 1721.1/128865 | hdl-access = free | doi-access = free }}</ref> Instead of focusing on the intrinsic characteristics of big data, this alternative perspective pushes forward a relational understanding of the object claiming that what matters is the way in which data is collected, stored, made available and analyzed.

=== Big data vs. business intelligence ===
The growing maturity of the concept more starkly delineates the difference between "big data" and "[[business intelligence]]":<ref>{{cite web|url=http://www.bigdataparis.com/presentation/mercredi/PDelort.pdf?PHPSESSID=tv7k70pcr3egpi2r6fi3qbjtj6#page=4|format=PDF|title=avec focalisation sur Big Data & Analytique|website=Bigdataparis.com|access-date=8 October 2017|archive-date=25 February 2021|archive-url=https://web.archive.org/web/20210225014647/https://www.bigdataparis.com/presentation/mercredi/PDelort.pdf?PHPSESSID=tv7k70pcr3egpi2r6fi3qbjtj6#page=4|url-status=dead}}</ref>
* Business intelligence uses applied mathematics tools and [[descriptive statistics]] with data with high information density to measure things, detect trends, etc.
* Big data uses mathematical analysis, optimization, [[inductive statistics]], and concepts from [[nonlinear system identification]]<ref name="SAB1">Billings S.A. "Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains". Wiley, 2013</ref> to infer laws (regressions, nonlinear relationships, and causal effects) from large sets of data with low information density<ref>{{cite web|url=http://www.andsi.fr/tag/dsi-big-data/|title=le Blog ANDSI » DSI Big Data|website=Andsi.fr|access-date=8 October 2017|archive-date=10 October 2017|archive-url=https://web.archive.org/web/20171010205333/http://www.andsi.fr/tag/dsi-big-data/|url-status=live}}</ref> to reveal relationships and dependencies, or to perform predictions of outcomes and behaviors.<ref name="SAB1" /><ref>{{cite web|url=http://lecercle.lesechos.fr/entrepreneur/tendances-innovation/221169222/big-data-low-density-data-faible-densite-information-com|title=Les Echos – Big Data car Low-Density Data ? La faible densité en information comme facteur discriminant – Archives|author=Les Echos|date=3 April 2013|website=Lesechos.fr|access-date=8 October 2017|archive-date=30 April 2014|archive-url=https://web.archive.org/web/20140430153257/http://lecercle.lesechos.fr/entrepreneur/tendances-innovation/221169222/big-data-low-density-data-faible-densite-information-com|url-status=live}}</ref>{{promotional source|date=December 2018}}

==Characteristics==
[[File:Big Data.png|thumb|This image shows the growth of big data's primary characteristics of volume, velocity, and variety.]]
Big data can be described by the following characteristics:

; Volume: The quantity of generated and stored data. The size of the data determines the value and potential insight, and whether it can be considered big data or not. The size of big data is usually larger than terabytes and petabytes.<ref>{{cite book |last1=Sagiroglu |first1=Seref |title=2013 International Conference on Collaboration Technologies and Systems (CTS) |chapter=Big data: A review |date=2013 |pages=42–47 |doi=10.1109/CTS.2013.6567202|isbn=978-1-4673-6404-1 |s2cid=5724608 }}</ref>

; Variety: The type and nature of the data. Earlier technologies like RDBMSs were capable to handle structured data efficiently and effectively. However, the change in type and nature from structured to semi-structured or unstructured challenged the existing tools and technologies. Big data technologies evolved with the prime intention to capture, store, and process the semi-structured and unstructured (variety) data generated with high speed (velocity), and huge in size (volume). Later, these tools and technologies were explored and used for handling structured data also but preferable for storage. Eventually, the processing of structured data was still kept as optional, either using big data or traditional RDBMSs. This helps in analyzing data towards effective usage of the hidden insights exposed from the data collected via social media, log files, sensors, etc. Big data draws from text, images, audio, video; plus it completes missing pieces through [[data fusion]].

; Velocity: The speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development. Big data is often available in real-time. Compared to [[small data]], big data is produced more continually. Two kinds of velocity related to big data are the frequency of generation and the frequency of handling, recording, and publishing.<ref>{{cite journal |last1=Kitchin |first1=Rob |last2=McArdle |first2=Gavin |title=What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets |journal=Big Data & Society |date=17 February 2016 |volume=3 |issue=1 |pages=205395171663113 |doi=10.1177/2053951716631130|doi-access=free }}</ref>

;Veracity: The truthfulness or reliability of the data, which refers to the data quality and the data value.<ref>{{Cite journal|last1=Onay|first1=Ceylan|last2=Öztürk|first2=Elif|date=2018|title=A review of credit scoring research in the age of Big Data|journal=Journal of Financial Regulation and Compliance|volume=26|issue=3|pages=382–405|doi=10.1108/JFRC-06-2017-0054|s2cid=158895306}}</ref> Big data must not only be large in size, but also must be reliable in order to achieve value in the analysis of it. The [[data quality]] of captured data can vary greatly, affecting an accurate analysis.<ref>[https://web.archive.org/web/20180731105912/https://spotlessdata.com/blog/big-datas-fourth-v Big Data's Fourth V]</ref>

; Value: The worth in information that can be achieved by the processing and analysis of large datasets. Value also can be measured by an assessment of the other qualities of big data.<ref>{{Cite web|title=Measuring the Business Value of Big Data {{!}} IBM Big Data & Analytics Hub|url=https://www.ibmbigdatahub.com/blog/measuring-business-value-big-data|access-date=2021-01-20|website=www.ibmbigdatahub.com|archive-date=28 January 2021|archive-url=https://web.archive.org/web/20210128191754/https://www.ibmbigdatahub.com/blog/measuring-business-value-big-data|url-status=live}}</ref> Value may also represent the profitability of information that is retrieved from the analysis of big data.

; Variability: The characteristic of the changing formats, structure, or sources of big data. Big data can include structured, unstructured, or combinations of structured and unstructured data. Big data analysis may integrate raw data from multiple sources. The processing of raw data may also involve transformations of unstructured data to structured data.

Other possible characteristics of big data are:<ref>{{Cite journal|last1=Kitchin|first1=Rob|last2=McArdle|first2=Gavin|date=5 January 2016|title=What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets|journal=Big Data & Society|language=en|volume=3|issue=1|pages=205395171663113|doi=10.1177/2053951716631130|issn=2053-9517|doi-access=free}}</ref>

;Exhaustive: Whether the entire system (i.e., <math display="inline">n</math>=all) is captured or recorded or not. Big data may or may not include all the available data from sources.

; Fine-grained and uniquely lexical: Respectively, the proportion of specific data of each element per element collected and if the element and its characteristics are properly indexed or identified.

; Relational: If the data collected contains common fields that would enable a conjoining, or meta-analysis, of different data sets.

; Extensional: If new fields in each element of the data collected can be added or changed easily.

; Scalability: If the size of the big data storage system can expand rapidly.

==Architecture==
Big data repositories have existed in many forms, often built by corporations with a special need. Commercial vendors historically offered parallel database management systems for big data beginning in the 1990s. For many years, WinterCorp published the largest database report.<ref>{{cite web |url=http://www.eweek.com/database/survey-biggest-databases-approach-30-terabytes|title=Survey: Biggest Databases Approach 30 Terabytes|website=Eweek.com|date=8 November 2003|access-date=8 October 2017}}</ref>{{promotional source|date=December 2018}}

[[Teradata]] Corporation in 1984 marketed the parallel processing [[DBC 1012]] system. Teradata systems were the first to store and analyze 1 terabyte of data in 1992. Hard disk drives were 2.5 GB in 1991 so the definition of big data continuously evolves. Teradata installed the first petabyte class RDBMS based system in 2007. {{as of|2017}}, there are a few dozen petabyte class Teradata relational databases installed, the largest of which exceeds 50 PB. Systems up until 2008 were 100% structured relational data. Since then, Teradata has added semi structured data types including [[XML]], [[JSON]], and [[Apache Avro|Avro]].

In 2000, Seisint Inc. (now [[LexisNexis Risk Solutions]]) developed a [[C++]]-based distributed platform for data processing and querying known as the [[HPCC Systems]] platform. This system automatically partitions, distributes, stores and delivers structured, semi-structured, and unstructured data across multiple commodity servers. Users can write data processing pipelines and queries in a declarative dataflow programming language called ECL. Data analysts working in ECL are not required to define data schemas upfront and can rather focus on the particular problem at hand, reshaping data in the best possible manner as they develop the solution. In 2004, LexisNexis acquired Seisint Inc.<ref>{{cite news|url=https://www.washingtonpost.com/wp-dyn/articles/A50577-2004Jul14.html|title=LexisNexis To Buy Seisint For $775 Million|newspaper=[[The Washington Post]]|access-date=15 July 2004|archive-date=24 July 2008|archive-url=https://web.archive.org/web/20080724195717/http://www.washingtonpost.com/wp-dyn/articles/A50577-2004Jul14.html|url-status=live}}</ref> and their high-speed parallel processing platform and successfully used this platform to integrate the data systems of Choicepoint Inc. when they acquired that company in 2008.<ref>{{Cite news |url=https://www.washingtonpost.com/wp-dyn/content/article/2008/02/21/AR2008022100809.html |title=The Washington Post |newspaper=[[The Washington Post]] |access-date=24 August 2017 |archive-date=19 October 2016 |archive-url=https://web.archive.org/web/20161019022657/http://www.washingtonpost.com/wp-dyn/content/article/2008/02/21/AR2008022100809.html |url-status=live }}</ref> In 2011, the HPCC systems platform was open-sourced under the Apache v2.0 License.

[[CERN]] and other physics experiments have collected big data sets for many decades, usually analyzed via [[high-throughput computing]] rather than the map-reduce architectures usually meant by the current "big data" movement.

In 2004, [[Google]] published a paper on a process called [[MapReduce]] that uses a similar architecture. The MapReduce concept provides a parallel processing model, and an associated implementation was released to process huge amounts of data. With MapReduce, queries are split and distributed across parallel nodes and processed in parallel (the "map" step). The results are then gathered and delivered (the "reduce" step). The framework was very successful,<ref>Bertolucci, Jeff [http://www.informationweek.com/software/hadoop-from-experiment-to-leading-big-data-platform/d/d-id/1110491 "Hadoop: From Experiment To Leading Big Data Platform"] {{Webarchive|url=https://web.archive.org/web/20201123153924/https://www.informationweek.com/software/hadoop-from-experiment-to-leading-big-data-platform/d/d-id/1110491 |date=23 November 2020 }}, "Information Week", 2013. Retrieved on 14 November 2013.</ref> so others wanted to replicate the algorithm. Therefore, an [[implementation]] of the MapReduce framework was adopted by an Apache open-source project named "[[Apache Hadoop|Hadoop]]".<ref>Webster, John. [http://research.google.com/archive/mapreduce-osdi04.pdf "MapReduce: Simplified Data Processing on Large Clusters"] {{Webarchive|url=https://web.archive.org/web/20091214180106/http://research.google.com/archive/mapreduce-osdi04.pdf |date=14 December 2009 }}, "Search Storage", 2004. Retrieved on 25 March 2013.</ref> [[Apache Spark]] was developed in 2012 in response to limitations in the MapReduce paradigm, as it adds [[in-memory processing]] and the ability to set up many operations (not just map followed by reducing).

[[MIKE2.0 Methodology|MIKE2.0]] is an open approach to information management that acknowledges the need for revisions due to big data implications identified in an article titled "Big Data Solution Offering".<ref>{{cite web| url=http://mike2.openmethodology.org/wiki/Big_Data_Solution_Offering| title=Big Data Solution Offering| publisher=MIKE2.0| access-date=8 December 2013| archive-date=16 March 2013| archive-url=https://web.archive.org/web/20130316050909/http://mike2.openmethodology.org/wiki/Big_Data_Solution_Offering| url-status=live}}</ref> The methodology addresses handling big data in terms of useful [[permutation]]s of data sources, [[complexity]] in interrelationships, and difficulty in deleting (or modifying) individual records.<ref>{{cite web|url=http://mike2.openmethodology.org/wiki/Big_Data_Definition|title=Big Data Definition|publisher=MIKE2.0|access-date=9 March 2013|archive-date=25 September 2018|archive-url=https://web.archive.org/web/20180925030939/http://mike2.openmethodology.org/wiki/Big_Data_Definition|url-status=live}}</ref>

Studies in 2012 showed that a multiple-layer architecture was one option to address the issues that big data presents. A [[List of file systems#Distributed parallel file systems|distributed parallel]] architecture distributes data across multiple servers; these parallel execution environments can dramatically improve data processing speeds. This type of architecture inserts data into a parallel DBMS, which implements the use of MapReduce and Hadoop frameworks. This type of framework looks to make the processing power transparent to the end-user by using a front-end application server.<ref>{{cite journal|last=Boja|first=C|author2=Pocovnicu, A |author3=Bătăgan, L. |title=Distributed Parallel Architecture for Big Data|journal=Informatica Economica|year=2012 |volume=16|issue=2| pages=116–127}}</ref>

The [[data lake]] allows an organization to shift its focus from centralized control to a shared model to respond to the changing dynamics of information management. This enables quick segregation of data into the data lake, thereby reducing the overhead time.<ref>{{cite web|url=http://www.hcltech.com/sites/default/files/solving_key_businesschallenges_with_big_data_lake_0.pdf|title=Solving Key Business Challenges With a Big Data Lake|date=August 2014|website=Hcltech.com|access-date=8 October 2017|archive-date=3 July 2017|archive-url=https://web.archive.org/web/20170703074658/https://www.hcltech.com/sites/default/files/solving_key_businesschallenges_with_big_data_lake_0.pdf|url-status=live}}</ref><ref>{{cite web | url= https://secplab.ppgia.pucpr.br/files/papers/2015-0.pdf | title= Method for testing the fault tolerance of MapReduce frameworks | publisher= Computer Networks | year= 2015 | access-date= 13 April 2016 | archive-date= 22 July 2016 | archive-url= https://web.archive.org/web/20160722133706/https://secplab.ppgia.pucpr.br/files/papers/2015-0.pdf | url-status= live }}</ref>

==Technologies==
A 2011 [[McKinsey & Company|McKinsey Global Institute]] report characterizes the main components and ecosystem of big data as follows:<ref name="McKinsey">{{cite web | last1 = Manyika | first1 = James | first2 = Michael | last2 = Chui | first3 = Jaques | last3 = Bughin | first4 = Brad | last4 = Brown | first5 = Richard | last5 = Dobbs | first6 = Charles | last6 = Roxburgh | first7 = Angela Hung | last7 = Byers | title = Big Data: The next frontier for innovation, competition, and productivity | publisher = McKinsey Global Institute | date = May 2011 | url = https://www.mckinsey.com/~/media/mckinsey/business%20functions/mckinsey%20digital/our%20insights/big%20data%20the%20next%20frontier%20for%20innovation/mgi_big_data_full_report.pdf | access-date = 22 May 2021 | archive-date = 25 July 2021 | archive-url = https://web.archive.org/web/20210725123303/https://www.mckinsey.com/~/media/mckinsey/business%20functions/mckinsey%20digital/our%20insights/big%20data%20the%20next%20frontier%20for%20innovation/mgi_big_data_full_report.pdf | url-status = live }}</ref>
* Techniques for analyzing data, such as [[A/B testing]], [[machine learning]], and [[natural language processing]]
* Big data technologies, like [[business intelligence]], [[cloud computing]], and [[database]]s
* Visualization, such as charts, graphs, and other displays of the data

Multidimensional big data can also be represented as [[OLAP]] data cubes or, mathematically, [[tensor]]s. [[Array DBMS|Array database systems]] have set out to provide storage and high-level query support on this data type.
Additional technologies being applied to big data include efficient tensor-based computation,<ref>{{cite web|title=Future Directions in Tensor-Based Computation and Modeling|date=May 2009|url=http://www.cs.cornell.edu/cv/tenwork/finalreport.pdf|access-date=4 January 2013|archive-date=17 April 2018|archive-url=https://web.archive.org/web/20180417231223/http://www.cs.cornell.edu/CV/TenWork/FinalReport.pdf|url-status=live}}</ref> such as [[multilinear subspace learning]],<ref name="MSLsurvey">{{cite journal | first1 = Haiping | last1 = Lu | first2 = K.N. | last2 = Plataniotis | first3 = A.N. | last3 = Venetsanopoulos | url = http://www.dsp.utoronto.ca/~haiping/Publication/SurveyMSL_PR2011.pdf | title = A Survey of Multilinear Subspace Learning for Tensor Data | journal = Pattern Recognition | volume = 44 | number = 7 | pages = 1540–1551 | year = 2011 | doi = 10.1016/j.patcog.2011.01.004 | bibcode = 2011PatRe..44.1540L | access-date = 21 January 2013 | archive-date = 10 July 2019 | archive-url = https://web.archive.org/web/20190710225429/http://www.dsp.utoronto.ca/~haiping/Publication/SurveyMSL_PR2011.pdf | url-status = live }}</ref> massively parallel-processing ([[Massive parallel processing|MPP]]) databases, [[search-based application]]s, [[data mining]],<ref>{{cite book|last1=Pllana|first1=Sabri|title=2011 14th International Conference on Network-Based Information Systems|pages=341–348|last2=Janciak|first2=Ivan|last3=Brezany|first3=Peter|last4=Wöhrer|first4=Alexander|chapter=A Survey of the State of the Art in Data Mining and Integration Query Languages |publisher=IEEE Computer Society|bibcode=2016arXiv160301113P|year=2016|arxiv=1603.01113|doi=10.1109/NBiS.2011.58|isbn=978-1-4577-0789-6|s2cid=9285984}}</ref> [[distributed file system]]s, distributed cache (e.g., [[burst buffer]] and [[Memcached]]), [[distributed database]]s, [[cloud computing|cloud]] and [[supercomputer|HPC-based]] infrastructure (applications, storage and computing resources),<ref>{{cite book|chapter=Characterization and Optimization of Memory-Resident MapReduce on HPC Systems|publisher=IEEE|date=October 2014|doi=10.1109/IPDPS.2014.87|title=2014 IEEE 28th International Parallel and Distributed Processing Symposium|pages=799–808|last1=Wang|first1=Yandong|last2=Goldstone|first2=Robin|last3=Yu|first3=Weikuan|last4=Wang|first4=Teng|s2cid=11157612|isbn=978-1-4799-3800-1}}</ref> and the Internet.{{Citation needed|date=September 2011}} Although, many approaches and technologies have been developed, it still remains difficult to carry out machine learning with big data.<ref>{{Cite journal|last1=L'Heureux|first1=A.|last2=Grolinger|first2=K.|last3=Elyamany|first3=H. F.|last4=Capretz|first4=M. A. M.|date=2017|title=Machine Learning With Big Data: Challenges and Approaches|journal=IEEE Access|volume=5|pages=7776–7797|doi=10.1109/ACCESS.2017.2696365|bibcode=2017IEEEA...5.7776L |issn=2169-3536|doi-access=free}}</ref>

Some [[Massive parallel processing|MPP]] relational databases have the ability to store and manage petabytes of data. Implicit is the ability to load, monitor, back up, and optimize the use of the large data tables in the [[RDBMS]].<ref>{{cite web |author=Monash, Curt |title=eBay's two enormous data warehouses |date=30 April 2009 |url=http://www.dbms2.com/2009/04/30/ebays-two-enormous-data-warehouses/ |access-date=11 November 2010 |archive-date=31 March 2019 |archive-url=https://web.archive.org/web/20190331110659/http://www.dbms2.com/2009/04/30/ebays-two-enormous-data-warehouses/ |url-status=live }}<br />{{cite web |author=Monash, Curt |title=eBay followup&nbsp;– Greenplum out, Teradata > 10 petabytes, Hadoop has some value, and more |date=6 October 2010 |url=http://www.dbms2.com/2010/10/06/ebay-followup-greenplum-out-teradata-10-petabytes-hadoop-has-some-value-and-more/ |access-date=11 November 2010 |archive-date=31 March 2019 |archive-url=https://web.archive.org/web/20190331133707/http://www.dbms2.com/2010/10/06/ebay-followup-greenplum-out-teradata-10-petabytes-hadoop-has-some-value-and-more/ |url-status=live }}</ref>{{promotional source|date=December 2018}}

[[DARPA]]'s [[Topological Data Analysis]] program seeks the fundamental structure of massive data sets and in 2008 the technology went public with the launch of a company called "Ayasdi".<ref>{{cite web|url=http://www.ayasdi.com/resources/|title=Resources on how Topological Data Analysis is used to analyze big data|publisher=Ayasdi|access-date=5 March 2013|archive-date=3 March 2013|archive-url=https://web.archive.org/web/20130303225426/http://www.ayasdi.com/resources/|url-status=live}}</ref>{{third-party inline|date=December 2018}}

The practitioners of big data analytics processes are generally hostile to slower shared storage,<ref>{{cite web |title=Storage area networks need not apply |author=CNET News |work=CNET |date=1 April 2011 |url=http://news.cnet.com/8301-21546_3-20049693-10253464.html |access-date=17 April 2013 |archive-date=18 October 2013 |archive-url=https://web.archive.org/web/20131018042918/http://news.cnet.com/8301-21546_3-20049693-10253464.html |url-status=live }}</ref> preferring direct-attached storage ([[Direct-attached storage|DAS]]) in its various forms from solid state drive ([[SSD]]) to high capacity [[Serial ATA|SATA]] disk buried inside parallel processing nodes. The perception of shared storage architectures—[[storage area network]] (SAN) and [[network-attached storage]] (NAS)— is that they are relatively slow, complex, and expensive. These qualities are not consistent with big data analytics systems that thrive on system performance, commodity infrastructure, and low cost.

Real or near-real-time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in direct-attached memory or disk is good—data on memory or disk at the other end of an [[Fiber connector|FC]] [[Storage area network|SAN]] connection is not. The cost of an [[Storage area network|SAN]] at the scale needed for analytics applications is much higher than other storage techniques.

==Applications==
[[File:2013-09-11 Bus wrapped with SAP Big Data parked outside IDF13 (9730051783).jpg|thumb|Bus wrapped with [[SAP AG|SAP]] big data parked outside [[Intel Developer Forum|IDF13]]]]
Big data has increased the demand of information management specialists so much so that [[Software AG]], [[Oracle Corporation]], [[IBM]], [[Microsoft]], [[SAP AG|SAP]], [[EMC Corporation|EMC]], [[Hewlett-Packard|HP]], and [[Dell]] have spent more than $15&nbsp;billion on software firms specializing in data management and analytics. In 2010, this industry was worth more than $100&nbsp;billion and was growing at almost 10&nbsp;percent a year, about twice as fast as the software business as a whole.{{r|Economist}}

Developed economies increasingly use data-intensive technologies. There are 4.6&nbsp;billion mobile-phone subscriptions worldwide, and between 1&nbsp;billion and 2&nbsp;billion people accessing the internet.{{r|Economist}} Between 1990 and 2005, more than 1&nbsp;billion people worldwide entered the middle class, which means more people became more literate, which in turn led to information growth. The world's effective capacity to exchange information through telecommunication networks was 281 [[petabytes]] in 1986, 471 petabytes in 1993, 2.2 exabytes in 2000, 65 [[exabytes]] in 2007<ref name="martinhilbert.net"/> and predictions put the amount of internet traffic at 667 exabytes annually by 2014.{{r|Economist}} According to one estimate, one-third of the globally stored information is in the form of alphanumeric text and still image data,<ref name="HilbertContent">{{cite journal | title= What is the Content of the World's Technologically Mediated Information and Communication Capacity: How Much Text, Image, Audio, and Video? | doi= 10.1080/01972243.2013.873748 | volume= 30 | issue= 2 | journal= The Information Society | pages= 127–143 | year= 2014 | last1= Hilbert | first1= Martin | s2cid= 45759014 | url= https://escholarship.org/uc/item/87w5f6wb | access-date= 12 July 2019 | archive-date= 24 June 2020 | archive-url= https://web.archive.org/web/20200624012730/https://escholarship.org/uc/item/87w5f6wb | url-status= live }}</ref> which is the format most useful for most big data applications. This also shows the potential of yet unused data (i.e. in the form of video and audio content).

While many vendors offer off-the-shelf products for big data, experts promote the development of in-house custom-tailored systems if the company has sufficient technical capabilities.<ref>{{cite web |url=http://www.kdnuggets.com/2014/07/interview-amy-gershkoff-ebay-in-house-BI-tools.html |title=Interview: Amy Gershkoff, Director of Customer Analytics & Insights, eBay on How to Design Custom In-House BI Tools |last1=Rajpurohit |first1=Anmol |date=11 July 2014 |website=KDnuggets |access-date=14 July 2014 |quote=Generally, I find that off-the-shelf business intelligence tools do not meet the needs of clients who want to derive custom insights from their data. Therefore, for medium-to-large organizations with access to strong technical talent, I usually recommend building custom, in-house solutions. |archive-date=14 July 2014 |archive-url=https://web.archive.org/web/20140714145352/http://www.kdnuggets.com/2014/07/interview-amy-gershkoff-ebay-in-house-BI-tools.html |url-status=live }}</ref>

===Government===
{{See also|Government by algorithm|}}
{{More citations needed|section|date=September 2023}}
The use and adoption of big data within governmental processes allows efficiencies in terms of cost, productivity, and innovation,<ref>{{cite magazine |url=http://www.computerworld.com/article/2472667/government-it/the-government-and-big-data--use--problems-and-potential.html |title=The Government and big data: Use, problems and potential |date=21 March 2012 |magazine=[[Computerworld]] |access-date=12 September 2016 |archive-date=15 September 2016 |archive-url=https://web.archive.org/web/20160915110613/http://www.computerworld.com/article/2472667/government-it/the-government-and-big-data--use--problems-and-potential.html |url-status=live }}</ref> but comes with flaws. Data analysis often requires multiple parts of government (central and local) to work in collaboration and create new and innovative processes to deliver the desired outcome. A common government organization that makes use of big data is the National Security Administration ([[National Security Agency|NSA]]), which monitors the activities of the Internet constantly in search for potential patterns of suspicious or illegal activities their system may pick up.

[[Civil registration and vital statistics]] (CRVS) collects all certificates status from birth to death. CRVS is a source of big data for governments.

===International development===
Research on the effective usage of information and communication technologies for development (also known as "ICT4D") suggests that big data technology can make important contributions but also present unique challenges to [[international development]].<ref>{{cite web | url=http://www.unglobalpulse.org/projects/BigDataforDevelopment | title=White Paper: Big Data for Development: Opportunities & Challenges | year = 2012 | publisher =United Nations | work = Global Pulse | access-date=13 April 2016 | archive-date=1 June 2020 | archive-url= https://web.archive.org/web/20200601215207/https://www.unglobalpulse.org/document/big-data-for-development-opportunities-and-challenges-white-paper// | url-status=live }}</ref><ref>{{cite web | title= Big Data, Big Impact: New Possibilities for International Development| publisher =World Economic Forum & Vital Wave Consulting | access-date=24 August 2012| url= http://www.weforum.org/reports/big-data-big-impact-new-possibilities-international-development| archive-date=1 June 2020| archive-url=https://web.archive.org/web/20200601215225/https://www.weforum.org/reports/big-data-big-impact-new-possibilities-international-development/| url-status=live}}</ref> Advancements in big data analysis offer cost-effective opportunities to improve decision-making in critical development areas such as health care, employment, [[economic productivity]], crime, security, and [[natural disaster]] and resource management.{{Sfn | Hilbert | 2016}}{{Page needed |date=December 2023}}<ref>{{cite web|url=http://blogs.worldbank.org/ic4d/four-ways-to-talk-about-big-data/|title=Elena Kvochko, Four Ways To talk About Big Data (Information Communication Technologies for Development Series)|publisher=worldbank.org|access-date=30 May 2012|date=4 December 2012|archive-date=15 December 2012|archive-url=https://web.archive.org/web/20121215132444/http://blogs.worldbank.org/ic4d/four-ways-to-talk-about-big-data|url-status=live}}</ref><ref>{{cite web| title=Daniele Medri: Big Data & Business: An on-going revolution | url=http://www.statisticsviews.com/details/feature/5393251/Big-Data--Business-An-on-going-revolution.html| publisher=Statistics Views| date=21 October 2013| access-date=21 June 2015| archive-date= 17 June 2015| archive-url=https://web.archive.org/web/20150617211645/http://www.statisticsviews.com/details/feature/5393251/Big-Data--Business-An-on-going-revolution.html| url-status= dead}}</ref> Additionally, user-generated data offers new opportunities to give the unheard a voice.<ref>{{cite web|title=Responsible use of data|author=Tobias Knobloch and Julia Manske|work=D+C, Development and Cooperation|date=11 January 2016|url=http://www.dandc.eu/en/article/opportunities-and-risks-user-generated-and-automatically-compiled-data|access-date= 11 January 2017|archive-date=13 January 2017|archive-url=https://web.archive.org/web/20170113111907/http://www.dandc.eu/en/article/opportunities-and-risks-user-generated-and-automatically-compiled-data|url-status=live}}</ref> However, longstanding challenges for developing regions such as inadequate technological infrastructure and economic and human resource scarcity exacerbate existing concerns with big data such as privacy, imperfect methodology, and interoperability issues.{{Sfn | Hilbert | 2016}}{{Page needed |date=December 2023}}  The challenge of "big data for development"{{Sfn | Hilbert | 2016}}{{Page needed |date=December 2023}} is currently evolving toward the application of this data through machine learning, known as "artificial intelligence for development (AI4D).<ref>Mann, S., & Hilbert, M. (2020). AI4D: Artificial Intelligence for Development. International Journal of Communication, 14(0), 21. https://www.martinhilbert.net/ai4d-artificial-intelligence-for-development/ {{Webarchive|url=https://web.archive.org/web/20210422113444/https://www.martinhilbert.net/ai4d-artificial-intelligence-for-development/ |date=22 April 2021 }}</ref>

====Benefits====
A major practical application of big data for development has been "fighting poverty with data".<ref>Blumenstock, J. E. (2016). Fighting poverty with data. Science, 353(6301), 753–754. https://doi.org/10.1126/science.aah5217 {{Webarchive|url=https://web.archive.org/web/20220601224735/https://www.science.org/doi/10.1126/science.aah5217 |date=1 June 2022 }}</ref> In 2015, Blumenstock and colleagues estimated predicted poverty and wealth from mobile phone metadata<ref>Blumenstock, J., Cadamuro, G., & On, R. (2015). Predicting poverty and wealth from mobile phone metadata. Science, 350(6264), 1073–1076. https://doi.org/10.1126/science.aac4420 {{Webarchive|url=https://web.archive.org/web/20220601224737/https://www.science.org/doi/10.1126/science.aac4420 |date=1 June 2022 }}</ref> and in 2016 Jean and colleagues combined satellite imagery and machine learning to predict poverty.<ref>Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science, 353(6301), 790–794. https://doi.org/10.1126/science.aaf7894 {{Webarchive|url=https://web.archive.org/web/20220601224736/https://www.science.org/doi/10.1126/science.aaf7894 |date=1 June 2022 }}</ref> Using digital trace data to study the labor market and the digital economy in Latin America, [[Martin Hilbert|Hilbert]] and colleagues <ref name="HilbertJobMarket">Hilbert, M., & Lu, K. (2020). The online job market trace in Latin America and the Caribbean (UN ECLAC LC/TS.2020/83; p. 79). United Nations Economic Commission for Latin America and the Caribbean. https://www.cepal.org/en/publications/45892-online-job-market-trace-latin-america-and-caribbean {{Webarchive|url=https://web.archive.org/web/20200922023629/https://www.cepal.org/en/publications/45892-online-job-market-trace-latin-america-and-caribbean |date=22 September 2020 }}</ref><ref>UN ECLAC, (United Nations Economic Commission for Latin America and the Caribbean). (2020). Tracking the digital footprint in Latin America and the Caribbean: Lessons learned from using big data to assess the digital economy (Productive Development, Gender Affairs LC/TS.2020/12; Documentos de Proyecto). United Nations ECLAC. https://repositorio.cepal.org/handle/11362/45484 {{Webarchive|url=https://web.archive.org/web/20200918174030/https://repositorio.cepal.org/handle/11362/45484 |date=18 September 2020 }}</ref> argue that digital trace data has several benefits such as:
* Thematic coverage: including areas that were previously difficult or impossible to measure
* Geographical coverage: providing sizable and comparable data for almost all countries, including many small countries that usually are not included in international inventories
* Level of detail: providing fine-grained data with many interrelated variables, and new aspects, like network connections
* Timeliness and timeseries: graphs can be produced within days of being collected

====Challenges====
At the same time, working with digital trace data instead of traditional survey data does not eliminate the traditional challenges involved when working in the field of international quantitative analysis. Priorities change, but the basic discussions remain the same. Among the main challenges are:
* Representativeness. While traditional development statistics is mainly concerned with the representativeness of random survey samples, digital trace data is never a random sample.<ref>{{Cite journal|last1=Banerjee|first1=Amitav|last2=Chaudhury|first2=Suprakash|date=2010|title=Statistics without tears: Populations and samples|journal=Industrial Psychiatry Journal|volume=19|issue=1|pages=60–65|doi=10.4103/0972-6748.77642|issn=0972-6748|pmc=3105563|pmid=21694795 |doi-access=free }}</ref>
* Generalizability. While observational data always represents this source very well, it only represents what it represents, and nothing more. While it is tempting to generalize from specific observations of one platform to broader settings, this is often very deceptive.
* Harmonization. Digital trace data still requires international harmonization of indicators. It adds the challenge of so-called "data-fusion", the harmonization of different sources.
* Data overload. Analysts and institutions are not used to effectively deal with a large number of variables, which is efficiently done with interactive dashboards. Practitioners still lack a standard workflow that would allow researchers, users and policymakers to efficiently and effectively deal with data.<ref name="HilbertJobMarket" />

=== Finance ===
Big Data is being rapidly adopted in Finance to 1) speed up processing and 2) deliver better, more informed inferences, both internally and to the clients of the financial institutions.<ref>{{Cite book |last=Aldridge |first=Irene |url=https://www.worldcat.org/oclc/972292212 |title=Real-Time Risk : What Investors Should Know about FinTech, High-Frequency Trading, and Flash Crashes. |date=2016 |publisher=John Wiley & Sons, Incorporated |others=Steven Krawciw |isbn=978-1-119-31906-1 |location=Somerset |oclc=972292212}}</ref> The financial applications of Big Data range from investing decisions and trading (processing volumes of available price data, limit order books, economic data and more, all at the same time), portfolio management (optimizing over an increasingly large array of financial instruments, potentially selected from different asset classes), risk management (credit rating based on extended information), and any other aspect where the data inputs are large.<ref>{{Cite book |last=Aldridge |first=Irene |url=https://www.worldcat.org/oclc/1184122216 |publisher=Wiley |title=Big data science in finance |date=2021 |others=Marco Avellaneda |isbn=978-1-119-60297-2 |location=Hoboken, New Jersey |oclc=1184122216}}</ref> Big Data has also been a typical concept within the field of [[alternative financial service]]. Some of the major areas involve crowd-funding platforms and crypto currency exchanges.<ref>{{Cite journal |last1=Hasan |first1=Md. Morshadul |last2=Popp |first2=József |last3=Oláh |first3=Judit |date=2020-03-12 |title=Current landscape and influence of big data on finance |journal=[[Journal of Big Data]] |volume=7 |issue=1 |pages=21 |doi=10.1186/s40537-020-00291-z |doi-access=free |issn=2196-1115}}</ref>

===Healthcare===
Big data analytics has been used in healthcare in providing personalized medicine and [[prescriptive analytics]], clinical risk intervention and predictive analytics, waste and care variability reduction, automated external and internal reporting of patient data, standardized medical terms and patient registries.<ref name="ref135">{{cite journal | vauthors = Huser V, Cimino JJ | title = Impending Challenges for the Use of Big Data | journal = International Journal of Radiation Oncology, Biology, Physics | volume = 95 | issue = 3 | pages = 890–894 | date = July 2016 | pmid = 26797535 | pmc = 4860172 | doi = 10.1016/j.ijrobp.2015.10.060 }}</ref><ref>{{Cite book|title=Signal Processing and Machine Learning for Biomedical Big Data.|others=Sejdić, Ervin, Falk, Tiago H.|isbn=9781351061216|location=[Place of publication not identified]|oclc=1044733829|last1 = Sejdic|first1 = Ervin|last2 = Falk|first2 = Tiago H.|date = 4 July 2018}}</ref><ref>{{cite journal | vauthors = Raghupathi W, Raghupathi V | title = Big data analytics in healthcare: promise and potential | journal = Health Information Science and Systems | volume = 2 | issue = 1 | pages = 3 | date = December 2014 | pmid = 25825667 | pmc = 4341817 | doi = 10.1186/2047-2501-2-3 | doi-access = free }}</ref><ref>{{cite journal | vauthors = Viceconti M, Hunter P, Hose R | title = Big data, big knowledge: big data for personalized healthcare | journal = IEEE Journal of Biomedical and Health Informatics | volume = 19 | issue = 4 | pages = 1209–15 | date = July 2015 | pmid = 26218867 | doi = 10.1109/JBHI.2015.2406883 | s2cid = 14710821 | url = http://eprints.whiterose.ac.uk/89104/1/pap%20JBHI%20BigData%20in%20VPH%20revision%20v2.pdf | doi-access = free | access-date = 21 September 2019 | archive-date = 23 July 2018 | archive-url = https://web.archive.org/web/20180723201503/http://eprints.whiterose.ac.uk/89104/1/pap%20JBHI%20BigData%20in%20VPH%20revision%20v2.pdf | url-status = live }}</ref> Some areas of improvement are more aspirational than actually implemented. The level of data generated within [[Health system|healthcare systems]] is not trivial. With the added adoption of mHealth, eHealth and wearable technologies the volume of data will continue to increase. This includes [[electronic health record]] data, imaging data, patient generated data, sensor data, and other forms of difficult to process data. There is now an even greater need for such environments to pay greater attention to data and information quality.<ref>{{cite journal|title=Data Management Within mHealth Environments: Patient Sensors, Mobile Devices, and Databases |first1=John| last1=O'Donoghue |first2=John|last2=Herbert|s2cid=2318649|date=1 October 2012|volume=4|issue=1|pages=5:1–5:20| doi=10.1145/2378016.2378021 |journal=Journal of Data and Information Quality}}</ref> "Big data very often means '[[dirty data]]' and the fraction of data inaccuracies increases with data volume growth." Human inspection at the big data scale is impossible and there is a desperate need in health service for intelligent tools for accuracy and believability control and handling of information missed.<ref name="Mirkes2016">{{cite journal | vauthors = Mirkes EM, Coats TJ, Levesley J, Gorban AN | title = Handling missing data in large healthcare dataset: A case study of unknown trauma outcomes | journal = Computers in Biology and Medicine | volume = 75 | pages = 203–16 | date = August 2016 | pmid = 27318570 | doi = 10.1016/j.compbiomed.2016.06.004 | arxiv = 1604.00627 | bibcode = 2016arXiv160400627M | s2cid = 5874067 }}</ref> While extensive information in healthcare is now electronic, it fits under the big data umbrella as most is unstructured and difficult to use.<ref>{{cite journal | vauthors = Murdoch TB, Detsky AS | title = The inevitable application of big data to health care | journal = JAMA | volume = 309 | issue = 13 | pages = 1351–2 | date = April 2013 | pmid = 23549579 | doi = 10.1001/jama.2013.393 | s2cid = 20462354 }}</ref> The use of big data in healthcare has raised significant ethical challenges ranging from risks for individual rights, privacy and [[autonomy]], to transparency and trust.<ref>{{cite journal | vauthors = Vayena E, Salathé M, Madoff LC, Brownstein JS | title = Ethical challenges of big data in public health | journal = PLOS Computational Biology | volume = 11 | issue = 2 | pages = e1003904 | date = February 2015 | pmid = 25664461 | pmc = 4321985 | doi = 10.1371/journal.pcbi.1003904 | bibcode = 2015PLSCB..11E3904V | doi-access = free }}</ref>

Big data in health research is particularly promising in terms of exploratory biomedical research, as data-driven analysis can move forward more quickly than hypothesis-driven research.<ref>{{Cite journal|last=Copeland|first= CS |date= Jul–Aug 2017|title=Data Driving Discovery|url=http://claudiacopeland.com/uploads/3/5/5/6/35560346/_hjno_data_driving_discovery_2pv.pdf|journal=Healthcare Journal of New Orleans|pages=22–27|access-date=5 December 2019 |archive-date= 5 December 2019|archive-url=https://web.archive.org/web/20191205195456/http://claudiacopeland.com/uploads/3/5/5/6/35560346/_hjno_data_driving_discovery_2pv.pdf|url-status=live}}</ref> Then, trends seen in data analysis can be tested in traditional, hypothesis-driven follow up biological research and eventually clinical research.

A related application sub-area, that heavily relies on big data, within the healthcare field is that of [[computer-aided diagnosis]] in medicine.{{Sfn | Yanase | Triantaphyllou | 2019}}{{Page needed |date=December 2023}}  For instance, for [[epilepsy]] monitoring it is customary to create 5 to 10 GB of data daily.<ref>{{cite journal | vauthors = Dong X, Bahroos N, Sadhu E, Jackson T, Chukhman M, Johnson R, Boyd A, Hynes D| title = Leverage Hadoop framework for large scale clinical informatics applications | journal = AMIA Joint Summits on Translational Science Proceedings. AMIA Joint Summits on Translational Science | pages = 53 | date = 2013 | volume = 2013 | pmid = 24303235 }}</ref> Similarly, a single uncompressed image of breast [[tomosynthesis]] averages 450 MB of data.<ref>{{cite web | author = Clunie, D. | title = Breast tomosynthesis challenges digital imaging infrastructure | url = http://www.auntminnie.com/index.aspx?sec=prtf&sub=def&pag=dis&itemId=102872&printpage=true&fsec=ser&fsub=def | publisher= Science and Medicine Group | date = 2013 | access-date = 28 November 2023| archive-date = 24 February 2021 | archive-url = https://web.archive.org/web/20210224135732/https://www.auntminnie.com/index.aspx?sec=prtf&sub=def&pag=dis&itemId=102872&printpage=true&fsec=ser&fsub=def | url-status = live }}</ref>  These are just a few of the many examples where [[computer-aided diagnosis]] uses big data. For this reason, big data has been recognized as one of the seven key challenges that computer-aided diagnosis systems need to overcome in order to reach the next level of performance.<ref>{{cite journal | vauthors = Yanase J, Triantaphyllou E | title = The Seven Key Challenges for the Future of Computer-Aided Diagnosis in Medicine | journal = International Journal of Medical Informatics| volume = 129 | pages = 413–22 | year = 2019b | doi = 10.1016/j.ijmedinf.2019.06.017 | pmid = 31445285 | s2cid = 198287435 }}</ref>

===Education===
A [[McKinsey & Company|McKinsey Global Institute]] study found a shortage of 1.5 million highly trained data professionals and managers<ref name="McKinsey"/> and a number of universities<ref>{{cite web |url=https://www.forbes.com/sites/jmaureenhenderson/2013/07/30/degrees-in-big-data-fad-or-fast-track-to-career-success/ |access-date=21 February 2016 |website=[[Forbes]] |title=Degrees in Big Data: Fad or Fast Track to Career Success |archive-date=3 March 2016
|archive-url=https://web.archive.org/web/20160303210044/http://www.forbes.com/sites/jmaureenhenderson/2013/07/30/degrees-in-big-data-fad-or-fast-track-to-career-success/ |url-status=live}}</ref>{{better source needed|date=November 2018|reason=www.forbes.com/sites by contributors rather than staff are blogs, not reliable sources for facts.}} including [[University of Tennessee]] and [[UC Berkeley]], have created masters programs to meet this demand. Private boot camps have also developed programs to meet that demand, including paid programs like [[The Data Incubator]] or [[General Assembly]].<ref>{{cite news
|title=NY gets new boot camp for data scientists: It's free but harder to get into than Harvard
|newspaper=Venture Beat
|access-date=21 February 2016
|url=https://venturebeat.com/2014/04/15/ny-gets-new-bootcamp-for-data-scientists-its-free-but-harder-to-get-into-than-harvard/
|archive-date=15 February 2016
|archive-url=https://web.archive.org/web/20160215235820/http://venturebeat.com/2014/04/15/ny-gets-new-bootcamp-for-data-scientists-its-free-but-harder-to-get-into-than-harvard/
|url-status=live
}}</ref> In the specific field of marketing, one of the problems stressed by Wedel and Kannan<ref>{{cite journal|last=Wedel|first=Michel|author2=Kannan, PK|title= Marketing Analytics for Data-Rich Environments|journal=Journal of Marketing|year=2016|volume=80|issue=6|doi= 10.1509/jm.15.0413|pages=97–121|s2cid=168410284}}</ref> is that marketing has several sub domains (e.g., advertising, promotions,
product development, branding) that all use different types of data.

===Media===
To understand how the media uses big data, it is first necessary to provide some context into the mechanism used for media process. It has been suggested by Nick Couldry and Joseph Turow that practitioners in media and advertising approach big data as many actionable points of information about millions of individuals. The industry appears to be moving away from the traditional approach of using specific media environments such as newspapers, magazines, or television shows and instead taps into consumers with technologies that reach targeted people at optimal times in optimal locations. The ultimate aim is to serve or convey, a message or content that is (statistically speaking) in line with the consumer's mindset. For example, publishing environments are increasingly tailoring messages (advertisements) and content (articles) to appeal to consumers that have been exclusively gleaned through various [[data-mining]] activities.<ref>{{cite journal|last1=Couldry|first1=Nick|last2=Turow|first2=Joseph|title=Advertising, Big Data, and the Clearance of the Public Realm: Marketers' New Approaches to the Content Subsidy| journal=International Journal of Communication|date=2014|volume=8|pages=1710–1726}}</ref>
* Targeting of consumers (for advertising by marketers)<ref>{{cite web|url=https://ishti.org/2018/04/15/why-digital-advertising-agencies-suck-at-acquisition-and-are-in-dire-need-of-an-ai-assisted-upgrade/|title=Why Digital Advertising Agencies Suck at Acquisition and are in Dire Need of an AI Assisted Upgrade|website=Ishti.org|access-date=15 April 2018|date=15 April 2018|archive-date=12 February 2019|archive-url=https://web.archive.org/web/20190212174722/https://ishti.org/2018/04/15/why-digital-advertising-agencies-suck-at-acquisition-and-are-in-dire-need-of-an-ai-assisted-upgrade/|url-status=dead}}</ref>
* Data capture
* [[Data journalism]]: publishers and journalists use big data tools to provide unique and innovative insights and [[infographic]]s.

[[Channel 4]], the British [[Public service broadcasting in the United Kingdom|public-service]] television broadcaster, is a leader in the field of big data and [[data analysis]].<ref>{{cite web |url=https://www.ibc.org/tech-advances/big-data-and-analytics-c4-and-genius-digital/1076.article |title=Big data and analytics: C4 and Genius Digital |website=Ibc.org |access-date=8 October 2017 |archive-date=8 October 2017 |archive-url=https://web.archive.org/web/20171008231909/https://www.ibc.org/tech-advances/big-data-and-analytics-c4-and-genius-digital/1076.article |url-status=live }}</ref>

===Insurance===
Health insurance providers are collecting data on [[social determinants of health|social "determinants of health"]] such as food and [[Television consumption|TV consumption]], marital status, clothing size, and purchasing habits, from which they make predictions on health costs, in order to spot health issues in their clients. It is controversial whether these predictions are currently being used for pricing.<ref>{{Cite web|author=Marshall Allen|url=https://www.propublica.org/article/health-insurers-are-vacuuming-up-details-about-you-and-it-could-raise-your-rates|title=Health Insurers Are Vacuuming Up Details About You – And It Could Raise Your Rates|website=www.propublica.org|date=17 July 2018|access-date=21 July 2018|archive-date=21 July 2018|archive-url=https://web.archive.org/web/20180721011923/https://www.propublica.org/article/health-insurers-are-vacuuming-up-details-about-you-and-it-could-raise-your-rates|url-status=live}}</ref>

===Internet of things (IoT)===
{{Main|Internet of things}}
{{Further|Edge computing}}
Big data and the IoT work in conjunction. Data extracted from IoT devices provides a mapping of device inter-connectivity. Such mappings have been used by the media industry, companies, and governments to more accurately target their audience and increase media efficiency. The IoT is also increasingly adopted as a means of gathering sensory data, and this sensory data has been used in medical,<ref>{{cite web|url=http://www.businesswire.com/news/home/20170109006500/en/QuiO-Named-Innovation-Champion-Accenture-HealthTech-Innovation|title=QuiO Named Innovation Champion of the Accenture HealthTech Innovation Challenge|website=Businesswire.com|access-date=8 October 2017|date=10 January 2017|archive-date=22 March 2017|archive-url=https://web.archive.org/web/20170322111851/http://www.businesswire.com/news/home/20170109006500/en/QuiO-Named-Innovation-Champion-Accenture-HealthTech-Innovation|url-status=live}}</ref> manufacturing<ref>{{cite web|url=https://www.predix.com/sites/default/files/IDC_OT_Final_whitepaper_249120.pdf|title=A Software Platform for Operational Technology Innovation|website=Predix.com|access-date=8 October 2017|archive-date=22 March 2017|archive-url=https://web.archive.org/web/20170322111812/https://www.predix.com/sites/default/files/IDC_OT_Final_whitepaper_249120.pdf|url-status=dead}}</ref> and transportation<ref name="BigDataIoT16">{{cite web| url=http://www.wiomax.com/big-data-driven-smart-transportation-the-underlying-big-story-of-smart-iot-transformed-mobility/| title=Big Data Driven Smart Transportation: the Underlying Story of IoT Transformed Mobility| author=Z. Jenipher Wang| date=March 2017| access-date=4 July 2018| archive-date=4 July 2018| archive-url=https://web.archive.org/web/20180704093429/http://www.wiomax.com/big-data-driven-smart-transportation-the-underlying-big-story-of-smart-iot-transformed-mobility/| url-status=live}}</ref> contexts.

[[Kevin Ashton]], the digital innovation expert who is credited with coining the term,<ref>{{cite web|url=http://www.rfidjournal.com/articles/view?4986|title=That Internet Of Things Thing.|date=22 June 2009|access-date=29 December 2017|archive-date=2 May 2013|archive-url=https://web.archive.org/web/20130502134112/http://www.rfidjournal.com/articles/view?4986|url-status=dead}}</ref> defines the Internet of things in this quote: "If we had computers that knew everything there was to know about things—using data they gathered without any help from us—we would be able to track and count everything, and greatly reduce waste, loss, and cost. We would know when things needed replacing, repairing, or recalling, and whether they were fresh or past their best."

===Information technology===
Especially since 2015, big data has come to prominence within business operations as a tool to help employees work more efficiently and streamline the collection and distribution of [[information technology]] (IT). The use of big data to resolve IT and [[data collection]] issues within an enterprise is called [[IT operations analytics]] (ITOA).<ref name="ITOA1">{{cite web|last1=Solnik|first1=Ray|title=The Time Has Come: Analytics Delivers for IT Operations|url=http://www.datacenterjournal.com/time-analytics-delivers-operations/|website=Data Center Journal|access-date=21 June 2016|archive-date=4 August 2016|archive-url=https://web.archive.org/web/20160804175315/http://www.datacenterjournal.com/time-analytics-delivers-operations/|url-status=live}}</ref> By applying big data principles into the concepts of [[machine intelligence]] and deep computing, IT departments can predict potential issues and prevent them.<ref name="ITOA1" /> ITOA businesses offer platforms for [[systems management]] that bring [[data silos]] together and generate insights from the whole of the system rather than from isolated pockets of data.

===Survey science===
Compared to [[Survey methodology|survey]]-based data collection, big data has low cost per data point, applies analysis techniques via [[machine learning]] and [[data mining]], and includes diverse and new data sources, e.g., registers, social media, apps, and other forms digital data. Since 2018, survey scientists have started to examine how big data and survey science can complement each other to allow researchers and practitioners to improve the production of statistics and its quality. There have been three Big Data Meets Survey Science (BigSurv) conferences in 2018, 2020 (virtual), 2023, and {{as of|2023|lc=y}} one conference forthcoming in 2025,<ref>{{Cite web |title=BigSurv: Big Data Meets Survey Science |url=https://www.bigsurv.org/ |access-date=October 15, 2023}}</ref> a special issue in the ''[[Social Science Computer Review]]'',<ref>{{Cite journal |last1=Eck |first1=Adam |last2=Cazar |first2=Ana Lucía Córdova |last3=Callegaro |first3=Mario |last4=Biemer |first4=Paul |date= 2021|title="Big Data Meets Survey Science" |journal=Social Science Computer Review |language=en |volume=39 |issue=4 |pages=484–488 |doi=10.1177/0894439319883393 |issn=|doi-access=free }}</ref> a special issue in ''[[Journal of the Royal Statistical Society]]'',<ref>{{Cite journal |title=Special issue: Big data meets survey science |url=https://rss.onlinelibrary.wiley.com/toc/1467985x/current |journal=Journal of the Royal Statistical Society, Series A |volume=185 |issue=S2 |pages=S165–S166}}</ref> and a special issue in [[European Physical Journal|''EP J Data Science'']],<ref>{{Cite web |title=Integrating Survey and Non-survey Data to Measure Behavior and Public Opinion |url=https://www.springeropen.com/collections/BigSurv20 |access-date=2023-10-19 |website=www.springeropen.com |language=en}}</ref> and a book called ''Big Data Meets Social Sciences''<ref>{{Cite book |url=https://onlinelibrary.wiley.com/doi/book/10.1002/9781118976357 |title=Big Data Meets Survey Science: A Collection of Innovative Methods |date=2020-10-13 |publisher=Wiley |isbn=978-1-118-97632-6 |editor-last=Hill |editor-first=Craig A. |edition=1 |language=en |doi=10.1002/9781118976357 |s2cid=240797608 |editor-last2=Biemer |editor-first2=Paul P. |editor-last3=Buskirk |editor-first3=Trent D. |editor-last4=Japec |editor-first4=Lilli |editor-last5=Kirchner |editor-first5=Antje |editor-last6=Kolenikov |editor-first6=Stas |editor-last7=Lyberg |editor-first7=Lars E.}}</ref> edited by [[Craig A. Hill|Craig Hill]] and five other [[List of fellows of the American Statistical Association|Fellows of the American Statistical Association]]. In 2021, the founding members of BigSurv received the Warren J. Mitofsky Innovators Award from the [[American Association for Public Opinion Research]].<ref>{{Cite web |date=2023-06-07 |title=Past Warren J. Mitofsky Innovators Award Winners - AAPOR |url=https://aapor.org/past-mitofsky/ |access-date=2023-10-19 |language=en-US}}</ref>

=== Marketing ===
Big data is notable in marketing due to the constant "datafication"<ref name=":1">{{Cite book |last=Strong |first=Colin |title=Humanizing big data: marketing at the meeting of data, social science and consumer insight |date=2015 |publisher=Kogan Page |isbn=978-0-7494-7211-5 |location=London}}</ref> of everyday consumers of the internet, in which all forms of data are tracked. The datafication of consumers can be defined as  quantifying many of or all human behaviors for the purpose of marketing.<ref name=":1" /> The increasingly digital world of rapid datafication makes this idea relevant to marketing because the amount of data constantly grows exponentially. It is predicted to increase from 44 to 163 zettabytes within the span of five years.<ref>Berisha, B., Mëziu, E., & Shabani, I. (2022). Big data analytics in Cloud computing: An overview. ''Journal of Cloud Computing'', ''11''(1), 1-10. {{doi|10.1186/s13677-022-00301-w}}</ref> The size of big data can often be difficult to navigate for marketers.<ref>Bosch, Volker (2016-11-01). "Big Data in Market Research: Why More Data Does Not Automatically Mean Better Information". ''NIM Marketing Intelligence Review''. '''8''' (2): 56–63. {{doi|10.1515/gfkmir-2016-0017}}.</ref> As a result, adopters of big data may find themselves at a disadvantage. Algorithmic findings can be difficult to achieve with such large datasets.<ref>McFarland, Daniel A; McFarland, H Richard (2015-12-01). "Big Data and the danger of being precisely inaccurate". ''Big Data & Society''. '''2'''(2): 205395171560249. {{doi|10.1177/2053951715602495}}. {{ISSN|2053-9517}}.</ref> Big data in marketing is a highly lucrative tool that can be used for large corporations, its value being as a result of the possibility of predicting significant trends, interests, or statistical outcomes in a consumer-based manner.<ref>Sivarajah, Uthayasankar; Kamal, Muhammad Mustafa; Irani, Zahir; Weerakkody, Vishanth (2017-01-01). "Critical analysis of Big Data challenges and analytical methods". ''Journal of Business Research''. '''70''': 263–286. {{doi|10.1016/j.jbusres.2016.08.001}}. {{ISSN|0148-2963}}.</ref>

There are three significant factors in the use of big data in marketing:

# Big data provides customer behavior pattern spotting for marketers, since all human actions are being quantified into readable numbers for marketers to analyze and use for their research.<ref name="ReferenceA">De Luca, Luigi M.; Herhausen, Dennis; Troilo, Gabriele; Rossi, Andrea (2021-07-01). "How and when do big data investments pay off? The role of marketing affordances and service innovation". ''Journal of the Academy of Marketing Science''. '''49''' (4): 790–810.</ref> In addition, big data can also be seen as a customized product recommendation tool. Specifically, since big data is effective in analyzing customers' purchase behaviors and browsing patterns, this technology can assist companies in promoting specific personalized products to specific customers.<ref>{{Cite journal |last1=Ghasemaghaei |first1=Maryam |last2=Calic |first2=Goran |date=January 2020 |title=Assessing the impact of big data on firm innovation performance: Big data is not always better data |url=https://doi.org/10.1016/j.jbusres.2019.09.062 |journal=Journal of Business Research |volume=108 |pages=147–162 |doi=10.1016/j.jbusres.2019.09.062 |issn=0148-2963}}</ref>
# Real-time market responsiveness is important for marketers because of the ability to shift marketing efforts and correct to current trends, which is helpful in maintaining relevance to consumers. This can supply corporations with the information necessary to predict the wants and needs of consumers in advance.<ref name="ReferenceA"/>
# Data-driven market ambidexterity are being highly fueled by big data.<ref name="ReferenceA"/> New models and algorithms are being developed to make significant predictions about certain economic and social situations.<ref>Grybauskas, Andrius; Pilinkienė, Vaida; Stundžienė, Alina (2021-08-03). "Predictive analytics using Big Data for the real estate market during the COVID-19 pandemic". ''[[Journal of Big Data]]''. '''8'''(1): 105. {{doi|10.1186/s40537-021-00476-0}}. {{ISSN|2196-1115}}. [[PMC (identifier)|PMC]] 8329615. [[PMID (identifier)|PMID]] 34367876.</ref>
==Case studies==
===Government===

====China====
* The Integrated Joint Operations Platform (IJOP, 一体化联合作战平台) is used by the government to monitor the population, particularly [[Uyghurs]].<ref name="WP8218">{{cite news| url=https://www.washingtonpost.com/opinions/global-opinions/ethnic-cleansing-makes-a-comeback--in-china/2018/08/02/| archive-url=https://web.archive.org/web/20190331161843/https://www.washingtonpost.com/opinions/global-opinions/ethnic-cleansing-makes-a-comeback--in-china/2018/08/02/| url-status=dead| archive-date=31 March 2019|title=Ethnic cleansing makes a comeback – in China|author1=Josh Rogin|date=2 August 2018|access-date=4 August 2018|issue=Washington Post|quote=Add to that the unprecedented security and surveillance state in Xinjiang, which includes all-encompassing monitoring based on identity cards, checkpoints, facial recognition and the collection of DNA from millions of individuals. The authorities feed all this data into an artificial-intelligence machine that rates people's loyalty to the Communist Party in order to control every aspect of their lives.}}</ref> [[Biometrics]], including DNA samples, are gathered through a program of free physicals.<ref name="how022618">{{cite web|url=https://www.hrw.org/news/2018/02/26/china-big-data-fuels-crackdown-minority-region|title=China: Big Data Fuels Crackdown in Minority Region: Predictive Policing Program Flags Individuals for Investigations, Detentions|date=26 February 2018|website=hrw.org|publisher=Human Rights Watch|access-date=4 August 2018|archive-date=21 December 2019|archive-url=https://web.archive.org/web/20191221181307/https://www.hrw.org/news/2018/02/26/china-big-data-fuels-crackdown-minority-region|url-status=live}}</ref>
*By 2020, China plans to give all its citizens a personal "social credit" score based on how they behave.<ref>{{cite news |title=Discipline and Punish: The Birth of China's Social-Credit System |url=https://www.thenation.com/article/china-social-credit-system/ |work=The Nation |date=23 January 2019 |access-date=8 August 2019 |archive-date=13 September 2019 |archive-url=https://web.archive.org/web/20190913003747/https://www.thenation.com/article/china-social-credit-system/ |url-status=live }}</ref> The [[Social Credit System]], now being piloted in a number of Chinese cities, is considered a form of [[Mass surveillance in China|mass surveillance]] which uses big data analysis technology.<ref>{{cite news |title=China's behavior monitoring system bars some from travel, purchasing property |url=https://www.cbsnews.com/news/china-social-credit-system-surveillance-cameras/ |work=CBS News |date=24 April 2018 |access-date=8 August 2019 |archive-date=13 August 2019 |archive-url=https://web.archive.org/web/20190813005333/https://www.cbsnews.com/news/china-social-credit-system-surveillance-cameras/ |url-status=live }}</ref><ref>{{cite magazine |title=The complicated truth about China's social credit system |url=https://www.wired.co.uk/article/china-social-credit-system-explained |magazine=WIRED |date=21 January 2019 |access-date=8 August 2019 |archive-date=8 August 2019 |archive-url=https://web.archive.org/web/20190808115110/https://www.wired.co.uk/article/china-social-credit-system-explained |url-status=live }}</ref>

====India====
* Big data analysis was tried out for the [[Bharatiya Janata Party|BJP]] to win the 2014 Indian General Election.<ref>{{cite web|url = http://www.livemint.com/Industry/bUQo8xQ3gStSAy5II9lxoK/Are-Indian-companies-making-enough-sense-of-Big-Data.html|title = News: Live Mint|date = 23 June 2014|access-date = 22 November 2014|website = Are Indian companies making enough sense of Big Data?|publisher = Live Mint|archive-date = 29 November 2014|archive-url = https://web.archive.org/web/20141129063451/http://www.livemint.com/Industry/bUQo8xQ3gStSAy5II9lxoK/Are-Indian-companies-making-enough-sense-of-Big-Data.html|url-status = live}}</ref>
* The [[Government of India|Indian government]] uses numerous techniques to ascertain how the Indian electorate is responding to government action, as well as ideas for policy augmentation.

====Israel====
* Personalized diabetic treatments can be created through GlucoMe's big data solution.<ref>{{Cite news|url=https://www.timesofisrael.com/israeli-startup-uses-big-data-minimal-hardware-to-treat-diabetes/|title=Israeli startup uses big data, minimal hardware to treat diabetes|work=[[The Times of Israel]]|access-date=28 February 2018|archive-date=1 March 2018|archive-url=https://web.archive.org/web/20180301105651/https://www.timesofisrael.com/israeli-startup-uses-big-data-minimal-hardware-to-treat-diabetes/|url-status=live}}</ref>

====United Kingdom====
Examples of uses of big data in public services:

* Data on prescription drugs: by connecting origin, location and the time of each prescription, a research unit was able to exemplify and examine the considerable delay between the release of any given drug, and a UK-wide adaptation of the [[National Institute for Health and Care Excellence]] guidelines. This suggests that new or most up-to-date drugs take some time to filter through to the general patient.{{citation needed|date=January 2021}}<ref>{{Cite journal|last1=Singh |first1=Gurparkash|first2=Duane |last2=Schulthess|first3=Nigel |last3=Hughes|first4=Bart |last4=Vannieuwenhuyse|first5=Dipak |last5=Kalra|title=Real world big data for clinical research and drug development|journal=Drug Discovery Today|year=2018|volume=23|issue=3|pages=652–660|doi=10.1016/j.drudis.2017.12.002|pmid=29294362|doi-access=free}}</ref>
* Joining up data: a local authority [[Data blending|blended data]] about services, such as road gritting rotas, with services for people at risk, such as [[Meals on Wheels]]. The connection of data allowed the local authority to avoid any weather-related delay.<ref>{{cite web|url=https://www.researchgate.net/publication/297762848|title=Recent advances delivered by Mobile Cloud Computing and Internet of Things for Big Data applications: a survey|date=11 March 2016|publisher=International Journal of Network Management|access-date=14 September 2016|archive-date=1 June 2022|archive-url=https://web.archive.org/web/20220601224735/https://www.researchgate.net/publication/297762848|url-status=live}}</ref>

====United States====
* In 2012, the [[Presidency of Barack Obama|Obama administration]] announced the Big Data Research and Development Initiative, to explore how big data could be used to address important problems faced by the government.<ref name="WH_Big_Data">{{cite web|url=https://obamawhitehouse.archives.gov/blog/2012/03/29/big-data-big-deal|title=Big Data is a Big Deal|last=Kalil|first=Tom|access-date=26 September 2012|via=[[NARA|National Archives]]|work=[[whitehouse.gov]]|date=29 March 2012|archive-date=10 January 2017|archive-url=https://web.archive.org/web/20170110083537/https://www.whitehouse.gov/blog/2012/03/29/big-data-big-deal|url-status=live}}</ref> The initiative is composed of 84 different big data programs spread across six departments.<ref>{{cite web|url =https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/big_data_fact_sheet_final_1.pdf|archive-url =https://web.archive.org/web/20170121233257/https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/big_data_fact_sheet_final_1.pdf|url-status =live|archive-date =21 January 2017|title=Big Data Across the Federal Government|last=Executive Office of the President|date=March 2012 |via =[[NARA|National Archives]]|work =[[Office of Science and Technology Policy]]|access-date=26 September 2012}}</ref>
* Big data analysis played a large role in [[Barack Obama]]'s successful [[Barack Obama presidential campaign, 2012|2012 re-election campaign]].<ref name="infoworld_bigdata">{{cite web|url=http://www.infoworld.com/d/big-data/the-real-story-of-how-big-data-analytics-helped-obama-win-212862|title=The real story of how big data analytics helped Obama win|last=Lampitt|first=Andrew|work=[[InfoWorld]]|access-date=31 May 2014|date=14 February 2013|archive-date=5 July 2014|archive-url=https://web.archive.org/web/20140705220107/http://www.infoworld.com/d/big-data/the-real-story-of-how-big-data-analytics-helped-obama-win-212862|url-status=live}}</ref>
* The [[United States Federal Government]] owns four of the ten most powerful [[supercomputer]]s in the world.<ref>{{Cite web |title=November 2023 {{!}} TOP500 |url=https://www.top500.org/lists/top500/2023/11/ |url-status=live |archive-url=https://web.archive.org/web/20240407010056/https://www.top500.org/lists/top500/2023/11/ |archive-date=7 April 2024 |access-date=20 April 2024}}</ref><ref>{{cite web|url=http://www.informationweek.com/government/enterprise-applications/image-gallery-governments-10-most-powerf/224700271|title=Government's 10 Most Powerful Supercomputers|last=Hoover|first=J. Nicholas|work=Information Week|publisher=UBM|access-date=26 September 2012|archive-date=16 October 2013|archive-url=https://web.archive.org/web/20131016082026/http://www.informationweek.com/government/enterprise-applications/image-gallery-governments-10-most-powerf/224700271|url-status=live}}</ref>
* The [[Utah Data Center]] has been constructed by the United States [[National Security Agency]]. When finished, the facility will be able to handle a large amount of information collected by the NSA over the Internet. The exact amount of storage space is unknown, but more recent sources claim it will be on the order of a few [[exabyte]]s.<ref>{{cite magazine|url=https://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/all/1|title=The NSA Is Building the Country's Biggest Spy Center (Watch What You Say)|last=Bamford|first=James|date=15 March 2012|magazine=[[Wired (magazine)|Wired]]|access-date=18 March 2013|archive-date=4 April 2012|archive-url=https://web.archive.org/web/20120404172946/http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/all/1|url-status=live}}</ref><ref>{{cite web|url=http://www.nsa.gov/public_info/press_room/2011/utah_groundbreaking_ceremony.shtml|title=Groundbreaking Ceremony Held for $1.2 Billion Utah Data Center|publisher=National Security Agency Central Security Service|access-date=18 March 2013|archive-url=https://web.archive.org/web/20130905055004/http://www.nsa.gov/public_info/press_room/2011/utah_groundbreaking_ceremony.shtml|archive-date=5 September 2013|url-status=dead}}</ref><ref>{{cite magazine| url=https://www.forbes.com/sites/kashmirhill/2013/07/24/blueprints-of-nsa-data-center-in-utah-suggest-its-storage-capacity-is-less-impressive-than-thought/| title=Blueprints of NSA's Ridiculously Expensive Data Center in Utah Suggest It Holds Less Info Than Thought| last=Hill| first=Kashmir| magazine=Forbes| access-date=31 October 2013| archive-date=29 March 2018| archive-url=https://web.archive.org/web/20180329214044/https://www.forbes.com/sites/kashmirhill/2013/07/24/blueprints-of-nsa-data-center-in-utah-suggest-its-storage-capacity-is-less-impressive-than-thought/| url-status=live}}</ref> This has posed security concerns regarding the anonymity of the data collected.<ref>{{Cite news|url=https://www.huffingtonpost.com/2013/06/12/nsa-big-data_n_3423482.html|title=NSA Spying Controversy Highlights Embrace of Big Data|last1=Smith|first1=Gerry|date=12 June 2013|work=Huffington Post|access-date=7 May 2018|last2=Hallman|first2=Ben|archive-date=19 July 2017|archive-url=https://web.archive.org/web/20170719123431/http://www.huffingtonpost.com/2013/06/12/nsa-big-data_n_3423482.html|url-status=live}}</ref>

===Retail===
* [[Walmart]] handles more than 1 million customer transactions every hour, which are imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data—the equivalent of 167 times the information contained in all the books in the US [[Library of Congress]].{{r|Economist}}
* [[Windermere Real Estate]] uses location information from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day.<ref>{{cite news |last=Wingfield |first=Nick |url=http://bits.blogs.nytimes.com/2013/03/12/predicting-commutes-more-accurately-for-would-be-home-buyers/ |title=Predicting Commutes More Accurately for Would-Be Home Buyers |work=The New York Times |date=12 March 2013 |access-date=21 July 2013 |archive-date=29 May 2013 |archive-url=https://web.archive.org/web/20130529090534/http://bits.blogs.nytimes.com/2013/03/12/predicting-commutes-more-accurately-for-would-be-home-buyers/ |url-status=live }}</ref>
* FICO Card Detection System protects accounts worldwide.<ref name="fico.com">{{cite web |url=http://www.fico.com/en/Products/DMApps/Pages/FICO-Falcon-Fraud-Manager.aspx |title=FICO® Falcon® Fraud Manager |publisher=Fico.com |access-date=21 July 2013 |archive-date=11 November 2012 |archive-url=https://web.archive.org/web/20121111060238/http://www.fico.com/en/products/dmapps/pages/fico-falcon-fraud-manager.aspx |url-status=live }}</ref>
* Omnichannel retailing<ref>{{Cite journal |last1=Brynjolfsson |first1=Erik |last2=Hu |first2=Yu Jeffrey |last3=Rahman |first3=Mohammad S. |date=2013-05-21 |title=Competing in the Age of Omnichannel Retailing |url=https://sloanreview.mit.edu/article/competing-in-the-age-of-omnichannel-retailing/ |journal=MIT Sloan Management Review |language=en-US}}</ref> leverages online big data to improve offline experiences.

===Science===
* The [[Large Hadron Collider]] experiments represent about 150 million sensors delivering data 40&nbsp;million times per second. There are nearly 600&nbsp;million collisions per second. After filtering and refraining from recording more than 99.99995%<ref>{{cite web|last1=Alexandru|first1=Dan|title=Prof|url=https://cds.cern.ch/record/1504817/files/CERN-THESIS-2013-004.pdf|website=cds.cern.ch|publisher=CERN|access-date=24 March 2015|archive-date=15 July 2017|archive-url=https://web.archive.org/web/20170715091845/https://cds.cern.ch/record/1504817/files/CERN-THESIS-2013-004.pdf|url-status=live}}</ref> of these streams, there are 1,000 collisions of interest per second.<ref>{{cite web |title=LHC Brochure, English version. A presentation of the largest and the most powerful particle accelerator in the world, the Large Hadron Collider (LHC), which started up in 2008. Its role, characteristics, technologies, etc. are explained for the general public. |url=https://cds.cern.ch/record/1278169?ln=en |work=CERN-Brochure-2010-006-Eng. LHC Brochure, English version. |publisher=CERN |access-date=20 January 2013 |archive-date=19 March 2019 |archive-url=https://web.archive.org/web/20190319224056/https://cds.cern.ch/record/1278169?ln=en |url-status=live }}</ref><ref>{{cite web |title=LHC Guide, English version. A collection of facts and figures about the Large Hadron Collider (LHC) in the form of questions and answers. |url=https://cds.cern.ch/record/1092437?ln=en |work=CERN-Brochure-2008-001-Eng. LHC Guide, English version. |publisher=CERN |access-date=20 January 2013 |archive-date=7 April 2020 |archive-url=https://web.archive.org/web/20200407070110/https://login.cern.ch/adfs/ls/?wa=wsignin1.0&wreply=https%3A%2F%2Fcds.cern.ch%2FShibboleth.sso%2FADFS&wct=2020-04-07T07%3A01%3A09Z&wtrealm=https%3A%2F%2Fcds.cern.ch%2FShibboleth.sso%2FADFS&wctx=cookie%3A1586242869_c84d |url-status=live }}</ref><ref name="nature">{{cite journal |title= High-energy physics: Down the petabyte highway |journal= Nature |date= 19 January 2011 |first= Geoff |last= Brumfiel |doi= 10.1038/469282a |volume= 469 |issue= 7330 |pages= 282–83 |pmid= 21248814 |bibcode= 2011Natur.469..282B |s2cid= 533166 |doi-access= free }}</ref>
** As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication ({{as of|2012|lc=y}}). This becomes nearly 200 petabytes after replication.
** If all sensor data were recorded in LHC, the data flow would be extremely hard to work with. The data flow would exceed 150 million petabytes annual rate, or nearly 500 [[exabyte]]s per day, before replication. To put the number in perspective, this is equivalent to 500 [[quintillion]] (5×10<sup>20</sup>) bytes per day, almost 200 times more than all the other sources combined in the world.
* The [[Square Kilometre Array]] is a radio telescope built of thousands of antennas. It is expected to be operational by 2024. Collectively, these antennas are expected to gather 14 exabytes and store one petabyte per day.<ref>{{cite web|url=http://www.zurich.ibm.com/pdf/astron/CeBIT+2013+Background+DOME.pdf|title=IBM Research – Zurich|website=Zurich.ibm.com|access-date=8 October 2017|archive-date=1 June 2022|archive-url=https://web.archive.org/web/20220601224737/https://www.zurich.ibm.com|url-status=live}}</ref><ref>{{cite web|url=https://arstechnica.com/science/2012/04/future-telescope-array-drives-development-of-exabyte-processing/|title=Future telescope array drives development of Exabyte processing|work=Ars Technica|date=2 April 2012|access-date=15 April 2015|archive-date=31 March 2019|archive-url=https://web.archive.org/web/20190331030706/https://arstechnica.com/science/2012/04/future-telescope-array-drives-development-of-exabyte-processing/|url-status=live}}</ref> It is considered one of the most ambitious scientific projects ever undertaken.<ref>{{cite web|url=http://theconversation.com/australias-bid-for-the-square-kilometre-array-an-insiders-perspective-4891|title=Australia's bid for the Square Kilometre Array – an insider's perspective|date=1 February 2012|publisher=[[The Conversation (website)|The Conversation]]|access-date=27 September 2016|archive-date=12 October 2016|archive-url=https://web.archive.org/web/20161012094526/http://theconversation.com/australias-bid-for-the-square-kilometre-array-an-insiders-perspective-4891|url-status=live}}</ref>
* When the [[Sloan Digital Sky Survey]] (SDSS) began to collect astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy previously. Continuing at a rate of about 200&nbsp;GB per night, SDSS has amassed more than 140 terabytes of information.<ref name="Economist">{{cite news |title=Data, data everywhere |url=http://www.economist.com/node/15557443 |newspaper=The Economist |date=25 February 2010 |access-date=9 December 2012 |archive-date=27 May 2018 |archive-url=https://web.archive.org/web/20180527012646/https://www.economist.com/node/15557443 |url-status=live }}</ref> When the [[Large Synoptic Survey Telescope]], successor to SDSS, comes online in 2020, its designers expect it to acquire that amount of data every five days.{{r|Economist}}
*[[Human Genome Project|Decoding the human genome]] originally took 10 years to process; now it can be achieved in less than a day. The DNA sequencers have divided the sequencing cost by 10,000 in the last ten years, which is 100 times less expensive than the reduction in cost predicted by [[Moore's law]].<ref>{{cite web|url=http://www.oecd.org/sti/ieconomy/Session_3_Delort.pdf#page=6|title=Delort P., OECD ICCP Technology Foresight Forum, 2012.|website=Oecd.org|access-date=8 October 2017|archive-date=19 June 2017|archive-url=https://web.archive.org/web/20170619214305/http://www.oecd.org/sti/ieconomy/Session_3_Delort.pdf#page=6|url-status=live}}</ref>
* The [[NASA]] Center for Climate Simulation (NCCS) stores 32 petabytes of climate observations and simulations on the Discover supercomputing cluster.<ref>{{cite web|url=http://www.nasa.gov/centers/goddard/news/releases/2010/10-051.html|title=NASA – NASA Goddard Introduces the NASA Center for Climate Simulation|website=Nasa.gov|access-date=13 April 2016|archive-date=3 April 2016|archive-url=https://web.archive.org/web/20160403061211/http://www.nasa.gov/centers/goddard/news/releases/2010/10-051.html|url-status=live}}</ref><ref>{{cite web|last=Webster |first=Phil|title=Supercomputing the Climate: NASA's Big Data Mission| url=http://www.csc.com/cscworld/publications/81769/81773-supercomputing_the_climate_nasa_s_big_data_mission |work=CSC World|publisher=Computer Sciences Corporation|access-date=18 January 2013|url-status=dead| archive-url =https://web.archive.org/web/20130104220150/http://www.csc.com/cscworld/publications/81769/81773-supercomputing_the_climate_nasa_s_big_data_mission|archive-date=4 January 2013}}</ref>
* Google's DNAStack compiles and organizes DNA samples of genetic data from around the world to identify diseases and other medical defects. These fast and exact calculations eliminate any "friction points", or human errors that could be made by one of the numerous science and biology experts working with the DNA. DNAStack, a part of Google Genomics, allows scientists to use the vast sample of resources from Google's search server to scale social experiments that would usually take years, instantly.<ref>{{cite news|url=https://www.theglobeandmail.com/life/health-and-fitness/health/these-six-great-neuroscience-ideas-could-make-the-leap-from-lab-to-market/article21681731/|title=These six great neuroscience ideas could make the leap from lab to market|date=20 November 2014|work=[[The Globe and Mail]]|access-date=1 October 2016|archive-date=11 October 2016|archive-url=https://web.archive.org/web/20161011132848/http://www.theglobeandmail.com/life/health-and-fitness/health/these-six-great-neuroscience-ideas-could-make-the-leap-from-lab-to-market/article21681731/|url-status=live}}</ref><ref>{{cite web|url=https://cloud.google.com/customers/dnastack/|title=DNAstack tackles massive, complex DNA datasets with Google Genomics|publisher=Google Cloud Platform|access-date=1 October 2016|archive-date=24 September 2016|archive-url=https://web.archive.org/web/20160924212549/https://cloud.google.com/customers/dnastack/|url-status=live}}</ref>
* [[23andme]]'s [[DNA database]] contains the genetic information of over 1,000,000 people worldwide.<ref>{{cite web|title=23andMe – Ancestry|url=https://www.23andme.com/en-int/ancestry/|website=23andme.com|access-date=29 December 2016|archive-date=18 December 2016|archive-url=https://web.archive.org/web/20161218134337/https://www.23andme.com/en-int/ancestry/|url-status=live}}</ref> The company explores selling the "anonymous aggregated genetic data" to other researchers and pharmaceutical companies for research purposes if patients give their consent.<ref name=verge1>{{cite web|last1=Potenza|first1=Alessandra|title=23andMe wants researchers to use its kits, in a bid to expand its collection of genetic data|url=https://www.theverge.com/2016/7/13/12166960/23andme-genetic-testing-database-genotyping-research|website=The Verge|access-date=29 December 2016|date=13 July 2016|archive-date=29 December 2016|archive-url=https://web.archive.org/web/20161229171240/http://www.theverge.com/2016/7/13/12166960/23andme-genetic-testing-database-genotyping-research|url-status=live}}</ref><ref>{{cite magazine|title=This Startup Will Sequence Your DNA, So You Can Contribute To Medical Research|url=https://www.fastcompany.com/3066775/innovation-agents/this-startup-will-sequence-your-dna-so-you-can-contribute-to-medical-resea|magazine=[[Fast Company]]|access-date=29 December 2016|date=23 December 2016|archive-date=29 December 2016|archive-url=https://web.archive.org/web/20161229102128/https://www.fastcompany.com/3066775/innovation-agents/this-startup-will-sequence-your-dna-so-you-can-contribute-to-medical-resea|url-status=live}}</ref><ref>{{cite magazine|last1=Seife|first1=Charles|title=23andMe Is Terrifying, but Not for the Reasons the FDA Thinks|url=https://www.scientificamerican.com/article/23andme-is-terrifying-but-not-for-the-reasons-the-fda-thinks/|magazine=[[Scientific American]]|access-date=29 December 2016|archive-date=29 December 2016|archive-url=https://web.archive.org/web/20161229170606/https://www.scientificamerican.com/article/23andme-is-terrifying-but-not-for-the-reasons-the-fda-thinks/|url-status=live}}</ref><ref>{{cite web|last1=Zaleski|first1=Andrew|title=This biotech start-up is betting your genes will yield the next wonder drug|url=https://www.cnbc.com/2016/06/22/23andme-thinks-your-genes-are-the-key-to-blockbuster-drugs.html|publisher=CNBC|access-date=29 December 2016|date=22 June 2016|archive-date=29 December 2016|archive-url=https://web.archive.org/web/20161229101419/http://www.cnbc.com/2016/06/22/23andme-thinks-your-genes-are-the-key-to-blockbuster-drugs.html|url-status=live}}</ref><ref>{{cite magazine|last1=Regalado|first1=Antonio|title=How 23andMe turned your DNA into a $1 billion drug discovery machine|url=https://www.technologyreview.com/s/601506/23andme-sells-data-for-drug-search/|magazine=[[MIT Technology Review]]|access-date=29 December 2016|archive-date=29 December 2016|archive-url=https://web.archive.org/web/20161229100618/https://www.technologyreview.com/s/601506/23andme-sells-data-for-drug-search/|url-status=live}}</ref> Ahmad Hariri, professor of psychology and neuroscience at [[Duke University]] who has been using 23andMe in his research since 2009 states that the most important aspect of the company's new service is that it makes genetic research accessible and relatively cheap for scientists.<ref name=verge1/> A study that identified 15 genome sites linked to depression in 23andMe's database lead to a surge in demands to access the repository with 23andMe fielding nearly 20 requests to access the depression data in the two weeks after publication of the paper.<ref>{{cite web|title =23andMe reports jump in requests for data in wake of Pfizer depression study {{!}} FierceBiotech|url =http://www.fiercebiotech.com/it/23andme-reports-jump-requests-for-data-wake-pfizer-depression-study|website =fiercebiotech.com|date =22 August 2016|access-date =29 December 2016|archive-date =29 December 2016|archive-url =https://web.archive.org/web/20161229102058/http://www.fiercebiotech.com/it/23andme-reports-jump-requests-for-data-wake-pfizer-depression-study|url-status =live}}</ref>
*Computational fluid dynamics ([[Computational fluid dynamics|CFD]]) and hydrodynamic [[turbulence]] research generate massive data sets. The Johns Hopkins Turbulence Databases ([http://turbulence.pha.jhu.edu JHTDB]) contains over 350 terabytes of spatiotemporal fields from Direct Numerical simulations of various turbulent flows. Such data have been difficult to share using traditional methods such as downloading flat simulation output files. The data within JHTDB can be accessed using "virtual sensors" with various access modes ranging from direct web-browser queries, access through Matlab, Python, Fortran and C programs executing on clients' platforms, to cut out services to download raw data. The data have been used in over 150 scientific publications.

===Sports===
Big data can be used to improve training and understanding competitors, using sport sensors. It is also possible to predict winners in a match using big data analytics.<ref>{{cite web|url=http://www.itweb.co.za/index.php?option=com_content&view=article&id=147241|title=Data scientists predict Springbok defeat|author=Admire Moyo|work=itweb.co.za|date=23 October 2015|access-date=12 December 2015|archive-date=22 December 2015|archive-url=https://web.archive.org/web/20151222123957/http://www.itweb.co.za/index.php?option=com_content&view=article&id=147241|url-status=live}}</ref>
Future performance of players could be predicted as well.<ref>{{cite journal |last1=Bai |first1=Zhongbo |last2=Bai |first2=Xiaomei |title=Sports Big Data: Management, Analysis, Applications, and Challenges |journal=Complexity |date=2021 |volume=2021 |issue= |pages=1–11 |doi=10.1155/2021/6676297 |doi-access=free }}</ref> Thus, players' value and salary is determined by data collected throughout the season.<ref>{{cite web|url=http://www.itweb.co.za/index.php?option=com_content&view=article&id=147852|title=Predictive analytics, big data transform sports|author=Regina Pazvakavambwa|work=itweb.co.za|date=17 November 2015|access-date=12 December 2015|archive-date=22 December 2015|archive-url=https://web.archive.org/web/20151222085858/http://www.itweb.co.za/index.php?option=com_content&view=article&id=147852|url-status=live}}</ref>

In [[Formula One]] races, race cars with hundreds of sensors generate terabytes of data. These sensors collect data points from tire pressure to fuel burn efficiency.<ref>{{cite web|url=https://www.huffingtonpost.com/dave-ryan/sports-where-big-data-fin_b_8553884.html|title=Sports: Where Big Data Finally Makes Sense|author=Dave Ryan|work=huffingtonpost.com|date=13 November 2015|access-date=12 December 2015|archive-date=22 December 2015|archive-url=https://web.archive.org/web/20151222163418/http://www.huffingtonpost.com/dave-ryan/sports-where-big-data-fin_b_8553884.html|url-status=live}}</ref>
Based on the data, engineers and data analysts decide whether adjustments should be made in order to win a race. Besides, using big data, race teams try to predict the time they will finish the race beforehand, based on simulations using data collected over the season.<ref>{{cite magazine|url=https://www.forbes.com/sites/frankbi/2014/11/13/how-formula-one-teams-are-using-big-data-to-get-the-inside-edge/|title=How Formula One Teams Are Using Big Data To Get The Inside Edge|author=Frank Bi|magazine=Forbes|access-date=12 December 2015|archive-date=20 December 2015|archive-url=https://web.archive.org/web/20151220134302/http://www.forbes.com/sites/frankbi/2014/11/13/how-formula-one-teams-are-using-big-data-to-get-the-inside-edge/|url-status=live}}</ref>

===Technology===
* {{as of|2013}}, [[eBay.com]] uses two [[data warehouse]]s at 7.5 [[petabytes]] and 40PB as well as a 40PB [[Hadoop]] cluster for search, consumer recommendations, and merchandising.<ref>{{cite web | last=Tay | first=Liz | url=http://www.itnews.com.au/news/inside-ebay8217s-90pb-data-warehouse-342615 | title=Inside eBay's 90PB data warehouse | publisher=ITNews | access-date=12 February 2016 | archive-date=15 February 2016 | archive-url=https://web.archive.org/web/20160215123809/http://www.itnews.com.au/news/inside-ebay8217s-90pb-data-warehouse-342615 | url-status=live }}</ref>
* [[Amazon.com]] handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and {{as of|2005|lc=on}} they had the world's three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.<ref>{{cite web |last=Layton |first=Julia |url=http://money.howstuffworks.com/amazon1.htm |title=Amazon Technology |date=25 January 2006 |publisher=Money.howstuffworks.com |access-date=5 March 2013 |archive-date=28 February 2013 |archive-url=https://web.archive.org/web/20130228040458/http://money.howstuffworks.com/amazon1.htm |url-status=live }}</ref>
* [[Facebook]] handles 50&nbsp;billion photos from its user base.<ref>{{cite web | url=https://www.facebook.com/notes/facebook-engineering/scaling-facebook-to-500-million-users-and-beyond/409881258919 | title=Scaling Facebook to 500 Million Users and Beyond | publisher=Facebook.com | access-date=21 July 2013 | archive-date=5 July 2013 | archive-url=https://web.archive.org/web/20130705180356/https://www.facebook.com/notes/facebook-engineering/scaling-facebook-to-500-million-users-and-beyond/409881258919 | url-status=live }}</ref> {{as of|2017|June}}, Facebook reached 2 billion [[monthly active users]].<ref>{{Cite news|url=https://techcrunch.com/2017/06/27/facebook-2-billion-users/|title=Facebook now has 2 billion monthly users… and responsibility|last=Constine|first=Josh|date=27 June 2017|work=TechCrunch|access-date=3 September 2018|archive-date=27 December 2020|archive-url=https://web.archive.org/web/20201227043624/https://techcrunch.com/2017/06/27/facebook-2-billion-users/|url-status=live}}</ref>
* [[Google]] was handling roughly 100&nbsp;billion searches per month {{as of|2012|08|lc=on}}.<ref>{{cite web|url=http://searchengineland.com/google-1-trillion-searches-per-year-212940|title=Google Still Doing at Least 1 Trillion Searches Per Year|date=16 January 2015|work=Search Engine Land|access-date=15 April 2015|archive-date=15 April 2015|archive-url=https://web.archive.org/web/20150415121237/http://searchengineland.com/google-1-trillion-searches-per-year-212940|url-status=live}}</ref>

===COVID-19===
During the [[COVID-19 pandemic]], big data was raised as a way to minimise the impact of the disease. Significant applications of big data included minimising the spread of the virus, case identification and development of medical treatment.<ref>{{cite journal |last1=Haleem |first1=Abid |last2=Javaid |first2=Mohd |last3=Khan |first3=Ibrahim |last4=Vaishya |first4=Raju |title=Significant Applications of Big Data in COVID-19 Pandemic |journal=Indian Journal of Orthopaedics |date=2020 |volume=54 |issue=4 |pages=526–528 |doi=10.1007/s43465-020-00129-z |pmid=32382166 |pmc=7204193 }}</ref>

Governments used big data to track infected people to minimise spread. Early adopters included China, Taiwan, South Korea, and Israel.<ref>{{cite news |last1=Manancourt |first1=Vincent |title=Coronavirus tests Europe's resolve on privacy |url=https://www.politico.eu/article/coronavirus-tests-europe-resolve-on-privacy-tracking-apps-germany-italy/ |access-date=30 October 2020 |work=Politico |date=10 March 2020 |archive-date=20 March 2020 |archive-url=https://web.archive.org/web/20200320105744/https://www.politico.eu/article/coronavirus-tests-europe-resolve-on-privacy-tracking-apps-germany-italy/ |url-status=live }}</ref><ref>{{cite news |last1=Choudhury |first1=Amit Roy |title=Gov in the Time of Corona |url=https://govinsider.asia/innovation/gov-in-the-time-of-corona/ |access-date=30 October 2020 |work=Gov Insider |date=27 March 2020 |archive-date=20 March 2020 |archive-url=https://web.archive.org/web/20200320125215/https://govinsider.asia/innovation/gov-in-the-time-of-corona/ |url-status=live }}</ref><ref>{{cite news |last1=Cellan-Jones |first1=Rory |title=China launches coronavirus 'close contact detector' app |url=https://www.bbc.com/news/technology-51439401 |access-date=30 October 2020 |work=BBC |date=11 February 2020|archive-url=https://web.archive.org/web/20200228003957/https://www.bbc.com/news/technology-51439401 |archive-date=28 February 2020 }}</ref>

==Research activities==
Encrypted search and cluster formation in big data were demonstrated in March 2014 at the American Society of Engineering Education. Gautam Siwach engaged at ''Tackling the challenges of Big Data'' by [[MIT Computer Science and Artificial Intelligence Laboratory]] and Amir Esmailpour at the UNH Research Group investigated the key features of big data as the formation of clusters and their interconnections. They focused on the security of big data and the orientation of the term towards the presence of different types of data in an encrypted form at cloud interface by providing the raw definitions and real-time examples within the technology. Moreover, they proposed an approach for identifying the encoding technique to advance towards an expedited search over encrypted text leading to the security enhancements in big data.<ref>{{cite conference |url=http://asee-ne.org/proceedings/2014/Student%20Papers/210.pdf |title=Encrypted Search & Cluster Formation in Big Data |last1=Siwach |first1=Gautam |last2=Esmailpour |first2=Amir |date=March 2014 |conference=ASEE 2014 Zone I Conference |conference-url=http://ubconferences.org/ |location=[[University of Bridgeport]], [[Bridgeport, Connecticut|Bridgeport]], Connecticut, US |access-date=26 July 2014 |archive-url=https://web.archive.org/web/20140809045242/http://asee-ne.org/proceedings/2014/Student%20Papers/210.pdf |archive-date=9 August 2014 |url-status=dead }}</ref>

In March 2012, The White House announced a national "Big Data Initiative" that consisted of six federal departments and agencies committing more than $200&nbsp;million to big data research projects.<ref>{{cite web|title=Obama Administration Unveils "Big Data" Initiative:Announces $200 Million in New R&D Investments| url=https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf |url-status =live| archive-url =https://web.archive.org/web/20170121233309/https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf |via=[[NARA|National Archives]]|work=[[Office of Science and Technology Policy]]|archive-date=21 January 2017}}</ref>

The initiative included a National Science Foundation "Expeditions in Computing" grant of $10 million over five years to the AMPLab<ref>{{cite web |url=http://amplab.cs.berkeley.edu |title=AMPLab at the University of California, Berkeley |publisher=Amplab.cs.berkeley.edu |access-date=5 March 2013 |archive-date=6 May 2011 |archive-url=https://web.archive.org/web/20110506051646/http://amplab.cs.berkeley.edu/ |url-status=live }}</ref> at the University of California, Berkeley.<ref>{{cite web|title=NSF Leads Federal Efforts in Big Data|date=29 March 2012|publisher=National Science Foundation (NSF)|url=https://www.nsf.gov/news/news_summ.jsp?cntn_id=123607&org=NSF&from=news|access-date=6 April 2018|archive-date=31 March 2019|archive-url=https://web.archive.org/web/20190331113404/https://www.nsf.gov/news/news_summ.jsp?cntn_id=123607&org=NSF&from=news|url-status=live}}</ref> The AMPLab also received funds from [[DARPA]], and over a dozen industrial sponsors and uses big data to attack a wide range of problems from predicting traffic congestion<ref>{{cite conference|url=https://amplab.cs.berkeley.edu/publication/scaling-the-mobile-millennium-system-in-the-cloud-2/|author1=Timothy Hunter|date=October 2011|author2=Teodor Moldovan|author3=Matei Zaharia|author4=Justin Ma|author5=Michael Franklin|author6-link=Pieter Abbeel|author6=Pieter Abbeel|author7=Alexandre Bayen|title=Scaling the Mobile Millennium System in the Cloud|conference=|access-date=2 November 2012|archive-date=31 March 2019|archive-url=https://web.archive.org/web/20190331073928/https://amplab.cs.berkeley.edu/publication/scaling-the-mobile-millennium-system-in-the-cloud-2/|url-status=live}}</ref> to fighting cancer.<ref>{{cite news|title=Computer Scientists May Have What It Takes to Help Cure Cancer|author=David Patterson|work=The New York Times|date=5 December 2011|url=https://www.nytimes.com/2011/12/06/science/david-patterson-enlist-computer-scientists-in-cancer-fight.html|access-date=26 February 2017|archive-date=30 January 2017|archive-url=https://web.archive.org/web/20170130045503/http://www.nytimes.com/2011/12/06/science/david-patterson-enlist-computer-scientists-in-cancer-fight.html|url-status=live}}</ref>

The White House Big Data Initiative also included a commitment by the Department of Energy to provide $25 million in funding over five years to establish the Scalable Data Management, Analysis and Visualization (SDAV) Institute,<ref>{{cite web |title=Secretary Chu Announces New Institute to Help Scientists Improve Massive Data Set Research on DOE Supercomputers |publisher=energy.gov |url=http://energy.gov/articles/secretary-chu-announces-new-institute-help-scientists-improve-massive-data-set-research-doe |access-date=2 November 2012 |archive-date=3 April 2019 |archive-url=https://web.archive.org/web/20190403084845/https://www.energy.gov/articles/secretary-chu-announces-new-institute-help-scientists-improve-massive-data-set-research-doe |url-status=live }}</ref> led by the Energy Department's [[Lawrence Berkeley National Laboratory]]. The SDAV Institute aims to bring together the expertise of six national laboratories and seven universities to develop new tools to help scientists manage and visualize data on the department's supercomputers.

The U.S. state of [[Massachusetts]] announced the Massachusetts Big Data Initiative in May 2012, which provides funding from the state government and private companies to a variety of research institutions.<ref>{{Cite news|last=Young|first=Shannon|date=2012-05-30|title=Mass. governor, MIT announce big data initiative|work=Boston.com|url=http://archive.boston.com/news/local/massachusetts/articles/2012/05/30/mass_gov_and_mit_to_announce_data_initiative/|access-date=2021-07-29|archive-date=29 July 2021|archive-url=https://web.archive.org/web/20210729153535/http://archive.boston.com/news/local/massachusetts/articles/2012/05/30/mass_gov_and_mit_to_announce_data_initiative/|url-status=live}}</ref> The [[Massachusetts Institute of Technology]] hosts the Intel Science and Technology Center for Big Data in the [[MIT Computer Science and Artificial Intelligence Laboratory]], combining government, corporate, and institutional funding and research efforts.<ref>{{cite web |url=http://bigdata.csail.mit.edu/ |title=Big Data @ CSAIL |publisher=Bigdata.csail.mit.edu |date=22 February 2013 |access-date=5 March 2013 |archive-date=30 March 2013 |archive-url=https://web.archive.org/web/20130330022413/http://bigdata.csail.mit.edu/ |url-status=dead }}</ref>

The European Commission is funding the two-year-long Big Data Public Private Forum through their Seventh Framework Program to engage companies, academics and other stakeholders in discussing big data issues. The project aims to define a strategy in terms of research and innovation to guide supporting actions from the European Commission in the successful implementation of the big data economy. Outcomes of this project will be used as input for [[Horizon 2020]], their next [[Framework Programmes for Research and Technological Development|framework program]].<ref>{{cite web |url=https://cordis.europa.eu/project/id/318062 |title=Big Data Public Private Forum |publisher=cordis.europa.eu |date=1 September 2012 |access-date=16 March 2020 |archive-date=9 March 2021 |archive-url=https://web.archive.org/web/20210309035521/https://cordis.europa.eu/project/id/318062 |url-status=live }}</ref>

The British government announced in March 2014 the founding of the [[Alan Turing Institute]], named after the computer pioneer and code-breaker, which will focus on new ways to collect and analyze large data sets.<ref>{{cite news|url=https://www.bbc.co.uk/news/technology-26651179|title=Alan Turing Institute to be set up to research big data|work=[[BBC News]]|access-date=19 March 2014|date=19 March 2014|archive-date=18 August 2021|archive-url=https://web.archive.org/web/20210818142000/https://www.bbc.co.uk/news/technology-26651179|url-status=live}}</ref>

At the [[University of Waterloo Stratford Campus]] Canadian Open Data Experience (CODE) Inspiration Day, participants demonstrated how using data visualization can increase the understanding and appeal of big data sets and communicate their story to the world.<ref>{{cite web|url= http://www.betakit.com/event/inspiration-day-at-university-of-waterloo-stratford-campus/|title= Inspiration day at University of Waterloo, Stratford Campus|publisher= betakit.com/|access-date= 28 February 2014|archive-date= 26 February 2014|archive-url= https://web.archive.org/web/20140226181442/http://www.betakit.com/event/inspiration-day-at-university-of-waterloo-stratford-campus/|url-status= dead}}</ref>

[[Computational social science]]s&nbsp;– Anyone can use application programming interfaces (APIs) provided by big data holders, such as Google and Twitter, to do research in the social and behavioral sciences.<ref name=pigdata>{{cite journal|last=Reips|first=Ulf-Dietrich|author2=Matzat, Uwe|title=Mining "Big Data" using Big Data Services|journal=International Journal of Internet Science|year=2014|volume=1|issue=1|pages=1–8|url=http://www.ijis.net/ijis9_1/ijis9_1_editorial_pre.html|access-date=14 August 2014|archive-date=14 August 2014|archive-url=https://web.archive.org/web/20140814194342/http://www.ijis.net/ijis9_1/ijis9_1_editorial_pre.html|url-status=live}}</ref> Often these APIs are provided for free.<ref name="pigdata" /> [[Tobias Preis]] et al. used [[Google Trends]] data to demonstrate that Internet users from countries with a higher per capita gross domestic products (GDPs) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviors and real-world economic indicators.<ref>{{cite journal | vauthors = Preis T, Moat HS, Stanley HE, Bishop SR | title = Quantifying the advantage of looking forward | journal = Scientific Reports | volume = 2 | pages = 350 | year = 2012 | pmid = 22482034 | pmc = 3320057 | doi = 10.1038/srep00350 | bibcode = 2012NatSR...2..350P }}</ref><ref>{{cite news | url=https://www.newscientist.com/article/dn21678-online-searches-for-future-linked-to-economic-success.html | title=Online searches for future linked to economic success | first=Paul | last=Marks | work=New Scientist | date=5 April 2012 | access-date=9 April 2012 | archive-date=8 April 2012 | archive-url=https://web.archive.org/web/20120408184531/http://www.newscientist.com/article/dn21678-online-searches-for-future-linked-to-economic-success.html | url-status=live }}</ref><ref>{{cite news | url=https://arstechnica.com/gadgets/news/2012/04/google-trends-reveals-clues-about-the-mentality-of-richer-nations.ars | title=Google Trends reveals clues about the mentality of richer nations | first=Casey | last=Johnston | work=Ars Technica | date=6 April 2012 | access-date=9 April 2012 | archive-date=7 April 2012 | archive-url=https://web.archive.org/web/20120407171138/http://arstechnica.com/gadgets/news/2012/04/google-trends-reveals-clues-about-the-mentality-of-richer-nations.ars | url-status=live }}</ref> The authors of the study examined Google queries logs made by ratio of the volume of searches for the coming year (2011) to the volume of searches for the previous year (2009), which they call the "[[future orientation index]]".<ref>{{cite web | url = http://www.tobiaspreis.de/bigdata/future_orientation_index.pdf | title = Supplementary Information: The Future Orientation Index is available for download | author = Tobias Preis | date = 24 May 2012 | access-date = 24 May 2012 | archive-date = 17 January 2013 | archive-url = https://web.archive.org/web/20130117211931/http://www.tobiaspreis.de/bigdata/future_orientation_index.pdf | url-status = live }}</ref> They compared the future orientation index to the per capita GDP of each country, and found a strong tendency for countries where Google users inquire more about the future to have a higher GDP.

[[Tobias Preis]] and his colleagues Helen Susannah Moat and [[H. Eugene Stanley]] introduced a method to identify online precursors for stock market moves, using trading strategies based on search volume data provided by Google Trends.<ref>{{cite journal | url=http://www.nature.com/news/counting-google-searches-predicts-market-movements-1.12879 | title=Counting Google searches predicts market movements | author=Philip Ball | journal=Nature | date=26 April 2013 | doi=10.1038/nature.2013.12879 | s2cid=167357427 | access-date=9 August 2013 | author-link=Philip Ball | archive-date=27 September 2013 | archive-url=https://web.archive.org/web/20130927152926/http://www.nature.com/news/counting-google-searches-predicts-market-movements-1.12879 | url-status=live }}</ref> Their analysis of [[Google]] search volume for 98 terms of varying financial relevance, published in ''[[Scientific Reports]]'',<ref>{{cite journal | vauthors = Preis T, Moat HS, Stanley HE | title = Quantifying trading behavior in financial markets using Google Trends | journal = Scientific Reports | volume = 3 | pages = 1684 | year = 2013 | pmid = 23619126 | pmc = 3635219 | doi = 10.1038/srep01684 | bibcode = 2013NatSR...3.1684P }}</ref> suggests that increases in search volume for financially relevant search terms tend to precede large losses in financial markets.<ref>{{cite news | url=http://bits.blogs.nytimes.com/2013/04/26/google-search-terms-can-predict-stock-market-study-finds/ | title=Google Search Terms Can Predict Stock Market, Study Finds | author=Nick Bilton | work=[[The New York Times]] | date=26 April 2013 | access-date=9 August 2013 | archive-date=2 June 2013 | archive-url=https://web.archive.org/web/20130602104710/http://bits.blogs.nytimes.com/2013/04/26/google-search-terms-can-predict-stock-market-study-finds/ | url-status=live }}</ref><ref>{{cite magazine | url=https://business.time.com/2013/04/26/trouble-with-your-investment-portfolio-google-it/ | title=Trouble With Your Investment Portfolio? Google It! | author=Christopher Matthews | magazine=[[Time (magazine)|Time]] | date=26 April 2013 | access-date=9 August 2013 | archive-date=21 August 2013 | archive-url=https://web.archive.org/web/20130821075908/http://business.time.com/2013/04/26/trouble-with-your-investment-portfolio-google-it/ | url-status=live }}</ref><ref>{{cite journal | url=http://www.nature.com/news/counting-google-searches-predicts-market-movements-1.12879 | title=Counting Google searches predicts market movements | author=Philip Ball | journal=[[Nature (journal)|Nature]] | date=26 April 2013 | doi=10.1038/nature.2013.12879 | s2cid=167357427 | access-date=9 August 2013 | archive-date=27 September 2013 | archive-url=https://web.archive.org/web/20130927152926/http://www.nature.com/news/counting-google-searches-predicts-market-movements-1.12879 | url-status=live }}</ref><ref>{{cite news | url=http://www.businessweek.com/articles/2013-04-25/big-data-researchers-turn-to-google-to-beat-the-markets | title='Big Data' Researchers Turn to Google to Beat the Markets | author=Bernhard Warner | work=[[Bloomberg Businessweek]] | date=25 April 2013 | access-date=9 August 2013 | archive-date=23 July 2013 | archive-url=https://web.archive.org/web/20130723195732/http://www.businessweek.com/articles/2013-04-25/big-data-researchers-turn-to-google-to-beat-the-markets | url-status=dead }}</ref><ref>{{cite news | url=https://www.independent.co.uk/news/business/comment/hamish-mcrae/hamish-mcrae-need-a-valuable-handle-on-investor-sentiment-google-it-8590991.html | title=Hamish McRae: Need a valuable handle on investor sentiment? Google it | author=Hamish McRae | work=[[The Independent]] | date=28 April 2013 | access-date=9 August 2013 | location=London | archive-date=25 July 2018 | archive-url=https://web.archive.org/web/20180725072017/https://www.independent.co.uk/news/business/comment/hamish-mcrae/hamish-mcrae-need-a-valuable-handle-on-investor-sentiment-google-it-8590991.html | url-status=live }}</ref><ref>{{cite web | url=http://www.ft.com/intl/cms/s/0/e5d959b8-acf2-11e2-b27f-00144feabdc0.html | title=Google search proves to be new word in stock market prediction | author=Richard Waters | work=[[Financial Times]] | date=25 April 2013 | access-date=9 August 2013 | archive-date=1 June 2022 | archive-url=https://web.archive.org/web/20220601224737/https://www.ft.com/content/e5d959b8-acf2-11e2-b27f-00144feabdc0 | url-status=live }}</ref><ref>{{cite news | url=https://www.bbc.co.uk/news/science-environment-22293693 | title=Google searches predict market moves | author=Jason Palmer | work=[[BBC]] | date=25 April 2013 | access-date= 9 August 2013 | archive-date=5 June 2013 | archive-url=https://web.archive.org/web/20130605173345/http://www.bbc.co.uk/news/science-environment-22293693 | url-status=live }}</ref>

Big data sets come with algorithmic challenges that previously did not exist. Hence, there is seen by some to be a need to fundamentally change the processing ways.<ref>E. Sejdić (March 2014). "Adapt current tools for use with big data". ''Nature''. '''507''' (7492): 306.</ref>

===Sampling big data===
A research question that is asked about big data sets is whether it is necessary to look at the full data to draw certain conclusions about the properties of the data or if is a sample is good enough. The name big data itself contains a term related to size and this is an important characteristic of big data. But [[Sampling (statistics)|sampling]] enables the selection of right data points from within the larger data set to estimate the characteristics of the whole population. In manufacturing different types of sensory data such as acoustics, vibration, pressure, current, voltage, and controller data are available at short time intervals. To predict downtime it may not be necessary to look at all the data but a sample may be sufficient. Big data can be broken down by various data point categories such as demographic, psychographic, behavioral, and transactional data. With large sets of data points, marketers are able to create and use more customized segments of consumers for more strategic targeting.

==Critique==
Critiques of the big data paradigm come in two flavors: those that question the implications of the approach itself, and those that question the way it is currently done.<ref name="Kimble and Milolidakis (2015)">{{Cite Q|Q56532925}}</ref> One approach to this criticism is the field of [[critical data studies]].

===Critiques of the big data paradigm===
"A crucial problem is that we do not know much about the underlying empirical micro-processes that lead to the emergence of the[se] typical network characteristics of Big Data."{{Sfn | Snijders | Matzat | Reips | 2012}}{{Page needed |date=December 2023}}  In their critique, Snijders, Matzat, and [[Ulf-Dietrich Reips|Reips]] point out that often very strong assumptions are made about mathematical properties that may not at all reflect what is really going on at the level of micro-processes. Mark Graham has leveled broad critiques at [[Chris Anderson (writer)|Chris Anderson]]'s assertion that big data will spell the end of theory:<ref>{{Cite magazine|url=https://www.wired.com/science/discoveries/magazine/16-07/pb_theory|title=The End of Theory: The Data Deluge Makes the Scientific Method Obsolete |author=Chris Anderson|date=23 June 2008|magazine=Wired|access-date=5 March 2017|archive-date=27 March 2014|archive-url= https://web.archive.org/web/20140327145018/http://www.wired.com/science/discoveries/magazine/16-07/pb_theory/|url-status=live}}</ref> focusing in particular on the notion that big data must always be contextualized in their social, economic, and political contexts.<ref>{{cite news |author=Graham M. |title=Big data and the end of theory? |newspaper=The Guardian |url= https://www.theguardian.com/news/datablog/2012/mar/09/big-data-theory |location=London |date=9 March 2012 |access-date=14 December 2016 |archive-date=24 July 2013 |archive-url= https://web.archive.org/web/20130724152353/http://www.guardian.co.uk/news/datablog/2012/mar/09/big-data-theory |url-status=live }}</ref> Even as companies invest eight- and nine-figure sums to derive insight from information streaming in from suppliers and customers, less than 40% of employees have sufficiently mature processes and skills to do so. To overcome this insight deficit, big data, no matter how comprehensive or well analyzed, must be complemented by "big judgment", according to an article in the ''[[Harvard Business Review]]''.<ref>{{cite journal |title= Good Data Won't Guarantee Good Decisions |journal= [[Harvard Business Review]]|url=http://hbr.org/2012/04/good-data-wont-guarantee-good-decisions/ar/1|author=Shah, Shvetank |author2= Horne, Andrew|author3=Capellá, Jaime|access-date= 8 September 2012|date=April 2012|archive-date=11 September 2012|archive-url=https://web.archive.org/web/20120911235259/http://hbr.org/2012/04/good-data-wont-guarantee-good-decisions/ar/1 |url-status= live}}</ref>

Much in the same line, it has been pointed out that the decisions based on the analysis of big data are inevitably "informed by the world as it was in the past, or, at best, as it currently is".{{Sfn | Hilbert | 2016}}{{Page needed |date=December 2023}} Fed by a large number of data on past experiences, algorithms can predict future development if the future is similar to the past.<ref name="HilbertTEDx">[https://www.youtube.com/watch?v=UXef6yfJZAI Big Data requires Big Visions for Big Change.] {{Webarchive|url=https://web.archive.org/web/20161202010931/https://www.youtube.com/watch?v=UXef6yfJZAI |date=2 December 2016 }}, Hilbert, M. (2014). London: TEDx UCL, x=independently organized TED talks</ref> If the system's dynamics of the future change (if it is not a [[stationary process]]), the past can say little about the future. In order to make predictions in changing environments, it would be necessary to have a thorough understanding of the systems dynamic, which requires theory.<ref name="HilbertTEDx"/> As a response to this critique Alemany Oliver and Vayre suggest to use "abductive reasoning as a first step in the research process in order to bring context to consumers' digital traces and make new theories emerge".<ref>{{cite journal|last=Alemany Oliver|first=Mathieu |author2=Vayre, Jean-Sebastien |s2cid=111360835 |title= Big Data and the Future of Knowledge Production in Marketing Research: Ethics, Digital Traces, and Abductive Reasoning|journal=Journal of Marketing Analytics |year=2015|volume=3|issue=1|doi= 10.1057/jma.2015.1|pages=5–13}}</ref>  Additionally, it has been suggested to combine big data approaches with computer simulations, such as [[agent-based model]]s{{Sfn | Hilbert | 2016}}{{Page needed |date=December 2023}} and [[complex systems]]. Agent-based models are increasingly getting better in predicting the outcome of social complexities of even unknown future scenarios through computer simulations that are based on a collection of mutually interdependent algorithms.<ref>{{cite web |url= https://www.theatlantic.com/magazine/archive/2002/04/seeing-around-corners/302471/|title=Seeing Around Corners|author=Jonathan Rauch |date= 1 April 2002|work=[[The Atlantic]]|access-date=5 March 2017|archive-date=4 April 2017|archive-url=https://web.archive.org/web/20170404134619/https://www.theatlantic.com/magazine/archive/2002/04/seeing-around-corners/302471/|url-status= live}}</ref><ref>Epstein, J. M., & Axtell, R. L. (1996). Growing Artificial Societies: Social Science from the Bottom Up. A Bradford Book.</ref> Finally, the use of multivariate methods that probe for the latent structure of the data, such as [[factor analysis]] and [[cluster analysis]], have proven useful as analytic approaches that go well beyond the bi-variate approaches (e.g. [[contingency table]]s) typically employed with smaller data sets.

In health and biology, conventional scientific approaches are based on experimentation. For these approaches, the limiting factor is the relevant data that can confirm or refute the initial hypothesis.<ref>{{cite web|url=http://www.bigdataparis.com/documents/Pierre-Delort-INSERM.pdf#page=5|title=Delort P., Big data in Biosciences, Big Data Paris, 2012 |website= Bigdata Paris |access-date= 8 October 2017|archive-date=30 July 2016|archive-url=https://web.archive.org/web/20160730211146/http://www.bigdataparis.com/documents/Pierre-Delort-INSERM.pdf#page=5|url-status= dead}}</ref>  A new postulate is accepted now in biosciences: the information provided by the data in huge volumes ([[omics]]) without prior hypothesis is complementary and sometimes necessary to conventional approaches based on experimentation.<ref>{{cite web|url=https://www.cs.cmu.edu/~durand/03-711/2011/Literature/Next-Gen-Genomics-NRG-2010.pdf|title=Next-generation genomics: an integrative approach|date=July 2010|publisher=nature|access-date=18 October 2016|archive-date=13 August 2017|archive-url=https://web.archive.org/web/20170813145203/http://www.cs.cmu.edu/%7Edurand/03-711/2011/Literature/Next-Gen-Genomics-NRG-2010.pdf|url-status=live}}</ref><ref>{{cite web|url=https://www.researchgate.net/publication/283298499|title=Big Data in Biosciences|date=October 2015|access-date=18 October 2016|archive-date=1 June 2022|archive-url=https://web.archive.org/web/20220601224842/https://www.researchgate.net/publication/283298499|url-status=live}}</ref> In the massive approaches it is the formulation of a relevant hypothesis to explain the data that is the limiting factor.<ref>{{cite news|url=https://next.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0|title=Big data: are we making a big mistake?|date=28 March 2014|work=Financial Times|access-date=20 October 2016|archive-date=30 June 2016|archive-url=https://web.archive.org/web/20160630160149/https://next.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0|url-status=live}}</ref> The search logic is reversed and the limits of induction ("Glory of Science and Philosophy scandal", [[C. D. Broad]], 1926) are to be considered.{{Citation needed|date=April 2015}}

[[Consumer privacy|Privacy]] advocates are concerned about the threat to privacy represented by increasing storage and integration of [[personally identifiable information]]; expert panels have released various policy recommendations to conform practice to expectations of privacy.<ref>{{cite magazine |first=Paul |last=Ohm |title=Don't Build a Database of Ruin |magazine=Harvard Business Review |url=http://blogs.hbr.org/cs/2012/08/dont_build_a_database_of_ruin.html |date=23 August 2012 |access-date=29 August 2012 |archive-date=30 August 2012 |archive-url=https://web.archive.org/web/20120830045712/http://blogs.hbr.org/cs/2012/08/dont_build_a_database_of_ruin.html |url-status=live }}</ref> The misuse of big data in several cases by media, companies, and even the government has allowed for abolition of trust in almost every fundamental institution holding up society.<ref>Bond-Graham, Darwin (2018). [https://www.theperspective.com/debates/the-perspective-on-big-data/ "The Perspective on Big Data"] {{Webarchive|url=https://web.archive.org/web/20201109030023/https://www.theperspective.com/debates/the-perspective-on-big-data/ |date=9 November 2020 }}. [[The Perspective]].</ref>

Barocas and Nissenbaum argue that one way of protecting individual users is by being informed about the types of information being collected, with whom it is shared, under what constraints and for what purposes.<ref>{{Cite book|title=Big Data's End Run around Anonymity and Consent| last1 =Barocas |first1=Solon |last2=Nissenbaum |first2=Helen|last3=Lane|first3=Julia|last4=Stodden|first4=Victoria|last5=Bender|first5=Stefan|last6=Nissenbaum|first6=Helen| s2cid =152939392|date=June 2014| publisher =Cambridge University Press|isbn=9781107067356|pages=44–75|doi =10.1017/cbo9781107590205.004}}</ref>

===Critiques of the "V" model===
The "V" model of big data is concerning as it centers around computational scalability and lacks in a loss around the perceptibility and understandability of information. This led to the framework of [[cognitive big data]], which characterizes big data applications according to:<ref name=CSO_1>{{cite web| author1=Lugmayr, A.| author2=Stockleben, B| author3=Scheib, C.| author4=Mailaparampil, M.| author5=Mesia, N.|author6=Ranta, H.| author7=Lab, E.| title=A Comprehensive Survey On Big-Data Research and Its Implications – What is Really 'New' in Big Data? – It's Cognitive Big Data!| url=https://www.researchgate.net/publication/304784955| date=1 June 2016| access-date=27 November 2023| archive-date=1 June 2022| archive-url=https://web.archive.org/web/20220601224740/https://www.researchgate.net/publication/304784955| url-status=live}}</ref>
* Data completeness: understanding of the non-obvious from data
* Data correlation, causation, and predictability: causality as not essential requirement to achieve predictability
* Explainability and interpretability: humans desire to understand and accept what they understand, where algorithms do not cope with this
* Level of [[automated decision-making]]: algorithms that support automated decision making and algorithmic self-learning

===Critiques of novelty===
Large data sets have been analyzed by computing machines for well over a century, including the US census analytics performed by [[IBM]]'s punch-card machines which computed statistics including means and variances of populations across the whole continent. In more recent decades, science experiments such as [[CERN]] have produced data on similar scales to current commercial "big data". However, science experiments have tended to analyze their data using specialized custom-built [[high-performance computing]] (super-computing) clusters and grids, rather than clouds of cheap commodity computers as in the current commercial wave, implying a difference in both culture and technology stack.

===Critiques of big data execution===
[[Ulf-Dietrich Reips]] and Uwe Matzat wrote in 2014 that big data had become a "fad" in scientific research.<ref name="pigdata" /> Researcher [[Danah Boyd]] has raised concerns about the use of big data in science neglecting principles such as choosing a [[Sampling (statistics)|representative sample]] by being too concerned about handling the huge amounts of data.<ref name="danah">{{cite web | url=http://www.danah.org/papers/talks/2010/WWW2010.html | title=Privacy and Publicity in the Context of Big Data | author=Danah Boyd | work=[[World Wide Web Conference|WWW 2010 conference]] | date=29 April 2010 | access-date=18 April 2011 | author-link=danah boyd | archive-date=22 October 2018 | archive-url=https://web.archive.org/web/20181022043331/http://www.danah.org/papers/talks/2010/WWW2010.html | url-status=live }}</ref> This approach may lead to results that have a [[Bias (statistics)|bias]] in one way or another.<ref>{{Cite journal|last=Katyal|first=Sonia K.|date=2019|title=Artificial Intelligence, Advertising, and Disinformation|url=https://muse.jhu.edu/article/745987|journal=Advertising & Society Quarterly|language=en|volume=20|issue=4|doi=10.1353/asr.2019.0026|s2cid=213397212|issn=2475-1790|access-date=18 November 2020|archive-date=28 October 2020|archive-url=https://web.archive.org/web/20201028213611/https://muse.jhu.edu/article/745987|url-status=live}}</ref> Integration across heterogeneous data resources—some that might be considered big data and others not—presents formidable logistical as well as analytical challenges, but many researchers argue that such integrations are likely to represent the most promising new frontiers in science.<ref>{{cite journal |last1=Jones |first1=MB |last2=Schildhauer |first2=MP |last3=Reichman |first3=OJ |last4=Bowers |first4=S |title=The New Bioinformatics: Integrating Ecological Data from the Gene to the Biosphere |journal=Annual Review of Ecology, Evolution, and Systematics |volume=37 |issue=1 |pages=519–544 |year=2006 |doi=10.1146/annurev.ecolsys.37.091305.110031 |url=http://www.pnamp.org/sites/default/files/Jones2006_AREES.pdf |access-date=19 September 2012 |archive-date=8 July 2019 |archive-url=https://web.archive.org/web/20190708212441/https://www.pnamp.org/sites/default/files/Jones2006_AREES.pdf |url-status=live }}</ref>
In the provocative article "Critical Questions for Big Data",<ref name="danah2">{{cite journal | doi = 10.1080/1369118X.2012.678878| title = Critical Questions for Big Data| journal = Information, Communication & Society| volume = 15| issue = 5| pages = 662–679| year = 2012| last1 = Boyd | first1 = D. | last2 = Crawford | first2 = K. | s2cid = 51843165| hdl = 10983/1320| hdl-access = free}}</ref> the authors title big data a part of [[mythology]]: "large data sets offer a higher form of intelligence and knowledge [...], with the aura of truth, objectivity, and accuracy". Users of big data are often "lost in the sheer volume of numbers", and "working with Big Data is still subjective, and what it quantifies does not necessarily have a closer claim on objective truth".<ref name="danah2" /> Recent developments in BI domain, such as pro-active reporting especially target improvements in the usability of big data, through automated [[Filter (software)|filtering]] of [[spurious relationship|non-useful data and correlations]].<ref name="Big Decisions White Paper">[http://www.fortewares.com/Administrator/userfiles/Banner/forte-wares--pro-active-reporting_EN.pdf Failure to Launch: From Big Data to Big Decisions] {{Webarchive|url=https://web.archive.org/web/20161206145026/http://www.fortewares.com/Administrator/userfiles/Banner/forte-wares--pro-active-reporting_EN.pdf |date=6 December 2016 }}, Forte Wares.</ref> Big structures are full of spurious correlations<ref>{{Cite web | url=https://www.tylervigen.com/spurious-correlations | title=15 Insane Things That Correlate with Each Other | access-date=27 June 2019 | archive-date=27 June 2019 | archive-url=https://web.archive.org/web/20190627155800/http://tylervigen.com/spurious-correlations | url-status=live }}</ref> either because of non-causal coincidences ([[law of truly large numbers]]), solely nature of big randomness<ref>{{Cite web |url=https://onlinelibrary.wiley.com/loi/10982418 |title=Random structures & algorithms |doi=10.1002/(ISSN)1098-2418 |access-date=27 June 2019 |archive-date=27 June 2019 |archive-url=https://web.archive.org/web/20190627171127/https://onlinelibrary.wiley.com/loi/10982418 |url-status=live }}</ref> ([[Ramsey theory]]), or existence of [[confounding factor|non-included factors]] so the hope, of early experimenters to make large databases of numbers "speak for themselves" and revolutionize scientific method, is questioned.<ref>Cristian S. Calude, Giuseppe Longo, (2016), The Deluge of Spurious Correlations in Big Data, [[Foundations of Science]]</ref> [[Catherine Tucker]] has pointed to "hype" around big data, writing "By itself, big data is unlikely to be valuable." The article explains: "The many contexts where data is cheap relative to the cost of retaining talent to process it, suggests that processing skills are more important than data itself in creating value for a firm."<ref>Anja Lambrecht and Catherine Tucker (2016) "The 4 Mistakes Most Managers Make with Analytics," ''Harvard Business Review'', July 12. https://hbr.org/2016/07/the-4-mistakes-most-managers-make-with-analytics {{Webarchive|url=https://web.archive.org/web/20220126145830/https://hbr.org/2016/07/the-4-mistakes-most-managers-make-with-analytics |date=26 January 2022 }}</ref>

Big data analysis is often shallow compared to analysis of smaller data sets.<ref name="kdnuggets-berchthold">{{cite web|url=http://www.kdnuggets.com/2014/08/interview-michael-berthold-knime-research-big-data-privacy-part2.html|title=Interview: Michael Berthold, KNIME Founder, on Research, Creativity, Big Data, and Privacy, Part 2|date=12 August 2014|author=Gregory Piatetsky|author-link=Gregory I. Piatetsky-Shapiro|publisher=KDnuggets|access-date=13 August 2014|archive-date=13 August 2014|archive-url=https://web.archive.org/web/20140813212203/http://www.kdnuggets.com/2014/08/interview-michael-berthold-knime-research-big-data-privacy-part2.html|url-status=live}}</ref> In many big data projects, there is no large data analysis happening, but the challenge is the [[extract, transform, load]] part of data pre-processing.<ref name="kdnuggets-berchthold" />

Big data is a [[buzzword]] and a "vague term",<ref>{{cite news|last1=Pelt|first1=Mason|title="Big Data" is an over used buzzword and this Twitter bot proves it|url=http://siliconangle.com/blog/2015/10/26/big-data-is-an-over-used-buzzword-and-this-twitter-bot-proves-it/|newspaper=Siliconangle|access-date=4 November 2015|date=26 October 2015|archive-date=30 October 2015|archive-url=https://web.archive.org/web/20151030044002/http://siliconangle.com/blog/2015/10/26/big-data-is-an-over-used-buzzword-and-this-twitter-bot-proves-it/|url-status=live}}</ref><ref name="ft-harford">{{cite web |url=http://www.ft.com/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabdc0.html |title=Big data: are we making a big mistake? |last1=Harford |first1=Tim |date=28 March 2014 |website=[[Financial Times]] |access-date=7 April 2014 |archive-date=7 April 2014 |archive-url=https://web.archive.org/web/20140407024216/http://www.ft.com/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabdc0.html |url-status=live }}</ref> but at the same time an "obsession"<ref name="ft-harford" /> with entrepreneurs, consultants, scientists, and the media. Big data showcases such as [[Google Flu Trends]] failed to deliver good predictions in recent years, overstating the flu outbreaks by a factor of two. Similarly, [[Academy Awards]] and election predictions solely based on Twitter were more often off than on target.
Big data often poses the same challenges as small data; adding more data does not solve problems of bias, but may emphasize other problems. In particular data sources such as Twitter are not representative of the overall population, and results drawn from such sources may then lead to wrong conclusions. [[Google Translate]]—which is based on big data statistical analysis of text—does a good job at translating web pages. However, results from specialized domains may be dramatically skewed.
On the other hand, big data may also introduce new problems, such as the [[multiple comparisons problem]]: simultaneously testing a large set of hypotheses is likely to produce many false results that mistakenly appear significant.
Ioannidis argued that "most published research findings are false"<ref name="Ioannidis">{{cite journal | vauthors = Ioannidis JP | title = Why most published research findings are false | journal = PLOS Medicine | volume = 2 | issue = 8 | pages = e124 | date = August 2005 | pmid = 16060722 | pmc = 1182327 | doi = 10.1371/journal.pmed.0020124 | author-link1 = John P. A. Ioannidis | doi-access = free }}</ref> due to essentially the same effect: when many scientific teams and researchers each perform many experiments (i.e. process a big amount of scientific data; although not with big data technology), the likelihood of a "significant" result being false grows fast – even more so, when only positive results are published.
<!-- sorry, this started overlapping with above section more and more... merging is welcome; I already dropped the intended subheadline "Hype cycle and inflated expectations". -->
Furthermore, big data analytics results are only as good as the model on which they are predicated. In an example, big data took part in attempting to predict the results of the 2016 U.S. presidential election<ref>{{Cite news|url=https://www.nytimes.com/2016/11/10/technology/the-data-said-clinton-would-win-why-you-shouldnt-have-believed-it.html|title=How Data Failed Us in Calling an Election|last1=Lohr|first1=Steve|date=10 November 2016|last2=Singer|first2=Natasha|newspaper=The New York Times|issn=0362-4331|access-date=27 November 2016|archive-date=25 November 2016|archive-url=https://web.archive.org/web/20161125203904/http://www.nytimes.com/2016/11/10/technology/the-data-said-clinton-would-win-why-you-shouldnt-have-believed-it.html|url-status=live}}</ref> with varying degrees of success.

=== Critiques of big data policing and surveillance ===
Big data has been used in policing and surveillance by institutions like [[Law enforcement in the United States|law enforcement]] and [[Corporate surveillance|corporations]].<ref>{{Cite news|url=https://www.economist.com/open-future/2018/06/04/how-data-driven-policing-threatens-human-freedom|title=How data-driven policing threatens human freedom|date=4 June 2018|newspaper=The Economist|access-date=27 October 2019|issn=0013-0613|archive-date=27 October 2019|archive-url=https://web.archive.org/web/20191027164113/https://www.economist.com/open-future/2018/06/04/how-data-driven-policing-threatens-human-freedom|url-status=live}}</ref> Due to the less visible nature of data-based surveillance as compared to traditional methods of policing, objections to big data policing are less likely to arise. According to Sarah Brayne's ''Big Data Surveillance: The Case of Policing'',<ref>{{Cite journal|last=Brayne|first=Sarah|s2cid=3609838|date=29 August 2017|title=Big Data Surveillance: The Case of Policing|journal=American Sociological Review |volume=82|issue=5|pages=977–1008|language=en|doi=10.1177/0003122417725865|pmid=38322733 |pmc=10846878}}</ref> big data policing can reproduce existing [[Social inequality|societal inequalities]] in three ways:

* Placing people under increased surveillance by using the justification of a mathematical and therefore unbiased algorithm
* Increasing the scope and number of people that are subject to law enforcement tracking and exacerbating existing [[Race in the United States criminal justice system#Racial inequality in incarceration|racial overrepresentation]] in the criminal justice system
* Encouraging members of society to abandon interactions with institutions that would create a digital trace, thus creating obstacles to social inclusion

If these potential problems are not corrected or regulated, the effects of big data policing may continue to shape societal hierarchies. Conscientious usage of big data policing could prevent individual level biases from becoming institutional biases, Brayne also notes.

== See also ==

{{Category see also|LABEL=For a list of companies, and tools, see also|Big data}}
<!-- NO COMPANIES OR TOOL SPAM HERE. That would be an endless list! "See also" concepts, not linked above. -->
{{columns-list|colwidth=15em|
*{{annotated link|Big data ethics}}
*{{annotated link|Big data maturity model}}
*{{annotated link|Big memory}}
*{{annotated link|Data curation}}
*{{annotated link|Data defined storage}}
*{{annotated link|Data engineering}}
*{{annotated link|Data lineage}}
*{{annotated link|Data philanthropy}}
*{{annotated link|Data science}}
*{{annotated link|Datafication}}
*{{annotated link|Document-oriented database}}
*{{annotated link|List of big data companies}}
*{{annotated link|Very large database}}
*{{annotated link|Topological data analysis}}
*{{annotated link|XLDB}}
}}

== References ==
{{Reflist
|refs =
<!-- unused<ref name="2017-07-18_Gartner">{{cite web | url = https://research.gartner.com/definition-whatis-big-data | work = IT Glossary | title = What is Big Data? | publisher = [[Gartner]] | access-date = 18 July 2017 | archive-url = https://web.archive.org/web/20170718161704/https://research.gartner.com/definition-whatis-big-data | archive-date = 18 July 2017 | quote = Big Data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.}}</ref> -->
}}

== Bibliography ==
* {{Citation | last = Hilbert | first = M | year = 2016 | title = Big Data for Development: A Review of Promises and Challenges | journal = Development Policy Review | volume = 34 | issue = 1 | pages = 135–74 | doi = 10.1111/dpr.12142 | doi-access = free }}; [https://www.martinhilbert.net/big-data-for-development/  free access], {{Webarchive|url=https://web.archive.org/web/20210421222027/https://www.martinhilbert.net/big-data-for-development/ |date=21 April 2021 }}
* {{cite journal | last1 = Snijders | first1 = C. | last2 = Matzat | first2 = U. | last3 = Reips | first3 = U.-D. | year = 2012 | title = 'Big Data': Big gaps of knowledge in the field of Internet | url = http://www.ijis.net/ijis7_1/ijis7_1_editorial.html | journal = International Journal of Internet Science | volume = 7 | pages = 1–5 | access-date = 13 April 2013 | archive-date = 23 November 2019 | archive-url = https://web.archive.org/web/20191123051001/http://www.ijis.net/ijis7_1/ijis7_1_editorial.html | url-status = live }}
* {{cite journal | last1 = Yanase | first1 = J | last2 = Triantaphyllou | first2 = E| title = A Systematic Survey of Computer-Aided Diagnosis in Medicine: Past and Present Developments | journal = Expert Systems with Applications | volume = 138 | page = 112821 | date = 2019 | doi = 10.1016/j.eswa.2019.112821 | s2cid = 199019309}}

== Further reading ==
{{Library resources box}}

* {{cite magazine|editor1=Peter Kinnaird |editor2=Inbal Talgam-Cohen|magazine=[[XRDS (magazine)|XRDS: Crossroads, The ACM Magazine for Students]]|title=Big Data|volume=19 |issue= 1 |date= 2012 |publisher=[[Association for Computing Machinery]]|issn=1528-4980 |oclc=779657714 |url=http://dl.acm.org/citation.cfm?id=2331042}}
* {{cite book|title=Mining of massive datasets|author1=Jure Leskovec|author2-link=Anand Rajaraman|author2=Anand Rajaraman|author3-link=Jeffrey D. Ullman|author3=Jeffrey D. Ullman |year= 2014 |publisher=Cambridge University Press|url=http://mmds.org/|isbn=978-1-10707723-2 |oclc=888463433|author1-link=Jure Leskovec}}
* {{cite book|author1=Viktor Mayer-Schönberger|author2-link=Kenneth Cukier|author2=Kenneth Cukier|title=Big Data: A Revolution that Will Transform how We Live, Work, and Think |date= 2013 |publisher= Houghton Mifflin Harcourt|isbn=978-1-29990302-9 |oclc=828620988|author1-link=Viktor Mayer-Schönberger}}
* {{cite news |url=https://www.forbes.com/sites/gilpress/2013/05/09/a-very-short-history-of-big-data |title=A Very Short History of Big Data |first=Gil |last=Press |work=forbes.com |date=9 May 2013 |access-date=17 September 2016 |location=Jersey City, NJ}}
* {{cite book |title=Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are |year=2017 |first=Seth |last=Stephens-Davidowitz |publisher=Dey Street Books |isbn=978-0-06239085-1}}
* {{cite magazine |url=https://hbr.org/2012/10/big-data-the-management-revolution |title=Big Data: The Management Revolution|magazine=Harvard Business Review |date=October 2012|work=}}
* {{cite book |title=Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy |first=Cathy |last= O'Neil |year=2017 |publisher=Broadway Books |isbn= 978-0-55341883-5}}

== External links ==
*{{Commons-inline}}
* {{Wiktionary-inline|big data}}

{{Data}}
{{Authority control}}

[[Category:Big data| ]]
[[Category:Data management]]
[[Category:Business intelligence terms]]
[[Category:Social information processing]]
[[Category:Transaction processing]]
[[Category:Technology forecasting]]
[[Category:Data analysis]]
[[Category:Databases]]