View source for Ensemble learning

{{short description|Statistics and machine learning technique}}
{{Machine learning|Supervised learning}}

In [[statistics]] and [[machine learning]], '''ensemble methods''' use multiple learning algorithms to obtain better [[predictive inference|predictive performance]] than could be obtained from any of the constituent learning algorithms alone.<ref>{{cite journal
 |last1=Opitz |first1=D. |last2=Maclin |first2=R.
 |title=Popular ensemble methods: An empirical study
 |journal=[[Journal of Artificial Intelligence Research]]
 |volume=11 |pages=169&ndash;198 |year=1999
 |doi=10.1613/jair.614
|doi-access=free|arxiv=1106.0257}}</ref><ref>{{cite journal
 |last1=Polikar |first1=R.
 |title=Ensemble based systems in decision making
 |journal=IEEE Circuits and Systems Magazine
 |volume=6 |issue=3 |pages=21&ndash;45 |year=2006
 |doi=10.1109/MCAS.2006.1688199
|s2cid=18032543
 }}</ref><ref name="Rokach2010">{{cite journal
 |last1=Rokach |first1=L.
 |title=Ensemble-based classifiers
 |journal=Artificial Intelligence Review
 |volume=33
 |issue=1–2 |pages=1&ndash;39 |year=2010
 |doi=10.1007/s10462-009-9124-7
|s2cid=11149239
 |hdl=11323/1748
 |hdl-access=free
 }}</ref>
Unlike a [[statistical ensemble]] in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

== Overview ==

[[Supervised learning]] algorithms search through a [[hypothesis]] space to find a suitable hypothesis that will make good predictions with a particular problem.<ref>{{cite book |author=Blockeel H.|title=Encyclopedia of Machine Learning |chapter=Hypothesis Space |date=2011 |pages=511–513 |doi=10.1007/978-0-387-30164-8_373 |isbn=978-0-387-30768-8 |chapter-url=https://lirias.kuleuven.be/handle/123456789/298291 }}</ref> Even if this space contains hypotheses that are very well-suited for a particular problem, it may be very difficult to find a good one. Ensembles combine multiple hypotheses to form one which should be theoretically better.

''Ensemble learning'' trains two or more machine learning algorithms on a specific [[classification]] or [[Regression analysis|regression]] task. The algorithms within the ensemble model are generally referred as "base models", "base learners", or "weak learners" in literature. These base models can be constructed using a single modelling algorithm, or several different algorithms. The idea is to train a diverse set of weak models on the same modelling task, such that the outputs of each weak learner have poor predictive ability (i.e., high [[Bias (statistics)|bias]]), and among all weak learners, the outcome and error values exhibit high [[variance]]. Fundamentally, an ensemble learning model trains at least two high-bias (weak) and high-variance (diverse) models to be combined into a better-performing model. The set of weak models — which would not produce satisfactory predictive results individually — are combined or averaged to produce a single, high performing, accurate, and low-variance model to fit the task as required.

Ensemble learning typically refers to bagging ([[bootstrap aggregating]]), [[Boosting (machine learning)|boosting]] or stacking/blending techniques to induce high variance among the base models. Bagging creates diversity by generating random samples from the training observations and fitting the same model to each different sample — also known as ''homogeneous parallel ensembles''. Boosting follows an iterative process by sequentially training each base model on the up-weighted errors of the previous base model, producing an additive model to reduce the final model errors — also known as ''sequential ensemble learning''. Stacking or blending consists of different base models, each trained independently (i.e. diverse/high variance) to be combined into the ensemble model — producing a ''heterogeneous parallel ensemble''. Common applications of ensemble learning include [[random forest]]s (an extension of bagging), Boosted Tree models, and [[Gradient boosting|Gradient Boosted]] Tree Models. Models in applications of stacking are generally more task-specific — such as combining clustering techniques with other parametric and/or non-parametric techniques.<ref>{{cite book |author= Ibomoiye Domor Mienye, Yanxia Sun |title= A Survey of Ensemble Learning: Concepts, Algorithms, Applications and Prospects|date=2022 |url= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=survey+of+ensemble+learning&oq=survey+of+ensemble#d=gs_qabs&t=1711172531516&u=%23p%3DONpJl2uwB3UJ }}</ref>

The broader term of ''multiple classifier systems'' also covers hybridization of hypotheses that are not induced by the same base learner.{{citation needed|date=December 2017}}

Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model. In one sense, ensemble learning may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. On the other hand, the alternative is to do a lot more learning with one non-ensemble model. An ensemble may be more efficient at improving overall accuracy for the same increase in compute, storage, or communication resources by using that increase on two or more methods, than would have been improved by increasing resource use for a single method. Fast algorithms such as [[decision tree learning|decision trees]] are commonly used in ensemble methods (e.g., random forests), although slower algorithms can benefit from ensemble techniques as well.

By analogy, ensemble techniques have been used also in [[unsupervised learning]] scenarios, for example in [[consensus clustering]] or in [[anomaly detection]].

== Ensemble theory ==
Empirically, ensembles tend to yield better results when there is a significant diversity among the models.<ref>[[Ludmila Kuncheva|Kuncheva, L.]] and Whitaker, C., [https://link.springer.com/content/pdf/10.1023/A:1022859003006.pdf Measures of diversity in classifier ensembles], ''Machine Learning'', 51, pp. 181-207, 2003</ref><ref>Sollich, P. and Krogh, A., [https://proceedings.neurips.cc/paper/1995/file/1019c8091693ef5c5f55970346633f92-Paper.pdf ''Learning with ensembles: How overfitting can be useful''], Advances in Neural Information Processing Systems, volume 8, pp. 190-196, 1996.</ref> Many ensemble methods, therefore, seek to promote diversity among the models they combine.<ref>Brown, G. and Wyatt, J. and Harris, R. and Yao, X., [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.421.349&rep=rep1&type=pdf Diversity creation methods: a survey and categorisation]., ''Information Fusion'', 6(1), pp.5-20, 2005.</ref><ref>{{cite journal | last1 = Adeva | first1 = J. J. García | last2 = Cerviño | first2 = Ulises | last3 = Calvo | first3 = R. | title = Accuracy and Diversity in Ensembles of Text Categorisers | url = https://www.clei.org/cleiej/index.php/cleiej/article/view/319/112 | format = PDF | journal = CLEI Journal | date = December 2005 | volume = 8 | issue = 2| pages = 1:1–1:12 | doi = 10.19153/cleiej.8.2.1 | doi-broken-date = 1 November 2024 | doi-access = free }}</ref> Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).<ref>Ho, T., Random Decision Forests, ''Proceedings of the Third International Conference on Document Analysis and Recognition'', pp. 278-282, 1995.</ref> Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to ''dumb-down'' the models in order to promote diversity.<ref>{{cite book | last1 = Gashler | first1 = M. | last2 = Giraud-Carrier | first2 = C. | last3 = Martinez | first3 = T. | title = 2008 Seventh International Conference on Machine Learning and Applications | chapter = Decision Tree Ensemble: Small Heterogeneous is Better Than Large Homogeneous | year = 2008| chapter-url = http://axon.cs.byu.edu/papers/gashler2008icmla.pdf | volume = 2008 | pages = 900–905 | doi = 10.1109/ICMLA.2008.154 | isbn = 978-0-7695-3495-4 | s2cid = 614810 | url = https://scholarsarchive.byu.edu/facpub/902 }}</ref> It is possible to increase diversity in the training stage of the model using correlation for regression tasks <ref>{{cite journal |last1=Liu |first1=Y. |last2=Yao |first2=X. |title=Ensemble learning via negative correlation |journal=Neural Networks |date=December 1999 |volume=12 |issue=10 |pages=1399–1404 |doi=10.1016/S0893-6080(99)00073-8 |pmid=12662623 |url=https://doi.org/10.1016/S0893-6080(99)00073-8 |language=en |issn=0893-6080}}</ref> or using information measures such as cross entropy for classification tasks.<ref>{{cite book |last1=Shoham |first1=Ron |last2=Permuter |first2=Haim |title=Cyber Security Cryptography and Machine Learning |chapter=Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble (Brief Announcement) |series=Lecture Notes in Computer Science |date=2019 |volume=11527 |pages=202–207 |doi=10.1007/978-3-030-20951-3_18|isbn=978-3-030-20950-6 |s2cid=189926552 }}</ref>

[[Image:Combining_multiple_classifiers.svg|thumb|center|400px|An ensemble of classifiers usually has smaller classification error than base models.]]

Theoretically, one can justify the diversity concept because the lower bound of the error rate of an ensemble system can be decomposed into accuracy, diversity, and the other term.<ref>Terufumi Morishita et al, [https://proceedings.mlr.press/v162/morishita22a.html ''Rethinking Fano’s Inequality in Ensemble Learning''], International Conference on Machine Learning, 2022</ref>

=== The geometric framework ===
Ensemble learning, including both regression and classification tasks, can be explained using a geometric framework.<ref>Wu, S., Li, J., & Ding, W. (2023) A geometric framework for multiclass ensemble classifiers, ''Machine Learning'', 112(12), pp. 4929-4958. {{doi|10.1007/S10994-023-06406-W}}</ref> Within this framework, the output of each individual classifier or regressor for the entire dataset can be viewed as a point in a multi-dimensional space. Additionally, the target result is also represented as a point in this space, referred to as the "ideal point."

The Euclidean distance is used as the metric to measure both the performance of a single classifier or regressor (the distance between its point and the ideal point) and the dissimilarity between two classifiers or regressors (the distance between their respective points). This perspective transforms ensemble learning into a deterministic problem.

For example, within this geometric framework, it can be proved that the averaging of the outputs (scores) of all base classifiers or regressors can lead to equal or better results than the average of all the individual models. It can also be proved that if the optimal weighting scheme is used, then a weighted averaging approach can outperform any of the individual classifiers or regressors that make up the ensemble or as good as the best performer at least.

==Ensemble size==
While the number of component classifiers of an ensemble has a great impact on the accuracy of prediction, there is a limited number of studies addressing this problem. ''A priori'' determining of ensemble size and the volume and velocity of big data streams make this even more crucial for online ensemble classifiers. Mostly statistical tests were used for determining the proper number of components. More recently, a theoretical framework suggested that there is an ideal number of component classifiers for an ensemble such that having more or less than this number of classifiers would deteriorate the accuracy. It is called "the law of diminishing returns in ensemble construction." Their theoretical framework shows that using the same number of independent component classifiers as class labels gives the highest accuracy.<ref>{{cite conference |url=http://dl.acm.org/citation.cfm?id=2983907 |title=A Theoretical Framework on the Ideal Number of Classifiers for Online Ensembles in Data Streams |last1=R. Bonab |first1=Hamed |last2=Can |first2=Fazli |date=2016 |publisher=ACM |pages=2053 |location=USA |conference=CIKM}}</ref><ref>{{cite arXiv|eprint=1709.02925 |last1=Bonab |first1=Hamed |last2=Can |first2=Fazli |title=Less is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers |date=2017 |class=cs.LG }}</ref>

== Common types of ensembles ==

=== Bayes optimal classifier ===
{{Main article|Bayes classifier}}

The Bayes optimal classifier is a classification technique. It is an ensemble of all the hypotheses in the hypothesis space. On average, no other ensemble can outperform it.<ref>[[Tom M. Mitchell]], ''Machine Learning'', 1997, pp. 175</ref> The [[Naive Bayes classifier]] is a version of this that assumes that the data is conditionally independent on the class and makes the computation more feasible. Each hypothesis is given a vote proportional to the likelihood that the training dataset would be sampled from a system if that hypothesis were true. To facilitate training data of finite size, the vote of each hypothesis is also multiplied by the prior probability of that hypothesis. The Bayes optimal classifier can be expressed with the following equation:

:<math>y=\underset{c_j \in C}{\mathrm{argmax}} \sum_{h_i \in H}{P(c_j|h_i)P(T|h_i)P(h_i)}</math>

where <math>y</math> is the predicted class, <math>C</math> is the set of all possible classes, <math>H</math> is the hypothesis space, <math>P</math> refers to a ''probability'', and <math>T</math> is the training data. As an ensemble, the Bayes optimal classifier represents a hypothesis that is not necessarily in <math>H</math>. The hypothesis represented by the Bayes optimal classifier, however, is the optimal hypothesis in ''ensemble space'' (the space of all possible ensembles consisting only of hypotheses in <math>H</math>).

This formula can be restated using [[Bayes' theorem]], which says that the posterior is proportional to the likelihood times the prior:

:<math>P(h_i|T) \propto P(T|h_i)P(h_i)</math>

hence,

:<math>y=\underset{c_j \in C}{\mathrm{argmax}} \sum_{h_i \in H}{P(c_j|h_i)P(h_i|T)}</math>

=== Bootstrap aggregating (bagging) ===

{{main|Bootstrap aggregating}}[[File:Bootstrap set generation.png|thumb|upright=0.75|Three datasets bootstrapped from an original set. Example A occurs twice in set 1 because these are chosen with replacement.]]Bootstrap aggregation (''bagging'') involves training an ensemble on ''bootstrapped'' data sets. A bootstrapped set is created by selecting from original training data set with replacement. Thus, a bootstrap set may contain a given example zero, one, or multiple times. Ensemble members can also have limits on the features (e.g., nodes of a decision tree), to encourage exploring of diverse features.<ref>Salman, R., Alzaatreh, A., Sulieman, H., & Faisal, S. (2021). A Bootstrap Framework for Aggregating within and between Feature Selection Methods. Entropy (Basel, Switzerland), 23(2), 200. {{doi|10.3390/e23020200}}</ref> The variance of local information in the bootstrap sets and feature considerations promote diversity in the ensemble, and can strengthen the ensemble.<ref>Breiman, L., Bagging Predictors, ''Machine Learning'', 24(2), pp.123-140, 1996. {{doi|10.1007/BF00058655}}</ref> To reduce overfitting, a member can be validated using the out-of-bag set (the examples that are not in its bootstrap set).<ref>Brodeur, Z. P., Herman, J. D., & Steinschneider, S. (2020). Bootstrap aggregation and cross-validation methods to reduce overfitting in reservoir control policy search. Water Resources Research, 56, e2020WR027184. {{doi|10.1029/2020WR027184}}</ref>

Inference is done by '''voting''' of predictions of ensemble members, called '''aggregation'''. It is illustrated below with an ensemble of four decision trees. The query example is classified by each tree. Because three of the four predict the ''positive'' class, the ensemble's overall classification is ''positive''. [[Random forest]]s like the one shown are a common application of bagging.

[[File:Ensemble Aggregation.png|frameless|center|An example of the aggregation process for an ensemble of decision trees. Individual classifications are aggregated, and an overall classification is derived.|615x615px]]

=== Boosting ===
{{main|Boosting (meta-algorithm)}}
Boosting involves training successive models by emphasizing training data mis-classified by previously learned models. Initially, all data (D1) has equal weight and is used to learn a base model M1. The examples mis-classified by M1 are assigned a weight greater than correctly classified examples. This boosted data (D2) is used to train a second base model M2, and so on. Inference is done by voting.

In some cases, boosting has yielded better accuracy than bagging, but tends to over-fit more. The most common implementation of boosting is [[Adaboost]], but some newer algorithms are reported to achieve better results.{{Citation needed|date=January 2012}}

=== Bayesian model averaging ===

Bayesian model averaging (BMA) makes predictions by averaging the predictions of models weighted by their posterior probabilities given the data.<ref>e.g., {{cite Q|Q98974344}}<!-- Bayesian Model Averaging: A Tutorial --></ref> BMA is known to generally give better answers than a single model, obtained, e.g., via [[stepwise regression]], especially where very different models have nearly identical performance in the training set but may otherwise perform quite differently.

The question with any use of [[Bayes' theorem]] is the prior, i.e., the probability (perhaps subjective) that each model is the best to use for a given purpose.  Conceptually, BMA can be used with any prior.  ''R'' packages ensembleBMA<ref>{{cite Q|Q98972500}}<!-- ensembleBMA --></ref> and BMA<ref>{{cite Q|Q91674106}}<!-- BMA: Bayesian Model Averaging -->.</ref> use the prior implied by the [[Bayesian information criterion]], (BIC), following Raftery (1995).<ref>{{cite Q|Q91670340}}<!-- Bayesian model selection in social research --></ref> ''R'' package BAS supports the use of the priors implied by [[Akaike information criterion]] (AIC) and other criteria over the alternative models as well as priors over the coefficients.<ref>{{cite Q|Q98974089}}<!-- BAS: Bayesian Variable Selection and Model Averaging using Bayesian Adaptive Sampling -->.</ref>

The difference between BIC and AIC is the strength of preference for parsimony.  BIC's penalty for model complexity is <math>\ln(n) k</math> , while AIC's is <math>2k</math>. Large-sample asymptotic theory establishes that if there is a best model, then with increasing sample sizes, BIC is strongly consistent, i.e., will almost certainly find it, while AIC may not, because AIC may continue to place excessive posterior probability on models that are more complicated than they need to be. On the other hand, AIC and AICc are asymptotically "efficient" (i.e., minimum mean square prediction error), while BIC is not .<ref>{{cite Q|Q62568358}}<!-- Model selection and model averaging -->, ch. 4.</ref>

Haussler et al. (1994) showed that when BMA is used for classification, its expected error is at most twice the expected error of the Bayes optimal classifier.<ref>{{cite journal |last1=Haussler |first1=David |last2=Kearns |first2=Michael |last3=Schapire |first3=Robert E. |year=1994 |title=Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension |journal=Machine Learning |volume=14 |pages=83–113 |doi=10.1007/bf00993163 |doi-access=free}}</ref> Burnham and Anderson (1998, 2002) contributed greatly to introducing a wider audience to the basic ideas of Bayesian model averaging and popularizing the methodology.<ref>{{cite Q|Q62670082}}<!-- Burnam and Anderson (1998) Model Selection and Multimodel Inference --> and {{cite Q|Q76889160}}<!-- Burnham and Anderson (2002) Model Selection and Multimodel Inference, 2nd ed. -->.</ref> The availability of software, including other free open-source packages for [[R (programming language)|R]] beyond those mentioned above, helped make the methods accessible to a wider audience.<ref>The Wikiversity article on [[v:Searching R Packages|Searching R Packages]] mentions several ways to find available packages for something like this.  For example, "sos::findFn('{Bayesian model averaging}')" from within R will search for help files in contributed packages that includes the search term and open two tabs in the default browser.  The first will list all the help files found sorted by package.  The second summarizes the packages found, sorted by the apparent strength of the match.</ref>

=== Bayesian model combination ===
Bayesian model combination (BMC) is an algorithmic correction to Bayesian model averaging (BMA). Instead of sampling each model in the ensemble individually, it samples from the space of possible ensembles (with model weights drawn randomly from a Dirichlet distribution having uniform parameters). This modification overcomes the tendency of BMA to converge toward giving all the weight to a single model. Although BMC is somewhat more computationally expensive than BMA, it tends to yield dramatically better results. BMC has been shown to be better on average (with statistical significance) than BMA and bagging.<ref>{{cite conference
|author=Monteith, Kristine|author2=Carroll, James |author3=Seppi, Kevin |author4= Martinez, Tony. 
|url=http://axon.cs.byu.edu/papers/Kristine.ijcnn2011.pdf
|title=Turning Bayesian Model Averaging into Bayesian Model Combination
|conference=Proceedings of the International Joint Conference on Neural Networks IJCNN'11
|year=2011
|pages=2657&ndash;2663
}}</ref>

Use of Bayes' law to compute model weights requires computing the probability of the data given each model. Typically, none of the models in the ensemble are exactly the distribution from which the training data were generated, so all of them correctly receive a value close to zero for this term. This would work well if the ensemble were big enough to sample the entire model-space, but this is rarely possible. Consequently, each pattern in the training data will cause the ensemble weight to shift toward the model in the ensemble that is closest to the distribution of the training data. It essentially reduces to an unnecessarily complex method for doing model selection.

The possible weightings for an ensemble can be visualized as lying on a simplex. At each vertex of the simplex, all of the weight is given to a single model in the ensemble. BMA converges toward the vertex that is closest to the distribution of the training data. By contrast, BMC converges toward the point where this distribution projects onto the simplex. In other words, instead of selecting the one model that is closest to the generating distribution, it seeks the combination of models that is closest to the generating distribution.

The results from BMA can often be approximated by using cross-validation to select the best model from a bucket of models. Likewise, the results from BMC may be approximated by using cross-validation to select the best ensemble combination from a random sampling of possible weightings.

=== Bucket of models ===
A "bucket of models" is an ensemble technique in which a model selection algorithm is used to choose the best model for each problem. When tested with only one problem, a bucket of models can produce no better results than the best model in the set, but when evaluated across many problems, it will typically produce much better results, on average, than any model in the set.

The most common approach used for model-selection is [[cross-validation (statistics)|cross-validation]] selection (sometimes called a "bake-off contest"). It is described with the following pseudo-code:

 For each model m in the bucket:
     Do c times: (where 'c' is some constant)
         Randomly divide the training dataset into two sets: A and B
         Train m with A
         Test m with B
 Select the model that obtains the highest average score

Cross-Validation Selection can be summed up as: "try them all with the training set, and pick the one that works best".<ref>Saso Dzeroski, Bernard Zenko, ''[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.108.6096 Is Combining Classifiers Better than Selecting the Best One]'', Machine Learning, 2004, pp. 255-273</ref>

Gating is a generalization of Cross-Validation Selection. It involves training another learning model to decide which of the models in the bucket is best-suited to solve the problem. Often, a [[perceptron]] is used for the gating model. It can be used to pick the "best" model, or it can be used to give a linear weight to the predictions from each model in the bucket.

When a bucket of models is used with a large set of problems, it may be desirable to avoid training some of the models that take a long time to train. Landmark learning is a meta-learning approach that seeks to solve this problem. It involves training only the fast (but imprecise) algorithms in the bucket, and then using the performance of these algorithms to help determine which slow (but accurate) algorithm is most likely to do best.<ref>{{Cite book | chapter-url=https://link.springer.com/content/pdf/10.1007/3-540-45372-5_32.pdf | doi=10.1007/3-540-45372-5_32| isbn=978-3-540-41066-9| chapter=Discovering Task Neighbourhoods through Landmark Learning Performances| title=Principles of Data Mining and Knowledge Discovery| series=Lecture Notes in Computer Science| year=2000| last1=Bensusan| first1=Hilan| last2=Giraud-Carrier| first2=Christophe| volume=1910| pages=325–330}}</ref>

===Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble===
The most common approach for training classifier is using [[Cross-entropy]] cost function. However, one would like to train an ensemble of models that have diversity so when we combine them it would provide best results.<ref>{{cite book |last1=Shoham |first1=Ron |last2=Permuter |first2=Haim |chapter=Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble (Brief Announcement) |title=Cyber Security Cryptography and Machine Learning |series=Lecture Notes in Computer Science |date=2019 |volume=11527 |pages=202–207 |doi=10.1007/978-3-030-20951-3_18 |isbn=978-3-030-20950-6 }}</ref><ref>{{cite arXiv |last1=Shoham |first1=Ron |last2=Permuter |first2=Haim |title=Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement |date=2020 |class=cs.LG |eprint=2007.08140 }}</ref>
Assuming we use a simple ensemble of averaging <math> K </math> classifiers. Then the Amended Cross-Entropy Cost is
:<math>
    e^k = H(p,q^k)-\frac{\lambda}{K}\sum_{j\neq k}H(q^j,q^k)
</math>
where <math> e^k </math> is the cost function of the <math> k^{th} </math> classifier, <math> q^k </math> is the probability of the <math> k^{ th} </math> classifier, <math> p </math> is the true probability that we need to estimate and <math> \lambda </math> is a parameter between 0 and 1 that define the diversity that we would like to establish. When <math> \lambda=0 </math> we want each classifier to do its best regardless of the ensemble and when <math> \lambda=1 </math> we would like the classifier to be as diverse as possible.

=== Stacking ===

Stacking (sometimes called ''stacked generalization'') involves training a model to combine the predictions of several other learning algorithms. First, all of the other algorithms are trained using the available data, then a combiner algorithm (final estimator) is trained to make a final prediction using all the predictions of the other algorithms (base estimators) as additional inputs or using cross-validated predictions from the base estimators which can prevent overfitting.<ref>{{cite web |url=https://scikit-learn.org/stable/modules/ensemble.html#stacking |title=1.11. Ensemble methods }}</ref> If an arbitrary combiner algorithm is used, then stacking can theoretically represent any of the ensemble techniques described in this article, although, in practice, a [[logistic regression]] model is often used as the combiner.

Stacking typically yields performance better than any single one of the trained models.<ref>{{cite journal | last1 = Wolpert | year = 1992 | title = Stacked Generalization. | journal = Neural Networks | volume = 5 | issue = 2| pages = 241–259 | doi = 10.1016/s0893-6080(05)80023-1 }}</ref> It has been successfully used on both supervised learning tasks (regression,<ref>{{cite journal | doi = 10.1007/BF00117832 | volume=24 | title=Stacked regressions | year=1996 | journal=Machine Learning | pages=49–64 | last1 = Breiman | first1 = Leo| doi-access=free }}</ref> classification and distance learning <ref>{{cite arXiv |first1=M. |last1=Ozay |first2=F. T. |last2=Yarman Vural |title=A New Fuzzy Stacked Generalization Technique and Analysis of its Performance |year=2013 |class=cs.LG |eprint=1204.0171}}</ref>) and unsupervised learning (density estimation).<ref>{{cite journal | url=https://link.springer.com/content/pdf/10.1023/A:1007511322260.pdf | doi=10.1023/A:1007511322260 | year=1999 | last1=Smyth | first1=Padhraic | last2=Wolpert | first2=David | title=Linearly Combining Density Estimators via Stacking | journal=Machine Learning | volume=36 | issue=1 | pages=59–83 | s2cid=16006860 }}</ref> It has also been used to estimate bagging's error rate.<ref name="Rokach2010" /><ref>{{cite journal | url=https://link.springer.com/content/pdf/10.1023/A:1007519102914.pdf | doi=10.1023/A:1007519102914 | year=1999 | last1=Wolpert | first1=David H. | last2=MacReady | first2=William G. | title=An Efficient Method to Estimate Bagging's Generalization Error | journal=Machine Learning | volume=35 | issue=1 | pages=41–55 | s2cid=14357246 }}</ref> It has been reported to out-perform Bayesian model-averaging.<ref>Clarke, B., [https://www.jmlr.org/papers/volume4/clarke03a/clarke03a.pdf ''Bayes model averaging and stacking when model approximation error cannot be ignored''], Journal of Machine Learning Research, pp 683-712, 2003</ref> The two top-performers in the Netflix competition utilized blending, which may be considered a form of stacking.<ref>{{cite arXiv |last1=Sill |first1=J. |last2=Takacs |first2=G. |last3=Mackey |first3=L. |last4=Lin |first4=D. |title=Feature-Weighted Linear Stacking |year=2009 |class=cs.LG |eprint=0911.0460}}</ref>

=== Voting ===
Voting is another form of ensembling. See e.g. [[Weighted majority algorithm (machine learning)]].

== Implementations in statistics packages ==
* [[R (programming language)|R]]: at least three packages offer Bayesian model averaging tools,<ref>{{cite journal |first1=Shahram M. |last1=Amini |first2=Christopher F. |last2=Parmeter |title=Bayesian model averaging in R |journal=Journal of Economic and Social Measurement |volume=36 |issue=4 |pages=253–287 |year=2011 |doi= 10.3233/JEM-2011-0350|url=https://core.ac.uk/download/pdf/6494889.pdf }}</ref> including the {{mono|BMS}} (an acronym for Bayesian Model Selection) package,<ref>{{cite web |title=BMS: Bayesian Model Averaging Library |work=The Comprehensive R Archive Network |url=https://cran.r-project.org/web/packages/BMS/index.html |access-date=September 9, 2016 |date=2015-11-24 }}</ref> the {{mono|BAS}} (an acronym for Bayesian Adaptive Sampling) package,<ref>{{cite web |title=BAS: Bayesian Model Averaging using Bayesian Adaptive Sampling |work=The Comprehensive R Archive Network |url=https://cran.r-project.org/web/packages/BAS/index.html |access-date=September 9, 2016 }}</ref> and the {{mono|BMA}} package.<ref>{{cite web |title=BMA: Bayesian Model Averaging |work=The Comprehensive R Archive Network |url=https://cran.r-project.org/web/packages/BMA/index.html |access-date=September 9, 2016 }}</ref> 
* [[Python (programming language)|Python]]: [[scikit-learn]], a package for machine learning in Python offers packages for ensemble learning including packages for bagging, voting and averaging methods.
* [[MATLAB]]: classification ensembles are implemented in Statistics and Machine Learning Toolbox.<ref>{{cite web |title=Classification Ensembles |url=https://uk.mathworks.com/help/stats/classification-ensembles.html |work=MATLAB & Simulink |access-date=June 8, 2017 }}</ref>

== Ensemble learning applications ==

In recent years, due to growing computational power, which allows for training in large ensemble learning in a reasonable time frame, the number of ensemble learning applications has grown increasingly.<ref name="s1">{{cite journal |last1=Woźniak |first1=Michał |last2=Graña |first2=Manuel |last3=Corchado |first3=Emilio |title=A survey of multiple classifier systems as hybrid systems |journal=Information Fusion |date=March 2014 |volume=16 |pages=3–17 |doi=10.1016/j.inffus.2013.04.006|hdl=10366/134320 |s2cid=11632848 |hdl-access=free }}</ref> Some of the applications of ensemble classifiers include:

=== Remote sensing ===

{{main|Remote sensing}}

====Land cover mapping====

[[Image analysis#Land cover mapping|Land cover mapping]] is one of the major applications of [[Earth observation satellite]] sensors, using [[remote sensing]] and [[geospatial data]], to identify the materials and objects which are located on the surface of target areas. Generally, the classes of target materials include roads, buildings, rivers, lakes, and vegetation.<ref name="rodriguez">{{cite journal |last1=Rodriguez-Galiano |first1=V.F. |last2=Ghimire |first2=B. |last3=Rogan |first3=J. |last4=Chica-Olmo |first4=M. |last5=Rigol-Sanchez |first5=J.P. |title=An assessment of the effectiveness of a random forest classifier for land-cover classification |journal=ISPRS Journal of Photogrammetry and Remote Sensing |date=January 2012 |volume=67 |pages=93–104 |doi=10.1016/j.isprsjprs.2011.11.002|bibcode=2012JPRS...67...93R }}</ref> Some different ensemble learning approaches based on [[artificial neural networks]],<ref>{{cite journal |last1=Giacinto |first1=Giorgio |last2=Roli |first2=Fabio |title=Design of effective neural network ensembles for image classification purposes |journal=Image and Vision Computing |date=August 2001 |volume=19 |issue=9–10 |pages=699–707 |doi=10.1016/S0262-8856(01)00045-2|citeseerx=10.1.1.11.5820 }}</ref> [[kernel principal component analysis]] (KPCA),<ref>{{cite book |last1=Xia |first1=Junshi |last2=Yokoya |first2=Naoto |last3=Iwasaki |first3=Yakira |title=2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) |chapter=A novel ensemble classifier of hyperspectral and LiDAR data using morphological features |date=March 2017 |pages=6185–6189 |doi=10.1109/ICASSP.2017.7953345|isbn=978-1-5090-4117-6 |s2cid=40210273 }}</ref> [[decision trees]] with [[Boosting (machine learning)|boosting]],<ref>{{cite journal |last1=Mochizuki |first1=S. |last2=Murakami |first2=T. |title=Accuracy comparison of land cover mapping using the object-oriented image classification with machine learning algorithms |journal=33rd Asian Conference on Remote Sensing 2012, ACRS 2012 |date=November 2012 |volume=1 |pages=126–133}}</ref> [[random forest]]<ref name="rodriguez" /><ref>{{cite journal |last1=Liu |first1=Dan |last2=Toman |first2=Elizabeth |last3=Fuller |first3=Zane |last4=Chen |first4=Gang |last5=Londo |first5=Alexis |last6=Xuesong |first6=Zhang |last7=Kaiguang |first7=Zhao |title=Integration of historical map and aerial imagery to characterize long-term land-use change and landscape dynamics: An object-based analysis via Random Forests |journal=Ecological Indicators |date=2018 |volume=95 |issue=1 |pages=595–605 |doi=10.1016/j.ecolind.2018.08.004 |bibcode=2018EcInd..95..595L |s2cid=92025959 |url=https://pages.charlotte.edu/gang-chen/wp-content/uploads/sites/184/2018/08/Liu_2018_Intigration-historical-map-aerial-imagery-LCLUC.pdf}}</ref> and automatic design of multiple classifier systems,<ref>{{cite book |last1=Giacinto |first1=G. |last2=Roli |first2=F. |last3=Fumera |first3=G. |title=Proceedings 15th International Conference on Pattern Recognition. ICPR-2000 |chapter=Design of effective multiple classifier systems by clustering of classifiers |volume=2 |date=September 2000 |pages=160–163 |doi=10.1109/ICPR.2000.906039|isbn=978-0-7695-0750-7 |citeseerx=10.1.1.11.5328 |s2cid=2625643 }}</ref> are proposed to efficiently identify [[land cover]] objects.

====Change detection====

[[Change detection (GIS)|Change detection]] is an [[image analysis]] problem, consisting of the identification of places where the [[land cover]] has changed over time. [[Change detection (GIS)|Change detection]] is widely used in fields such as [[urban growth]], [[Forest dynamics|forest and vegetation dynamics]], [[land use]] and [[Disaster Monitoring Constellation|disaster monitoring]].<ref name="s2">{{cite journal |last1=Du |first1=Peijun |last2=Liu |first2=Sicong |last3=Xia |first3=Junshi |last4=Zhao |first4=Yindi |title=Information fusion techniques for change detection from multi-temporal remote sensing images |journal=Information Fusion |date=January 2013 |volume=14 |issue=1 |pages=19–27 |doi=10.1016/j.inffus.2012.05.003}}</ref>
The earliest applications of ensemble classifiers in change detection are designed with the majority [[plurality voting|voting]],<ref>Defined by Bruzzone et al. (2002)<!-- ref name="Bruzzone_et_al_2002" --> as "The data class that receives the largest number of votes is taken as the class of the input pattern", this is ''simple majority'', more accurately described as ''plurality'' voting.</ref> [[Bayesian model averaging]],<ref>{{cite journal |last1=Zhao |first1=Kaiguang |last2=Wulder |first2=Michael A |last3=Hu |first3=Tongx |last4=Bright |first4=Ryan |last5=Wu |first5=Qiusheng |last6=Qin |first6=Haiming |last7=Li |first7=Yang |title=Detecting change-point, trend, and seasonality in satellite time series data to track abrupt changes and nonlinear dynamics: A Bayesian ensemble algorithm |journal=Remote Sensing of Environment |date=2019 |volume=232 |page=111181 |doi=10.1016/j.rse.2019.04.034 |bibcode=2019RSEnv.23211181Z |s2cid=201310998 |url=https://go.osu.edu/beast2019|doi-access=free |hdl=11250/2651134 |hdl-access=free }}</ref> and the [[maximum posterior probability]].<ref name="Bruzzone_et_al_2002">{{cite journal |last1=Bruzzone |first1=Lorenzo |last2=Cossu |first2=Roberto |last3=Vernazza |first3=Gianni |title=Combining parametric and non-parametric algorithms for a partially unsupervised classification of multitemporal remote-sensing images |journal=Information Fusion |date=December 2002 |volume=3 |issue=4 |pages=289–297 |doi=10.1016/S1566-2535(02)00091-X|url=http://eprints.biblio.unitn.it/105/1/24.pdf }}</ref> Given the growth of satellite data over time, the past decade sees more use of time series methods for continuous change detection from image stacks.<ref>{{cite journal |last1=Theodomir |first1=Mugiraneza |last2=Nascetti |first2=Andrea |last3=Ban. |first3=Yifang |title=Continuous monitoring of urban land cover change trajectories with landsat time series and landtrendr-google earth engine cloud computing |journal=Remote Sensing |date=2020 |volume=12 |issue=18 |page=2883|doi=10.3390/rs12182883 |bibcode=2020RemS...12.2883M |doi-access=free }}</ref> One example is a Bayesian ensemble changepoint detection method called BEAST, with the software available as a package Rbeast in R, Python, and Matlab.<ref>{{cite web |last1=Li |first1=Yang |last2=Zhao |first2=Kaiguang |last3=Hu |first3=Tongxi |last4=Zhang |first4=Xuesong |title=BEAST: A Bayesian Ensemble Algorithm for Change-Point Detection and Time Series Decomposition |website=[[GitHub]] |url=https://github.com/zhaokg/Rbeast}}</ref>

=== Computer security ===

====Distributed denial of service====

[[Denial-of-service attack|Distributed denial of service]] is one of the most threatening [[cyber-attack]]s that may happen to an [[internet service provider]].<ref name="s1" /> By combining the output of single classifiers, ensemble classifiers reduce the total error of detecting and discriminating such attacks from legitimate [[flash crowd]]s.<ref>{{cite journal |last1=Raj Kumar |first1=P. Arun |last2=Selvakumar |first2=S. |title=Distributed denial of service attack detection using an ensemble of neural classifier |journal=Computer Communications |date=July 2011 |volume=34 |issue=11 |pages=1328–1341 |doi=10.1016/j.comcom.2011.01.012}}</ref>

====Malware Detection====

Classification of [[malware]] codes such as [[computer virus]]es, [[computer worm]]s, [[Trojan horses|trojans]], [[ransomware]] and [[spyware]]s with the usage of [[machine learning]] techniques, is inspired by the [[Document classification|document categorization problem]].<ref>{{cite journal |last1=Shabtai |first1=Asaf |last2=Moskovitch |first2=Robert |last3=Elovici |first3=Yuval |last4=Glezer |first4=Chanan |title=Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey |journal=Information Security Technical Report |date=February 2009 |volume=14 |issue=1 |pages=16–29 |doi=10.1016/j.istr.2009.03.003}}</ref> Ensemble learning systems have shown a proper efficacy in this area.<ref>{{cite book |first1=Boyun| last1=Zhang|first2=Jianping |last2=Yin |first3=Jingbo |last3=Hao |first4=Dingxing |last4=Zhang |first5=Shulin |last5=Wang | title=Autonomic and Trusted Computing| chapter=Malicious Codes Detection Based on Ensemble Learning| volume=4610|date=2007 |pages=468–477 |doi=10.1007/978-3-540-73547-2_48| series=Lecture Notes in Computer Science| isbn=978-3-540-73546-5}}</ref><ref>{{cite journal |last1=Menahem |first1=Eitan |last2=Shabtai |first2=Asaf |last3=Rokach |first3=Lior |last4=Elovici |first4=Yuval |title=Improving malware detection by applying multi-inducer ensemble |journal=Computational Statistics & Data Analysis |date=February 2009 |volume=53 |issue=4 |pages=1483–1494 |doi=10.1016/j.csda.2008.10.015|citeseerx=10.1.1.150.2722 }}</ref>

====Intrusion detection====

An [[intrusion detection system]] monitors [[computer network]] or [[computer system]]s to identify intruder codes like an [[anomaly detection]] process. Ensemble learning successfully aids such monitoring systems to reduce their total error.<ref>{{cite book |last1=Locasto |first1=Michael E. |last2=Wang |first2=Ke |last3=Keromytis |first3=Angeles D. |last4=Salvatore |first4=J. Stolfo |title=Recent Advances in Intrusion Detection |chapter=FLIPS: Hybrid Adaptive Intrusion Prevention |volume=3858 |date=2005 |pages=82–101 |doi=10.1007/11663812_5|series=Lecture Notes in Computer Science |isbn=978-3-540-31778-4 |citeseerx=10.1.1.60.3798 }}</ref><ref>{{cite journal |last1=Giacinto |first1=Giorgio |last2=Perdisci |first2=Roberto |last3=Del Rio |first3=Mauro |last4=Roli |first4=Fabio |title=Intrusion detection in computer networks by a modular ensemble of one-class classifiers |journal=Information Fusion |date=January 2008 |volume=9 |issue=1 |pages=69–82 |doi=10.1016/j.inffus.2006.10.002|citeseerx=10.1.1.69.9132 }}</ref>

=== Face recognition ===

{{main|Face recognition}}

[[Face recognition]], which recently has become one of the most popular research areas of [[pattern recognition]], copes with identification or verification of a person by their [[digital image]]s.<ref>{{cite book |last1=Mu |first1=Xiaoyan |last2=Lu |first2=Jiangfeng |last3=Watta |first3=Paul |last4=Hassoun |first4=Mohamad H. |title=2009 International Joint Conference on Neural Networks |chapter=Weighted voting-based ensemble classifiers with application to human face recognition and voice recognition |pages=2168–2171 |date=July 2009 |doi=10.1109/IJCNN.2009.5178708|isbn=978-1-4244-3548-7 |s2cid=18850747 }}</ref>

Hierarchical ensembles based on Gabor Fisher classifier and [[independent component analysis]] [[Data pre-processing|preprocessing]] techniques are some of the earliest ensembles employed in this field.<ref>{{cite book |last1=Yu |first1=Su |last2=Shan |first2=Shiguang |last3=Chen |first3=Xilin |last4=Gao |first4=Wen |title=7th International Conference on Automatic Face and Gesture Recognition (FGR06) |chapter=Hierarchical ensemble of Gabor Fisher classifier for face recognition |pages=91–96 |date=April 2006 |doi=10.1109/FGR.2006.64|isbn=978-0-7695-2503-7 |s2cid=1513315 }}</ref><ref>{{cite book |last1=Su |first1=Y. |last2=Shan |first2=S. |last3=Chen |first3=X. |last4=Gao |first4=W. |title=18th International Conference on Pattern Recognition (ICPR'06) |chapter=Patch-Based Gabor Fisher Classifier for Face Recognition |date=September 2006 |volume=2 |pages=528–531 |doi=10.1109/ICPR.2006.917|isbn=978-0-7695-2521-1 |s2cid=5381806 }}</ref><ref>{{cite book |last1=Liu |first1=Yang |last2=Lin |first2=Yongzheng |last3=Chen |first3=Yuehui |title=2008 Congress on Image and Signal Processing |chapter=Ensemble Classification Based on ICA for Face Recognition |date=July 2008 |pages=144–148 |doi=10.1109/CISP.2008.581 |isbn=978-0-7695-3119-9 |s2cid=16248842 }}</ref>

=== Emotion recognition ===

{{main|Emotion recognition}}

While [[speech recognition]] is mainly based on [[deep learning]] because most of the industry players in this field like [[Google]], [[Microsoft]] and [[IBM]] reveal that the core technology of their [[speech recognition]] is based on this approach, speech-based [[emotion recognition]] can also have a satisfactory performance with ensemble learning.<ref>{{cite book |last1=Rieger |first1=Steven A. |last2=Muraleedharan |first2=Rajani |last3=Ramachandran |first3=Ravi P. |title=The 9th International Symposium on Chinese Spoken Language Processing |chapter=Speech based emotion recognition using spectral feature extraction and an ensemble of KNN classifiers |date=2014 |pages=589–593 |doi=10.1109/ISCSLP.2014.6936711|isbn=978-1-4799-4219-0 |s2cid=31370450 }}</ref><ref>{{cite book |last1=Krajewski |first1=Jarek |last2=Batliner |first2=Anton |last3=Kessel |first3=Silke |title=2010 20th International Conference on Pattern Recognition |chapter=Comparing Multiple Classifiers for Speech-Based Detection of Self-Confidence - A Pilot Study |date=October 2010 |pages=3716–3719 |doi=10.1109/ICPR.2010.905|isbn=978-1-4244-7542-1 |s2cid=15431610 |url=https://opus.bibliothek.uni-augsburg.de/opus4/files/69603/69603.pdf }}</ref>

It is also being successfully used in [[Emotion recognition|facial emotion recognition]].<ref>{{cite journal |last1=Rani |first1=P. Ithaya |last2=Muneeswaran |first2=K. |title=Recognize the facial emotion in video sequences using eye and mouth temporal Gabor features |journal=Multimedia Tools and Applications |date=25 May 2016 |volume=76 |issue=7 |pages=10017–10040 |doi=10.1007/s11042-016-3592-y|s2cid=20143585 }}</ref><ref>{{cite journal |last1=Rani |first1=P. Ithaya |last2=Muneeswaran |first2=K. |title=Facial Emotion Recognition Based on Eye and Mouth Regions |journal=International Journal of Pattern Recognition and Artificial Intelligence |date=August 2016 |volume=30 |issue=7 |pages=1655020 |doi=10.1142/S021800141655020X}}</ref><ref>{{cite journal |last1=Rani |first1=P. Ithaya |last2=Muneeswaran |first2=K |title=Emotion recognition based on facial components |journal=Sādhanā |date=28 March 2018 |volume=43 |issue=3 |doi=10.1007/s12046-018-0801-6|doi-access=free }}</ref>

=== Fraud detection ===

{{main|Fraud detection}}

[[Fraud detection]] deals with the identification of [[bank fraud]], such as [[money laundering]], [[credit card fraud]] and [[telecommunication fraud]], which have vast domains of research and applications of [[machine learning]]. Because ensemble learning improves the robustness of the normal behavior modelling, it has been proposed as an efficient technique to detect such fraudulent cases and activities in banking and credit card systems.<ref>{{cite journal |last1=Louzada |first1=Francisco |last2=Ara |first2=Anderson |title=Bagging k-dependence probabilistic networks: An alternative powerful fraud detection tool |journal=Expert Systems with Applications |date=October 2012 |volume=39 |issue=14 |pages=11583–11592 |doi=10.1016/j.eswa.2012.04.024}}</ref><ref>{{cite journal |last1=Sundarkumar |first1=G. Ganesh |last2=Ravi |first2=Vadlamani |title=A novel hybrid undersampling method for mining unbalanced datasets in banking and insurance |journal=Engineering Applications of Artificial Intelligence |date=January 2015 |volume=37 |pages=368–377 |doi=10.1016/j.engappai.2014.09.019}}</ref>

=== Financial decision-making ===

The accuracy of prediction of business failure is a very crucial issue in financial decision-making. Therefore, different ensemble classifiers are proposed to predict [[financial crisis|financial crises]] and [[financial distress]].<ref name="ReferenceA">{{cite journal |last1=Kim |first1=Yoonseong |last2=Sohn |first2=So Young |title=Stock fraud detection using peer group analysis |journal=Expert Systems with Applications |date=August 2012 |volume=39 |issue=10 |pages=8986–8992 |doi=10.1016/j.eswa.2012.02.025}}</ref> Also, in the [[Market manipulation|trade-based manipulation]] problem, where traders attempt to manipulate [[stock price]]s by buying and selling activities, ensemble classifiers are required to analyze the changes in the [[stock market]] data and detect suspicious symptom of [[stock price]] [[Market manipulation|manipulation]].<ref name="ReferenceA"/>

=== Medicine ===

Ensemble classifiers have been successfully applied in [[neuroscience]], [[proteomics]] and [[medical diagnosis]] like in [[neurocognitive|neuro-cognitive disorder]] (i.e. [[Alzheimer]] or [[myotonic dystrophy]]) detection based on MRI datasets,<ref>{{cite journal |last1=Savio |first1=A. |last2=García-Sebastián |first2=M.T. |last3=Chyzyk |first3=D. |last4=Hernandez |first4=C. |last5=Graña |first5=M. |last6=Sistiaga |first6=A. |last7=López de Munain |first7=A. |last8=Villanúa |first8=J. |title=Neurocognitive disorder detection based on feature vectors extracted from VBM analysis of structural MRI |journal=Computers in Biology and Medicine |date=August 2011 |volume=41 |issue=8 |pages=600–610 |doi=10.1016/j.compbiomed.2011.05.010|pmid=21621760 }}</ref><ref>{{cite book |last1=Ayerdi |first1=B. |last2=Savio |first2=A. |last3=Graña |first3=M. |title=Natural and Artificial Computation in Engineering and Medical Applications |chapter=Meta-ensembles of Classifiers for Alzheimer's Disease Detection Using Independent ROI Features |volume=7931 |date=June 2013 |issue=Part 2 |pages=122–130 |doi=10.1007/978-3-642-38622-0_13|series=Lecture Notes in Computer Science |isbn=978-3-642-38621-3 }}</ref><ref>{{cite journal |last1=Gu |first1=Quan |last2=Ding |first2=Yong-Sheng |last3=Zhang |first3=Tong-Liang |title=An ensemble classifier based prediction of G-protein-coupled receptor classes in low homology |journal=Neurocomputing |date=April 2015 |volume=154 |pages=110–118 |doi=10.1016/j.neucom.2014.12.013}}</ref> cervical cytology classification.<ref>{{Cite journal|last1=Xue|first1=Dan|last2=Zhou|first2=Xiaomin|last3=Li|first3=Chen|last4=Yao|first4=Yudong|last5=Rahaman|first5=Md Mamunur|last6=Zhang|first6=Jinghua|last7=Chen|first7=Hao|last8=Zhang|first8=Jinpeng|last9=Qi|first9=Shouliang|last10=Sun|first10=Hongzan|date=2020|title=An Application of Transfer Learning and Ensemble Learning Techniques for Cervical Histopathology Image Classification|journal=IEEE Access|volume=8|pages=104603–104618|doi=10.1109/ACCESS.2020.2999816|s2cid=219689893|issn=2169-3536|doi-access=free|bibcode=2020IEEEA...8j4603X }}</ref><ref>{{Cite journal|last1=Manna|first1=Ankur|last2=Kundu|first2=Rohit|last3=Kaplun|first3=Dmitrii|last4=Sinitca|first4=Aleksandr|last5=Sarkar|first5=Ram|date=December 2021|title=A fuzzy rank-based ensemble of CNN models for classification of cervical cytology|journal=Scientific Reports|language=en|volume=11|issue=1|pages=14538|doi=10.1038/s41598-021-93783-8|issn=2045-2322|pmc=8282795|pmid=34267261|bibcode=2021NatSR..1114538M }}</ref> 

Besides, ensembles have been successfully applied in medical segmentation tasks, for example brain tumor<ref>{{Cite journal |last=Valenkova |first=Daria |last2=Lyanova |first2=Asya |last3=Sinitca |first3=Aleksandr |last4=Sarkar |first4=Ram |last5=Kaplun |first5=Dmitrii |date=April 2025 |title=A fuzzy rank-based ensemble of CNN models for MRI segmentation |url=https://linkinghub.elsevier.com/retrieve/pii/S1746809424014009 |journal=Biomedical Signal Processing and Control |language=en |volume=102 |pages=107342 |doi=10.1016/j.bspc.2024.107342}}</ref><ref>{{Cite journal |last=Rajput |first=Snehal |last2=Kapdi |first2=Rupal |last3=Roy |first3=Mohendra |last4=Raval |first4=Mehul S. |date=June 2024 |title=A triplanar ensemble model for brain tumor segmentation with volumetric multiparametric magnetic resonance images |url=https://linkinghub.elsevier.com/retrieve/pii/S2772442524000091 |journal=Healthcare Analytics |language=en |volume=5 |pages=100307 |doi=10.1016/j.health.2024.100307|doi-access=free }}</ref> and hyperintensities segmentation.<ref>{{Cite journal |last=Sundaresan |first=Vaanathi |last2=Zamboni |first2=Giovanna |last3=Rothwell |first3=Peter M. |last4=Jenkinson |first4=Mark |last5=Griffanti |first5=Ludovica |date=October 2021 |title=Triplanar ensemble U-Net model for white matter hyperintensities segmentation on MR images |url=https://linkinghub.elsevier.com/retrieve/pii/S1361841521002309 |journal=Medical Image Analysis |language=en |volume=73 |pages=102184 |doi=10.1016/j.media.2021.102184 |pmc=8505759 |pmid=34325148}}</ref>

== See also ==
* [[Ensemble averaging (machine learning)]]
* [[Bayesian structural time series]] (BSTS)
* [[Mixture of experts]]

== References ==
{{Reflist|30em}}

== Further reading==
* {{cite book |last=Zhou Zhihua |author-link=Zhou Zhihua |date=2012 |title=Ensemble Methods: Foundations and Algorithms |publisher= Chapman and Hall/CRC |isbn=978-1-439-83003-1 }}
* {{cite book |last1=Robert Schapire |last2=Yoav Freund |date=2012 |title=Boosting: Foundations and Algorithms |publisher=MIT |isbn=978-0-262-01718-3 |author1-link=Robert Schapire |author2-link=Yoav Freund }}

== External links ==
* {{scholarpedia|title=Ensemble learning|urlname=Ensemble_learning|curator=Robi Polikar}}
* The [[Waffles (machine learning)]] toolkit contains implementations of Bagging, Boosting, Bayesian Model Averaging, Bayesian Model Combination, Bucket-of-models, and other ensemble techniques

[[Category:Ensemble learning| ]]