Ensemble learning

Ensemble methods are used in statistics and for machine learning . They use a finite set of different learning algorithms to get better results than with a single learning algorithm. The calculation of the results of this set of algorithms takes longer than the evaluation of a single algorithm, but a result that is roughly the same can be achieved with a much lower calculation depth.

Decision trees are an important area of application for ensemble learning . A large decision tree tends to have high error rates and high variance, since there are many decision nodes from the root to the leaves, which are all traversed with uncertainty. Bagging would calculate many small decision trees and use the average of their results, which significantly reduces the variance (and thus the error rate).

Types of ensembles

Bayesian Optimal Classifier

The Bayes Optimal Classifier always delivers the best result of the following formula:

${\ displaystyle y = \ mathrm {argmax} _ {c_ {j} \ in C} \ sum _ {h_ {i} \ in H} {P (c_ {j} | h_ {i}) P (T | h_ {i}) P (h_ {i})}}$

It can even be shown that no other ensemble can on average outperform this method. Unfortunately, this method cannot be used in practice, since iterates over all hypotheses in the hypothesis space and this space is in the vast majority of cases much too large to be counted. ${\ displaystyle argmax}$

Bagging

Bagging combines several predictions from regression or classification models, whereby the individual predictions are weighted equally and the average of the predictions is determined at the end.

Boosting

Boosting merges many weak classifiers into one strong classifier. This general method has many different implementations, the best known being AdaBoost.

Individual evidence

^ Murphy: Machine Learning: A Probabilistic Perspective . Ed .: MIT Press. 2012.
^ Leo Breiman: Bagging predictors . In: Machine Learning . tape 24 , no. 2 , August 1, 1996, ISSN 0885-6125 , p. 123–140 , doi : 10.1007 / BF00058655 ( springer.com [accessed March 15, 2016]).

[1] Murphy: Machine Learning: A Probabilistic Perspective . Ed .: MIT Press. 2012.

[2] Leo Breiman: Bagging predictors . In: Machine Learning . tape 24 , no. 2 , August 1, 1996, ISSN 0885-6125 , p. 123–140 , doi : 10.1007 / BF00058655 ( springer.com [accessed March 15, 2016]).