Gradient Boosting Sample Clauses

Gradient Boosting. The (extreme) gradient boosting algorithm is another decision tree ensemble technique. However, it works on the principle of boosting instead of the bagging in the random forest method. Just like in bagging, boosting works with multiple trees to then combine them to create a single predictive model. However, the way that the trees are built differs significantly. The trees are grown sequentially instead of independently. This means that each tree is grown using information from the previously grown trees, with the model precision being improved after each iteration. Furthermore, each tree is fit on the original full data set instead of involving bootstrap sampling (there is a hyperparameter for that though, cfr. infra). In this dissertation, the extreme gradient boosting version was performed. It works on the same principle as the general gradient boosting, but with a more regularized model formalization to control for overfitting1. As this method is also inherently binary, the one-vs-rest strategy was used here to make it multiclass. With this strategy, each class is fitted against all other classes for each classifier. Compared to the one-vs-one strategy, this saves computation time as only three classifiers will be needed (one for each class). Furthermore, it is also more interpretable as each class is only represented by one classifier. This is the default strategy for this method and it is also the default for almost all classifiers. Similar to random forest, boosting has hyperparameters that need to be tuned. Some examples are (Xxxxxxx & Xxxx, 2018):