A training algorithm where weak models are trained to iteratively improve the quality (reduce the loss) of a strong model. For example, a weak model could be a linear or small decision tree model. The strong model becomes the sum of all the previously trained weak models. In the simplest form of gradient boosting, at each iteration, a weak model is trained to predict the loss gradient of the strong model. Then, the strong model's output is updated by subtracting the predicted gradient, similar to gradient descent. where: - $F_\0\$ is the starting strong model. - $F_\i+1\$ is the next strong model. - $F_\i\$ is the current strong model. - $\\\$ is a value between 0.0 and 1.0 called shrinkage, which is analogous to the learning rate in gradient descent. - $f_\i\$ is the weak model trained to predict the loss gradient of $F_\i\$. Modern variations of gradient boosting also include the second derivative (Hessian) of the loss in their computation. Decision trees are commonly used as weak models in gradient boosting. See gradient boosted (decision) trees.
A side-by-side comparison of Random Forest and Gradient Boosting. Understand how aggregating many randomized trees differs from sequentially improving weak models to reduce prediction errors.
A side-by-side comparison of Gradient Boosting and XGBoost. Understand how the general boosting technique differs from the widely used implementation and when each term should be used.