Gradient Boosting

Gradient Boosting is a class of ensemble learning algorithms that iteratively trains weak learners (typically decision trees), with each new model focusing on correcting the residuals from the previous round.

Compared to deep learning, gradient boosting is generally more stable, more interpretable, and faster to train on small-to-medium-sized datasets.

Gradient boosting achieves its ensemble effect in essence by iteratively training weak learners, with each new tree focusing on fitting the residuals of the previous round.

Gradient Boosting achieved the lowest root mean square error (RMSE) of 6.67 MPa.

Gradient Boosting and XGBoost both achieved an R² of 0.903 on unseen data.

In the prediction experiments the author ran, TabPFN ranked first in all 9 runs: ROC-AUC was 0.721 with 300 historical bookings, 0.766 with 1,000, and 0.778 with 3,000; gradient boosting scored 0.682, 0.702 and 0.747 respectively; logistic regression scored 0.699, 0.741 and 0.749 respectively.

In predictive maintenance, anomaly detection usually employs unsupervised learning, such as autoencoders and isolation forests; fault classification relies on supervised learning, using random forests, gradient boosting, or time-series deep learning models.

AI和机器学习在微观模拟中的贡献包括：利用梯度提升、随机森林等算法更准确地预测个体对激励变化的反应，以及通过自然语言处理从非结构化文本中提取政策细节自动更新模型参数。

The initial model f0(x) in gradient boosting typically uses the mean of y as the initial prediction for all samples.

When using mean squared error (MSE) as the loss function, the negative gradient of the loss with respect to the predicted value is exactly equal to the residual y−prediction.

XGBoost and LightGBM are engineering implementations that extend and optimize the gradient boosting framework.

Gradient boosting and its derivative algorithms dominate data competitions such as Kaggle.

Gradient boosting starts from a rough initial model and continuously adds small models, with each new model correcting the errors left by all previous models.

The evolutionary thread of ensemble learning is: Bagging → Random Forest → Gradient Boosting → XGBoost.

Gradient descent computes the partial derivatives of the loss function with respect to model parameters and updates the parameters in the direction opposite to the gradient to approach the minimum of the loss function.

The author compared four models—linear regression, ridge regression, random forest, and gradient boosting—and all four exhibited exactly the same bias.