{"id":98,"date":"2026-09-24T14:31:29","date_gmt":"2026-09-24T14:31:29","guid":{"rendered":"https:\/\/evoxia.online\/?p=98"},"modified":"2026-09-24T14:31:29","modified_gmt":"2026-09-24T14:31:29","slug":"decision-trees-random-forests-and-gradient-boosting-the-models-that-still-rule-tabular-data","status":"publish","type":"post","link":"https:\/\/evoxia.online\/?p=98","title":{"rendered":"Decision Trees, Random Forests and Gradient Boosting: The Models That Still Rule Tabular Data"},"content":{"rendered":"\n

What a Decision Tree Does<\/strong><\/p>\n\n\n\n

A decision tree is a sequence of yes or no questions about the data. Is the applicant’s income above 30,000? If yes, is their debt ratio under 0.4? If no, have they missed a payment in the last year? Each question splits the dataset into two smaller groups, and the process repeats inside each group until the groups are pure enough, or small enough, that splitting further stops helping. The final groups are the leaves, and each leaf holds a prediction: a class for classification, an average value for regression.<\/p>\n\n\n\n

The model learns which questions to ask. At every node, the algorithm tries many candidate splits across every feature and keeps the one that separates the data best. For classification, \u00abbest\u00bb usually means the split that most reduces Gini impurity or entropy. For regression, it is the split that most reduces the variance of the target inside each child. This is a greedy procedure. It picks the best split right now without checking whether a worse split today would lead to a better tree later, and that turns out to be a reasonable compromise, because finding the globally optimal tree is NP-hard.<\/p>\n\n\n\n

The two classic algorithms come from the 1980s. Breiman, Friedman, Olshen and Stone published CART (Classification and Regression Trees) in 1984, and Ross Quinlan published ID3 in 1986, later refined into C4.5. Almost every tree implementation in use today descends from one of those two lines.<\/p>\n\n\n\n

Why a Single Tree Is Not Enough<\/strong><\/p>\n\n\n\n

Trees have real strengths. They handle numerical and categorical features without scaling, they do not care whether a relationship is linear, and a small tree can be printed on a page and read by someone with no statistics background. A doctor can follow the path a tree took to classify a patient.<\/p>\n\n\n\n

The problem is variance. A tree grown until every leaf is pure will memorize the training set, noise included. Change a few training examples and the top split can change, which changes everything below it. The result is a model that looks excellent on the data it has seen and unstable on everything else. Pruning helps (cutting back branches that do not improve validation performance), and so does limiting depth, but a pruned tree is also a weaker tree. For most of the 1990s this tension was the main complaint about the method: interpretable, but not very accurate.<\/p>\n\n\n\n

The fix came from a different direction. Instead of trying to build one good tree, build many mediocre ones and combine them.<\/p>\n\n\n\n

Random Forests<\/strong><\/p>\n\n\n\n

Leo Breiman proposed random forests in 2001, building on his earlier work on bagging (bootstrap aggregating). The recipe has two sources of randomness. First, each tree is trained on a bootstrap sample, a random sample of the training set drawn with replacement, so every tree sees a slightly different dataset. Second, at each split, the tree is only allowed to consider a random subset of the features. For classification, a common default is the square root of the total number of features.<\/p>\n\n\n\n

That second step is the important one. Without it, if one feature is very predictive, every tree would split on it first and the trees would end up highly correlated. Averaging correlated models does not reduce variance much. Forcing each split to choose from a random subset decorrelates the trees, and averaging hundreds of decorrelated, high-variance trees gives a model with much lower variance and almost the same bias.<\/p>\n\n\n\n

Random forests also come with a free validation estimate. Each bootstrap sample leaves out roughly a third of the training data, and those out-of-bag examples can be used to measure error without a separate hold-out set. The method is hard to break, needs little tuning, and trains in parallel since each tree is independent. For many years it was the default first model people tried on a new tabular problem, and it is still a strong baseline.<\/p>\n\n\n\n

Boosting: Trees That Correct Each Other<\/strong><\/p>\n\n\n\n

Boosting takes the opposite approach. Instead of training independent trees and averaging them, it trains trees one after another, and each new tree focuses on what the previous ones got wrong.<\/p>\n\n\n\n

AdaBoost, introduced by Freund and Schapire in the mid 1990s, did this by reweighting the data. After each round, misclassified examples get a higher weight, so the next weak learner (often a tree with a single split, called a stump) pays more attention to them. The final prediction is a weighted vote.<\/p>\n\n\n\n

Jerome Friedman generalized the idea in 2001 with gradient boosting. The model is built as a sum of trees, and each new tree is fitted to the negative gradient of the loss function with respect to the current predictions. For squared error, that gradient is just the residual: the difference between the true value and what the ensemble currently predicts. So the second tree learns to predict the errors of the first, the third learns the errors of the first two, and so on. A learning rate shrinks each tree’s contribution, which slows learning down on purpose and generally improves generalization.<\/p>\n\n\n\n

Because it works with any differentiable loss, gradient boosting handles regression, classification, ranking and custom objectives with the same machinery. It also tends to be more accurate than random forests, at the cost of being easier to overfit and more sensitive to hyperparameters like depth, learning rate and number of trees.<\/p>\n\n\n\n

XGBoost, LightGBM and CatBoost<\/strong><\/p>\n\n\n\n

Gradient boosting became dominant in practice once fast, scalable implementations appeared. XGBoost, by Tianqi Chen and Carlos Guestrin, was presented at KDD in 2016. It added a regularization term to the objective that penalizes complex trees, used second-order gradient information to choose splits, handled missing values by learning a default direction at each split, and was engineered for speed with cache-aware and parallel computation. The paper reported that among 29 winning solutions published on Kaggle in 2015, 17 used XGBoost.<\/p>\n\n\n\n

Microsoft’s LightGBM (2017) made training faster on large datasets by growing trees leaf-wise instead of level-wise and by using histogram-based split finding, plus sampling tricks that skip examples with small gradients. Yandex’s CatBoost focused on categorical features, using an ordered form of target encoding that avoids leaking the label into the features, and symmetric trees that make prediction fast.<\/p>\n\n\n\n

The three libraries are now close in accuracy on most problems, and the choice between them often comes down to dataset size, how many categorical columns there are, and what the team already knows.<\/p>\n\n\n\n

Why Deep Learning Has Not Taken Over Tabular Data<\/strong><\/p>\n\n\n\n

Neural networks replaced almost everything in vision, speech and language. Tabular data is the exception. Grinsztajn, Oyallon and Varoquaux studied this directly in a NeurIPS 2022 paper, benchmarking tree-based models and several deep learning architectures on 45 datasets with a careful hyperparameter search for each. Tree-based models stayed ahead on medium-sized data (around 10,000 samples), even before considering that they train much faster.<\/p>\n\n\n\n

Their analysis pointed to inductive biases. Neural networks are biased toward smooth functions, while real tabular targets are often irregular, with sharp thresholds (a tax bracket, a credit cutoff). Tabular datasets also tend to contain many uninformative columns, and trees are good at simply ignoring them. And neural networks are rotation invariant in a way that mixes features together, which does not match data where each column has its own meaning.<\/p>\n\n\n\n

There is a newer twist. In January 2025, Hollmann and colleagues published TabPFN in Nature, a transformer pre-trained on millions of synthetic tabular datasets. It predicts on a new dataset in a single forward pass without training on it. On datasets with up to 10,000 samples, the authors report that it beat tuned ensembles of strong baselines, and in classification it did so in 2.8 seconds against baselines tuned for four hours. It does not replace boosting on large datasets, but it is the first serious sign that the gap is closing for small ones.<\/p>\n\n\n\n

Reading What the Model Learned<\/strong><\/p>\n\n\n\n

An ensemble of 500 trees is not interpretable the way a single tree is, but the structure still helps. Feature importance can be measured by how much each feature reduces impurity across all splits, or by permutation importance: shuffle one column and see how much accuracy drops. Impurity-based importance has a known bias toward features with many unique values, so permutation importance is often the safer choice.<\/p>\n\n\n\n

SHAP values go further. The TreeSHAP algorithm by Lundberg and colleagues computes exact Shapley values for tree ensembles in polynomial time, which gives a per-prediction breakdown of how much each feature pushed the output up or down. This is a large part of why boosted trees are common in regulated settings like credit scoring and insurance, where a lender may need to explain an individual decision.<\/p>\n\n\n\n

Where These Models Are Used<\/strong><\/p>\n\n\n\n

Fraud detection, credit risk, churn prediction, demand forecasting, click-through prediction for ads, insurance pricing, and a large share of clinical risk scores run on gradient boosted trees or random forests. In the physical sciences they appear in particle physics event classification and in materials property prediction. The common thread is data that lives in a table, with a few dozen to a few thousand columns, where each column means something specific.<\/p>\n\n\n\n

Reflection<\/strong><\/p>\n\n\n\n

Decision trees are one of the oldest ideas in machine learning, and the best-performing models on the most common kind of business data are still made of them. The individual tree never got much better. What changed was how trees are combined: averaging many decorrelated ones in random forests, or stacking small corrections one after another in boosting. When someone reaches for a neural network by default, it is worth training a gradient boosted model first. On a spreadsheet, it is usually the model to beat.<\/p>\n\n\n\n

By: Max Johnson B.<\/p>\n","protected":false},"excerpt":{"rendered":"

What a Decision Tree Does A decision tree is a sequence of yes or no questions about the data. Is the applicant’s income above 30,000? If yes, is their debt ratio under 0.4? If no, have they missed a payment in the last year? Each question splits the dataset into two smaller groups, and the […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-98","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/98","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=98"}],"version-history":[{"count":0,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/98\/revisions"}],"wp:attachment":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=98"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=98"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=98"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}