Introduction

In scikit-learn's GradientBoostingClassifier, the default feature importance is based on how much each feature reduces the split criterion across the trees in the ensemble. In other words, features that produce larger weighted impurity reductions across many splits receive higher importance scores.

The Core Idea: Total Reduction in Impurity

Each decision tree in gradient boosting splits nodes using a criterion such as Gini impurity or log-loss-related improvement, depending on the configuration. Whenever a feature is used to split a node, that split reduces impurity by some amount.

The importance calculation accumulates those reductions for each feature across all trees, weighted by how many samples reach the split.

A simplified mental model is:

feature importance(feature_j)
  = sum of impurity decrease from all splits using feature_j

After summing across the ensemble, scikit-learn normalizes the values so they add up to 1.0.

Example in scikit-learn

Here is a small runnable example:

1from sklearn.datasets import make_classification
2from sklearn.ensemble import GradientBoostingClassifier
3
4X, y = make_classification(
5    n_samples=1000,
6    n_features=5,
7    n_informative=3,
8    n_redundant=0,
9    random_state=42,
10)
11
12model = GradientBoostingClassifier(random_state=42)
13model.fit(X, y)
14
15for i, score in enumerate(model.feature_importances_):
16    print(f"feature_{i}: {score:.4f}")

model.feature_importances_ is the impurity-based importance vector. The higher the value, the more that feature contributed to split quality across the boosted trees.

Why the Scores Are Not Just Split Counts

It is not enough for a feature to appear often. A feature used in many weak splits may matter less than a feature used in a few very strong splits near the top of the trees.

That is why importance reflects the quality of the splits, not only their count. Splits that affect many samples and create large impurity reductions usually matter more.

This is also why importance depends on the full fitted model rather than on any simple property of the raw dataset.

Look at Permutation Importance Too

Impurity-based importance is fast and convenient, but it has limitations. A useful comparison is permutation importance, which measures how much model performance drops when a feature is shuffled.

1from sklearn.inspection import permutation_importance
2
3result = permutation_importance(
4    model,
5    X,
6    y,
7    n_repeats=10,
8    random_state=42,
9)
10
11for i, score in enumerate(result.importances_mean):
12    print(f"feature_{i}: {score:.4f}")

Permutation importance is slower, but it often gives a more intuitive view of predictive importance because it asks: "How much does the model rely on this feature for its actual predictions?"

Important Limitations

Impurity-based importance can be biased toward:

• Features with many possible split points
• Continuous features
• Correlated features that share predictive signal

For example, if two features contain nearly the same information, the model may split importance between them in a way that makes each individual score look smaller than expected.

That does not mean the features are unimportant. It means the model has multiple ways to capture similar signal.

Common Pitfalls

The biggest mistake is interpreting feature importance as causation. These scores describe how the fitted model used the features, not whether the features cause the target.

Another issue is assuming the numbers are stable across different random seeds, data splits, or model hyperparameters. Because gradient boosting is a learned model, importance scores can shift when the training setup changes.

Developers also often compare impurity-based scores across different models as though they were directly interchangeable. They are mainly meaningful within the context of a specific fitted model.

Finally, do not rely on a single importance method when interpretation matters. It is often worth comparing impurity-based importance, permutation importance, and domain knowledge together.

Summary

• 'GradientBoostingClassifier computes default feature importance from total weighted impurity reduction across splits.'
• The scores are aggregated over all trees and normalized to sum to 1.0.
• Importance reflects split quality and sample weight, not just how often a feature appears.
• Correlated or high-cardinality features can distort impurity-based importance.
• Use permutation importance as a useful second view when interpretation matters.