Introduction
GradientBoostingClassifier from scikit-learn builds an ensemble of decision trees sequentially, where each tree corrects the errors of the previous ones. It is effective for tabular classification tasks and offers built-in feature importance. Key hyperparameters are n_estimators (number of trees), learning_rate (contribution of each tree), and max_depth (tree complexity). This article provides complete, runnable code for training, evaluating, and tuning a GradientBoostingClassifier.
Basic Training and Prediction
1from sklearn.ensemble import GradientBoostingClassifier
2from sklearn.model_selection import train_test_split
3from sklearn.datasets import load_breast_cancer
4from sklearn.metrics import accuracy_score, classification_report
5
6# Load dataset
7data = load_breast_cancer()
8X_train, X_test, y_train, y_test = train_test_split(
9 data.data, data.target, test_size=0.2, random_state=42
10)
11
12# Train the model
13gbc = GradientBoostingClassifier(
14 n_estimators=100,
15 learning_rate=0.1,
16 max_depth=3,
17 random_state=42,
18)
19gbc.fit(X_train, y_train)
20
21# Predict
22y_pred = gbc.predict(X_test)
23print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
24print(classification_report(y_test, y_pred, target_names=data.target_names))
The default parameters work well for many datasets. n_estimators=100 builds 100 sequential trees, each correcting residual errors from the previous ensemble.
Key Hyperparameters
1gbc = GradientBoostingClassifier(
2 # Core parameters
3 n_estimators=200, # Number of boosting stages (trees)
4 learning_rate=0.05, # Shrinks contribution of each tree
5 max_depth=4, # Maximum depth of each tree
6
7 # Regularization
8 min_samples_split=10, # Minimum samples to split a node
9 min_samples_leaf=5, # Minimum samples in a leaf node
10 subsample=0.8, # Fraction of samples per tree (stochastic boosting)
11 max_features='sqrt', # Features considered per split
12
13 # Loss function
14 loss='log_loss', # 'log_loss' for classification (default)
15
16 random_state=42,
17)
18gbc.fit(X_train, y_train)
Lower learning_rate with higher n_estimators gives better generalization but slower training. subsample < 1.0 introduces randomness (stochastic gradient boosting), reducing overfitting.
Hyperparameter Tuning with GridSearchCV
1from sklearn.model_selection import GridSearchCV
2
3param_grid = {
4 'n_estimators': [100, 200, 300],
5 'learning_rate': [0.01, 0.05, 0.1],
6 'max_depth': [3, 4, 5],
7 'subsample': [0.8, 1.0],
8}
9
10grid_search = GridSearchCV(
11 GradientBoostingClassifier(random_state=42),
12 param_grid,
13 cv=5,
14 scoring='accuracy',
15 n_jobs=-1,
16 verbose=1,
17)
18grid_search.fit(X_train, y_train)
19
20print(f"Best parameters: {grid_search.best_params_}")
21print(f"Best CV accuracy: {grid_search.best_score_:.4f}")
22
23# Evaluate on test set
24best_model = grid_search.best_estimator_
25print(f"Test accuracy: {best_model.score(X_test, y_test):.4f}")
GridSearchCV tries all parameter combinations with cross-validation. For large grids, use RandomizedSearchCV to sample a subset.
Feature Importance
1import numpy as np
2import matplotlib.pyplot as plt
3
4# Train the model
5gbc = GradientBoostingClassifier(n_estimators=100, random_state=42)
6gbc.fit(X_train, y_train)
7
8# Get feature importances
9importances = gbc.feature_importances_
10indices = np.argsort(importances)[::-1]
11
12# Plot top 10 features
13top_n = 10
14plt.figure(figsize=(10, 6))
15plt.bar(range(top_n), importances[indices[:top_n]])
16plt.xticks(range(top_n), data.feature_names[indices[:top_n]], rotation=45, ha='right')
17plt.title('Top 10 Feature Importances')
18plt.tight_layout()
19plt.show()
feature_importances_ ranks features by their contribution to reducing the loss function across all trees.
Training with Early Stopping
1import numpy as np
2
3gbc = GradientBoostingClassifier(
4 n_estimators=500,
5 learning_rate=0.05,
6 max_depth=3,
7 validation_fraction=0.1, # 10% of training data for validation
8 n_iter_no_change=10, # Stop if no improvement for 10 rounds
9 tol=1e-4,
10 random_state=42,
11)
12gbc.fit(X_train, y_train)
13
14print(f"Stopped at {gbc.n_estimators_} estimators (of 500 max)")
15print(f"Test accuracy: {gbc.score(X_test, y_test):.4f}")
Early stopping monitors validation loss and stops training when it plateaus, preventing overfitting and saving computation time.
Staged Prediction (Learning Curve)
1from sklearn.metrics import log_loss
2
3# Track loss at each boosting stage
4train_losses = []
5test_losses = []
6
7for y_pred_train in gbc.staged_predict_proba(X_train):
8 train_losses.append(log_loss(y_train, y_pred_train))
9
10for y_pred_test in gbc.staged_predict_proba(X_test):
11 test_losses.append(log_loss(y_test, y_pred_test))
12
13plt.figure(figsize=(10, 6))
14plt.plot(train_losses, label='Train Loss')
15plt.plot(test_losses, label='Test Loss')
16plt.xlabel('Boosting Stage')
17plt.ylabel('Log Loss')
18plt.legend()
19plt.title('GradientBoosting Learning Curve')
20plt.show()
staged_predict_proba yields predictions at each boosting stage, letting you visualize when the model starts overfitting.
Pipeline with Preprocessing
1from sklearn.pipeline import Pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.compose import ColumnTransformer
4
5# With mixed feature types
6preprocessor = ColumnTransformer(
7 transformers=[
8 ('num', StandardScaler(), numerical_columns),
9 ],
10 remainder='passthrough',
11)
12
13pipeline = Pipeline([
14 ('preprocess', preprocessor),
15 ('classifier', GradientBoostingClassifier(
16 n_estimators=200,
17 learning_rate=0.05,
18 max_depth=4,
19 random_state=42,
20 )),
21])
22
23pipeline.fit(X_train, y_train)
24print(f"Accuracy: {pipeline.score(X_test, y_test):.4f}")
Common Pitfalls
Training too many estimators without early stopping: GradientBoosting can overfit with too many trees. Use n_iter_no_change and validation_fraction to stop training automatically when validation loss stops improving.
Learning rate too high: A high learning_rate (e.g., 0.5+) makes each tree contribute too much, causing overfitting. Lower values (0.01-0.1) with more estimators give better generalization.
Slow training on large datasets: GradientBoosting trains trees sequentially and does not parallelize (n_jobs has no effect). For large datasets, consider HistGradientBoostingClassifier (sklearn) or XGBoost/LightGBM, which are much faster.
Not scaling features: GradientBoosting uses decision trees, which are scale-invariant. Scaling features is not required (unlike logistic regression or SVM). Adding unnecessary scaling wastes preprocessing time.
Ignoring class imbalance: GradientBoosting does not handle imbalanced classes by default. Use sample_weight in fit() or adjust class weights manually for imbalanced datasets.
Summary
GradientBoostingClassifier builds sequential trees that correct each other's errors
Key parameters: n_estimators, learning_rate, max_depth, subsample
Use lower learning_rate with higher n_estimators for better generalization
Enable early stopping with n_iter_no_change to prevent overfitting
Use staged_predict_proba to visualize the learning curve and detect overfitting
For large datasets, prefer HistGradientBoostingClassifier or XGBoost for faster training
