Generate code for sklearn's GradientBoostingClassifier

machine learning

sklearn

GradientBoostingClassifier

code generation

Python

Generate code for sklearn's GradientBoostingClassifier

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Machine Learning Python

Introduction

GradientBoostingClassifier from scikit-learn builds an ensemble of decision trees sequentially, where each tree corrects the errors of the previous ones. It is effective for tabular classification tasks and offers built-in feature importance. Key hyperparameters are n_estimators (number of trees), learning_rate (contribution of each tree), and max_depth (tree complexity). This article provides complete, runnable code for training, evaluating, and tuning a GradientBoostingClassifier.

Basic Training and Prediction

python

1from sklearn.ensemble import GradientBoostingClassifier
2from sklearn.model_selection import train_test_split
3from sklearn.datasets import load_breast_cancer
4from sklearn.metrics import accuracy_score, classification_report
5
6# Load dataset
7data = load_breast_cancer()
8X_train, X_test, y_train, y_test = train_test_split(
9    data.data, data.target, test_size=0.2, random_state=42
10)
11
12# Train the model
13gbc = GradientBoostingClassifier(
14    n_estimators=100,
15    learning_rate=0.1,
16    max_depth=3,
17    random_state=42,
18)
19gbc.fit(X_train, y_train)
20
21# Predict
22y_pred = gbc.predict(X_test)
23print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
24print(classification_report(y_test, y_pred, target_names=data.target_names))

The default parameters work well for many datasets. n_estimators=100 builds 100 sequential trees, each correcting residual errors from the previous ensemble.

Key Hyperparameters

python

1gbc = GradientBoostingClassifier(
2    # Core parameters
3    n_estimators=200,       # Number of boosting stages (trees)
4    learning_rate=0.05,     # Shrinks contribution of each tree
5    max_depth=4,            # Maximum depth of each tree
6
7    # Regularization
8    min_samples_split=10,   # Minimum samples to split a node
9    min_samples_leaf=5,     # Minimum samples in a leaf node
10    subsample=0.8,          # Fraction of samples per tree (stochastic boosting)
11    max_features='sqrt',    # Features considered per split
12
13    # Loss function
14    loss='log_loss',        # 'log_loss' for classification (default)
15
16    random_state=42,
17)
18gbc.fit(X_train, y_train)

Lower learning_rate with higher n_estimators gives better generalization but slower training. subsample < 1.0 introduces randomness (stochastic gradient boosting), reducing overfitting.

Hyperparameter Tuning with GridSearchCV

python

1from sklearn.model_selection import GridSearchCV
2
3param_grid = {
4    'n_estimators': [100, 200, 300],
5    'learning_rate': [0.01, 0.05, 0.1],
6    'max_depth': [3, 4, 5],
7    'subsample': [0.8, 1.0],
8}
9
10grid_search = GridSearchCV(
11    GradientBoostingClassifier(random_state=42),
12    param_grid,
13    cv=5,
14    scoring='accuracy',
15    n_jobs=-1,
16    verbose=1,
17)
18grid_search.fit(X_train, y_train)
19
20print(f"Best parameters: {grid_search.best_params_}")
21print(f"Best CV accuracy: {grid_search.best_score_:.4f}")
22
23# Evaluate on test set
24best_model = grid_search.best_estimator_
25print(f"Test accuracy: {best_model.score(X_test, y_test):.4f}")

GridSearchCV tries all parameter combinations with cross-validation. For large grids, use RandomizedSearchCV to sample a subset.

Feature Importance

python

1import numpy as np
2import matplotlib.pyplot as plt
3
4# Train the model
5gbc = GradientBoostingClassifier(n_estimators=100, random_state=42)
6gbc.fit(X_train, y_train)
7
8# Get feature importances
9importances = gbc.feature_importances_
10indices = np.argsort(importances)[::-1]
11
12# Plot top 10 features
13top_n = 10
14plt.figure(figsize=(10, 6))
15plt.bar(range(top_n), importances[indices[:top_n]])
16plt.xticks(range(top_n), data.feature_names[indices[:top_n]], rotation=45, ha='right')
17plt.title('Top 10 Feature Importances')
18plt.tight_layout()
19plt.show()

feature_importances_ ranks features by their contribution to reducing the loss function across all trees.

Training with Early Stopping

python

1import numpy as np
2
3gbc = GradientBoostingClassifier(
4    n_estimators=500,
5    learning_rate=0.05,
6    max_depth=3,
7    validation_fraction=0.1,  # 10% of training data for validation
8    n_iter_no_change=10,       # Stop if no improvement for 10 rounds
9    tol=1e-4,
10    random_state=42,
11)
12gbc.fit(X_train, y_train)
13
14print(f"Stopped at {gbc.n_estimators_} estimators (of 500 max)")
15print(f"Test accuracy: {gbc.score(X_test, y_test):.4f}")

Early stopping monitors validation loss and stops training when it plateaus, preventing overfitting and saving computation time.

Staged Prediction (Learning Curve)

python

1from sklearn.metrics import log_loss
2
3# Track loss at each boosting stage
4train_losses = []
5test_losses = []
6
7for y_pred_train in gbc.staged_predict_proba(X_train):
8    train_losses.append(log_loss(y_train, y_pred_train))
9
10for y_pred_test in gbc.staged_predict_proba(X_test):
11    test_losses.append(log_loss(y_test, y_pred_test))
12
13plt.figure(figsize=(10, 6))
14plt.plot(train_losses, label='Train Loss')
15plt.plot(test_losses, label='Test Loss')
16plt.xlabel('Boosting Stage')
17plt.ylabel('Log Loss')
18plt.legend()
19plt.title('GradientBoosting Learning Curve')
20plt.show()

staged_predict_proba yields predictions at each boosting stage, letting you visualize when the model starts overfitting.

Pipeline with Preprocessing

python

1from sklearn.pipeline import Pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.compose import ColumnTransformer
4
5# With mixed feature types
6preprocessor = ColumnTransformer(
7    transformers=[
8        ('num', StandardScaler(), numerical_columns),
9    ],
10    remainder='passthrough',
11)
12
13pipeline = Pipeline([
14    ('preprocess', preprocessor),
15    ('classifier', GradientBoostingClassifier(
16        n_estimators=200,
17        learning_rate=0.05,
18        max_depth=4,
19        random_state=42,
20    )),
21])
22
23pipeline.fit(X_train, y_train)
24print(f"Accuracy: {pipeline.score(X_test, y_test):.4f}")

Common Pitfalls

Training too many estimators without early stopping: GradientBoosting can overfit with too many trees. Use n_iter_no_change and validation_fraction to stop training automatically when validation loss stops improving.
Learning rate too high: A high learning_rate (e.g., 0.5+) makes each tree contribute too much, causing overfitting. Lower values (0.01-0.1) with more estimators give better generalization.
Slow training on large datasets: GradientBoosting trains trees sequentially and does not parallelize (n_jobs has no effect). For large datasets, consider HistGradientBoostingClassifier (sklearn) or XGBoost/LightGBM, which are much faster.
Not scaling features: GradientBoosting uses decision trees, which are scale-invariant. Scaling features is not required (unlike logistic regression or SVM). Adding unnecessary scaling wastes preprocessing time.
Ignoring class imbalance: GradientBoosting does not handle imbalanced classes by default. Use sample_weight in fit() or adjust class weights manually for imbalanced datasets.

Summary

GradientBoostingClassifier builds sequential trees that correct each other's errors
Key parameters: n_estimators, learning_rate, max_depth, subsample
Use lower learning_rate with higher n_estimators for better generalization
Enable early stopping with n_iter_no_change to prevent overfitting
Use staged_predict_proba to visualize the learning curve and detect overfitting
For large datasets, prefer HistGradientBoostingClassifier or XGBoost for faster training

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Generate code for sklearn's GradientBoostingClassifier

ML System Design practice on Codemia

Introduction

Basic Training and Prediction

Key Hyperparameters

Hyperparameter Tuning with GridSearchCV

Feature Importance

Training with Early Stopping

Staged Prediction (Learning Curve)

Pipeline with Preprocessing

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

Track what you have practised

ML System Design practice on Codemia