Introduction

GridSearchCV is scikit-learn's standard tool for trying multiple hyperparameter combinations under cross-validation and selecting the best model by score. With DecisionTreeClassifier, it is especially useful because tree quality can change a lot with settings such as max_depth, min_samples_split, min_samples_leaf, and criterion.

The Basic Pattern

A DecisionTreeClassifier works out of the box, but its defaults are rarely the best choice for a real dataset. GridSearchCV wraps the estimator, tries every combination in a parameter grid, and evaluates each combination across multiple folds.

A small runnable example with the Iris dataset:

1from sklearn.datasets import load_iris
2from sklearn.model_selection import GridSearchCV, train_test_split
3from sklearn.tree import DecisionTreeClassifier
4from sklearn.metrics import classification_report
5
6X, y = load_iris(return_X_y=True)
7
8X_train, X_test, y_train, y_test = train_test_split(
9    X,
10    y,
11    test_size=0.2,
12    random_state=42,
13    stratify=y,
14)
15
16param_grid = {
17    "criterion": ["gini", "entropy"],
18    "max_depth": [None, 2, 3, 4, 5],
19    "min_samples_split": [2, 5, 10],
20    "min_samples_leaf": [1, 2, 4],
21}
22
23search = GridSearchCV(
24    estimator=DecisionTreeClassifier(random_state=42),
25    param_grid=param_grid,
26    cv=5,
27    scoring="accuracy",
28    n_jobs=-1,
29    refit=True,
30)
31
32search.fit(X_train, y_train)
33
34y_pred = search.predict(X_test)
35
36print("Best params:", search.best_params_)
37print("Best CV score:", search.best_score_)
38print(classification_report(y_test, y_pred))

This code does four important things:

• splits train and test data before tuning
• defines a finite parameter grid
• runs cross-validation on the training set only
• evaluates the chosen model on held-out test data afterward

Why The Train/Test Split Still Matters

A common mistake is to run grid search on the whole dataset and report best_score_ as the final model quality. That score is a cross-validation score on the training process, not a true final estimate on unseen held-out data.

The safer workflow is:

1. create a train/test split
2. run GridSearchCV on the training split
3. evaluate best_estimator_ on the test split

That keeps model selection and final evaluation separate.

Choosing A Useful Parameter Grid

For decision trees, start with parameters that control complexity and split quality:

• 'criterion'
• 'max_depth'
• 'min_samples_split'
• 'min_samples_leaf'
• 'ccp_alpha'

For example, pruning via ccp_alpha can help reduce overfitting:

1param_grid = {
2    "max_depth": [None, 3, 5, 10],
3    "min_samples_leaf": [1, 2, 5],
4    "ccp_alpha": [0.0, 0.001, 0.01],
5}

Do not make the grid huge without reason. Grid search is exhaustive. The number of fits is roughly:

number of parameter combinations * number of CV folds

A grid with 100 combinations and cv=5 means 500 model fits.

Reading The Results

After fitting, the most important attributes are:

• 'best_params_'
• 'best_score_'
• 'best_estimator_'
• 'cv_results_'

Example:

print(search.best_params_)
print(search.best_score_)
print(search.best_estimator_)

If you want to inspect all runs, convert cv_results_ into a DataFrame:

1import pandas as pd
2
3results = pd.DataFrame(search.cv_results_)
4print(results[["params", "mean_test_score", "rank_test_score"]].sort_values("rank_test_score").head())

That is often more informative than looking only at the winner.

Scoring And Class Imbalance

The default classifier score is accuracy, but that is not always the right metric. If the classes are imbalanced, try a metric that reflects the real objective better, such as f1_macro, roc_auc_ovr, or a custom scorer.

1search = GridSearchCV(
2    estimator=DecisionTreeClassifier(random_state=42),
3    param_grid=param_grid,
4    cv=5,
5    scoring="f1_macro",
6    n_jobs=-1,
7)

A grid search is only as good as the metric it optimizes.

When To Use A Pipeline

A plain decision tree does not require feature scaling, so a pipeline is often unnecessary. But if preprocessing is involved, wrap everything in a Pipeline so cross-validation applies the transformations correctly inside each fold.

That avoids data leakage. Even when the final estimator is a tree, preprocessing steps still belong inside the pipeline if they learn from the data.

Common Pitfalls

• Running GridSearchCV on the full dataset and treating the internal CV score as the final test result.
• Making the parameter grid far larger than necessary and wasting compute.
• Forgetting random_state, which makes comparisons harder to reproduce.
• Tuning for accuracy when the real problem needs another metric.
• Inspecting only best_params_ and ignoring cv_results_, which often reveals close alternatives and instability.

Summary

• 'GridSearchCV tries parameter combinations under cross-validation and selects the best one by score.'
• With DecisionTreeClassifier, focus first on depth, split thresholds, leaf size, criterion, and pruning.
• Split the data before tuning so you can evaluate the chosen model on a true test set.
• Use a scoring metric that matches the real problem, not just the default.
• Review cv_results_ instead of relying only on the single best parameter set.