Introduction

GridSearchCV is scikit-learn's standard way to try several hyperparameter combinations under a cross-validation scheme. The API is easy to use, but the important part is not the constructor call itself. The important part is making sure the search evaluates the exact pipeline, scoring rule, and validation strategy that match the real modeling problem.

Basic GridSearchCV Workflow

A grid search needs four ingredients:

• an estimator or pipeline
• a parameter grid
• a cross-validation strategy
• a scoring rule

A minimal example with an SVM classifier looks like this:

1from sklearn.datasets import load_iris
2from sklearn.model_selection import GridSearchCV, train_test_split
3from sklearn.svm import SVC
4
5X, y = load_iris(return_X_y=True)
6X_train, X_test, y_train, y_test = train_test_split(
7    X, y, test_size=0.2, random_state=42, stratify=y
8)
9
10param_grid = {
11    "C": [0.1, 1, 10],
12    "kernel": ["linear", "rbf"],
13    "gamma": ["scale", "auto"],
14}
15
16search = GridSearchCV(
17    estimator=SVC(),
18    param_grid=param_grid,
19    cv=5,
20    scoring="accuracy",
21    n_jobs=-1,
22)
23
24search.fit(X_train, y_train)
25print(search.best_params_)
26print(search.best_score_)
27print(search.score(X_test, y_test))

This trains every parameter combination using 5-fold cross-validation on the training set and keeps the best-performing combination by mean validation score.

Put Preprocessing Inside a Pipeline

If the model depends on scaling, imputation, encoding, or feature selection, that preprocessing must be inside the pipeline being cross-validated. Otherwise, information from the full dataset can leak into validation folds.

1from sklearn.datasets import load_wine
2from sklearn.model_selection import GridSearchCV, StratifiedKFold
3from sklearn.pipeline import Pipeline
4from sklearn.preprocessing import StandardScaler
5from sklearn.svm import SVC
6
7X, y = load_wine(return_X_y=True)
8
9pipeline = Pipeline([
10    ("scaler", StandardScaler()),
11    ("model", SVC()),
12])
13
14param_grid = {
15    "model__C": [0.1, 1, 10],
16    "model__kernel": ["linear", "rbf"],
17    "model__gamma": ["scale", "auto"],
18}
19
20cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
21search = GridSearchCV(pipeline, param_grid=param_grid, cv=cv, n_jobs=-1)
22search.fit(X, y)
23
24print(search.best_params_)

The double underscore in parameter names tells scikit-learn which pipeline step to target.

Choose the Validation Strategy Deliberately

Using cv=5 is fine for many problems, but it is not always enough to express the real split logic. For imbalanced classification, stratification is often important. For time series, shuffled folds are usually wrong. For grouped observations, the same group must not appear in both training and validation folds.

That means the cross-validation object is part of the model design, not just a technical detail.

Inspect More Than best_params_

The best row is useful, but the full result table often tells a better story. Several parameter combinations may perform nearly the same, and the simplest one may be the more maintainable choice.

1import pandas as pd
2
3results = pd.DataFrame(search.cv_results_)
4print(
5    results[["params", "mean_test_score", "std_test_score", "rank_test_score"]]
6    .sort_values("rank_test_score")
7    .head()
8)

Looking at score variability helps you avoid overreacting to tiny differences that are not operationally meaningful.

Keep a Final Test Set Separate

Cross-validation inside GridSearchCV helps choose hyperparameters, but it is still part of the tuning process. If you want an unbiased estimate of final generalization performance, keep a separate test set that the grid search never sees.

That is why the earlier example split off X_test and y_test before tuning.

Without that separation, it is easy to report a performance number that already benefited from repeated parameter selection on the same data.

Do Not Make the Grid Bigger Than the Question

A good parameter grid is focused. It reflects model knowledge and reasonable candidate ranges. A huge grid with dozens of barely plausible values consumes time without adding much insight.

A smaller targeted grid is often better because it is:

• faster to run
• easier to explain
• easier to repeat
• less likely to hide the important comparisons in a giant result table

If the space is truly large, random search or Bayesian search may be better tools, but that is a different strategy from classical grid search.

Common Pitfalls

A common mistake is doing scaling or imputation before the grid search instead of inside the pipeline. That leaks information across folds.

Another mistake is optimizing the wrong metric. Accuracy may be fine for balanced classes, but it can be misleading for imbalanced problems.

Teams also focus only on best_params_ and ignore the full cv_results_ table. That hides whether the chosen settings were clearly better or just marginally ahead.

Summary

• 'GridSearchCV evaluates parameter combinations with cross-validation.'
• Build the search around the real estimator pipeline, including preprocessing.
• Choose a validation strategy that matches the data structure, not just the default integer cv.
• Inspect the full result table, not just the best row.
• Keep a final test set separate if you need an unbiased post-tuning evaluation.