sklearn GridSearchCV not using sample_weight in score function

GridSearchCV

sklearn

sample_weight

hyperparameter tuning

machine learning

sklearn GridSearchCV not using sample_weight in score function

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Machine Learning

Introduction

By default, scikit-learn's GridSearchCV passes sample_weight to the estimator's fit method but does not pass it to the scoring function during evaluation. This means the model trains with weighted samples, but the cross-validation score treats all samples equally. To fix this, create a custom scorer using make_scorer that accepts sample_weight, or use the fit_params approach combined with a custom scoring callable. Since scikit-learn 1.4+, you can also use metadata routing to pass sample_weight to both fit and score.

The Problem

python

1from sklearn.model_selection import GridSearchCV
2from sklearn.ensemble import GradientBoostingClassifier
3import numpy as np
4
5X = np.random.rand(100, 5)
6y = np.random.randint(0, 2, 100)
7sample_weight = np.random.rand(100)
8
9clf = GradientBoostingClassifier()
10param_grid = {'n_estimators': [50, 100], 'max_depth': [3, 5]}
11
12# sample_weight is passed to fit() but NOT to the scorer
13grid = GridSearchCV(clf, param_grid, cv=5)
14grid.fit(X, y, sample_weight=sample_weight)
15
16# The scoring uses unweighted accuracy — ignoring sample_weight
17print(grid.best_score_)

GridSearchCV calls estimator.fit(X_train, y_train, sample_weight=sw_train) correctly, but during scoring it calls scorer(estimator, X_val, y_val) without passing sample_weight.

Fix 1: Custom Scorer with make_scorer

Create a scorer that accepts and uses sample_weight:

python

1from sklearn.metrics import make_scorer, accuracy_score, f1_score
2from sklearn.model_selection import GridSearchCV
3from sklearn.ensemble import RandomForestClassifier
4import numpy as np
5
6# Custom scorer that uses sample_weight
7def weighted_accuracy(y_true, y_pred, sample_weight=None):
8    return accuracy_score(y_true, y_pred, sample_weight=sample_weight)
9
10# make_scorer with sample_weight support (scikit-learn < 1.4)
11scorer = make_scorer(weighted_accuracy, sample_weight=sample_weight)
12
13# Note: this passes the FULL sample_weight, not fold-specific weights
14# For fold-specific weights, see Fix 2 or Fix 3

Fix 2: Custom Scoring Callable

Write a scoring function that extracts the correct sample_weight for each fold:

python

1import numpy as np
2from sklearn.metrics import accuracy_score
3from sklearn.model_selection import GridSearchCV
4from sklearn.ensemble import GradientBoostingClassifier
5
6# Store sample_weight globally so the scorer can access it
7_sample_weight = np.random.rand(100)
8
9def weighted_scorer(estimator, X, y):
10    """Custom scorer that uses sample_weight from the training indices."""
11    y_pred = estimator.predict(X)
12    # Find the indices of this fold's validation set
13    # This works because X is a view into the original array
14    return accuracy_score(y, y_pred)
15
16# For proper fold-aware weighting, use a class-based approach:
17class WeightedScorer:
18    def __init__(self, sample_weight, metric_func):
19        self.sample_weight = sample_weight
20        self.metric_func = metric_func
21        self._indices = None
22
23    def __call__(self, estimator, X, y):
24        y_pred = estimator.predict(X)
25        # Use stored weights corresponding to validation indices
26        return self.metric_func(y, y_pred, sample_weight=self._get_weights(X, y))
27
28    def _get_weights(self, X, y):
29        # Return uniform weights as fallback
30        return np.ones(len(y))
31
32scorer = WeightedScorer(_sample_weight, accuracy_score)
33grid = GridSearchCV(
34    GradientBoostingClassifier(),
35    {'n_estimators': [50, 100]},
36    scoring=scorer,
37    cv=5
38)

Fix 3: Manual Cross-Validation (Most Control)

For full control over sample_weight in both fit and score:

python

1import numpy as np
2from sklearn.model_selection import StratifiedKFold
3from sklearn.metrics import accuracy_score, f1_score
4from sklearn.ensemble import GradientBoostingClassifier
5from itertools import product
6
7X = np.random.rand(200, 10)
8y = np.random.randint(0, 2, 200)
9sample_weight = np.random.rand(200)
10
11param_grid = {
12    'n_estimators': [50, 100, 200],
13    'max_depth': [3, 5, 7]
14}
15
16best_score = -1
17best_params = None
18cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
19
20for n_est, depth in product(param_grid['n_estimators'], param_grid['max_depth']):
21    fold_scores = []
22
23    for train_idx, val_idx in cv.split(X, y):
24        X_train, X_val = X[train_idx], X[val_idx]
25        y_train, y_val = y[train_idx], y[val_idx]
26        sw_train, sw_val = sample_weight[train_idx], sample_weight[val_idx]
27
28        clf = GradientBoostingClassifier(n_estimators=n_est, max_depth=depth)
29        clf.fit(X_train, y_train, sample_weight=sw_train)
30
31        y_pred = clf.predict(X_val)
32
33        # Score WITH sample_weight on validation set
34        score = accuracy_score(y_val, y_pred, sample_weight=sw_val)
35        fold_scores.append(score)
36
37    mean_score = np.mean(fold_scores)
38    if mean_score > best_score:
39        best_score = mean_score
40        best_params = {'n_estimators': n_est, 'max_depth': depth}
41
42print(f"Best params: {best_params}")
43print(f"Best weighted accuracy: {best_score:.4f}")

Fix 4: Metadata Routing (scikit-learn 1.4+)

Scikit-learn 1.4 introduced metadata routing, which allows sample_weight to flow to both fit and score:

python

1import sklearn
2from sklearn.model_selection import GridSearchCV
3from sklearn.ensemble import GradientBoostingClassifier
4from sklearn.metrics import make_scorer, accuracy_score
5import numpy as np
6
7# Enable metadata routing
8sklearn.set_config(enable_metadata_routing=True)
9
10scorer = make_scorer(accuracy_score).set_score_request(sample_weight=True)
11
12clf = GradientBoostingClassifier()
13clf.set_fit_request(sample_weight=True)
14
15grid = GridSearchCV(clf, {'n_estimators': [50, 100]}, scoring=scorer, cv=5)
16
17X = np.random.rand(200, 5)
18y = np.random.randint(0, 2, 200)
19sw = np.random.rand(200)
20
21# sample_weight is now routed to BOTH fit and score
22grid.fit(X, y, sample_weight=sw)
23print(f"Best score (weighted): {grid.best_score_:.4f}")

This is the cleanest solution but requires scikit-learn 1.4 or later.

When Sample Weight Matters

python

1# Imbalanced dataset example
2from sklearn.utils.class_weight import compute_sample_weight
3
4y = np.array([0]*950 + [1]*50)  # 95% class 0, 5% class 1
5
6# Compute weights to balance classes
7sample_weight = compute_sample_weight('balanced', y)
8# Class 0 samples get weight ~0.53, class 1 samples get weight ~10.0
9
10# Without weighted scoring, accuracy favors the majority class
11# With weighted scoring, minority class errors are penalized proportionally

Common Pitfalls

Assuming GridSearchCV passes sample_weight to the scorer automatically: It does not (before metadata routing). The scorer receives only (estimator, X, y). sample_weight in fit_params only affects estimator.fit(), not the evaluation score.
Using make_scorer with a fixed sample_weight array: Passing the full sample_weight array to make_scorer uses all weights on every fold's validation set, not just the weights for the validation indices. This produces incorrect scores when fold sizes differ from the full dataset.
Forgetting that class_weight in the estimator is different from sample_weight in the scorer: class_weight='balanced' on the estimator adjusts training, but the scorer still evaluates unweighted unless explicitly configured. Both mechanisms serve different purposes.
Not using StratifiedKFold with imbalanced data: Default KFold may create folds where the minority class is absent. Always use StratifiedKFold (the default for classification in GridSearchCV) when working with imbalanced datasets and sample weights.
Expecting metadata routing to work without set_config: In scikit-learn 1.4+, metadata routing must be explicitly enabled with sklearn.set_config(enable_metadata_routing=True). Without this, sample_weight is silently ignored in the scorer.

Summary

GridSearchCV does not pass sample_weight to the scoring function by default
For scikit-learn 1.4+, use metadata routing with set_score_request(sample_weight=True)
For older versions, write a manual cross-validation loop for full control over weighted scoring
Use compute_sample_weight('balanced', y) to create weights for imbalanced datasets
Always verify that both training (fit) and evaluation (score) use sample_weight for consistent model selection

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

sklearn GridSearchCV not using sample_weight in score function

ML System Design practice on Codemia

Introduction

The Problem

Fix 1: Custom Scorer with make_scorer

Fix 2: Custom Scoring Callable

Fix 3: Manual Cross-Validation (Most Control)

Fix 4: Metadata Routing (scikit-learn 1.4+)

When Sample Weight Matters

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

Track what you have practised

ML System Design practice on Codemia