Introduction

When sample weights seem to disappear during cross-validation, the issue is usually not that scikit-learn is broken. The real problem is that weights must be routed to the right consumer at the right stage, and older scikit-learn workflows handled that less cleanly than current ones.

Why This Feels Broken

sample_weight can affect at least two different places:

• the estimator’s fit method
• the scoring function used during evaluation

If weights reach one but not the other, results can look inconsistent. For example, you might train a weighted model but evaluate it with an unweighted metric, or try to pass weights through a helper that does not forward them as expected.

Current scikit-learn Approach

Modern scikit-learn uses metadata routing for this problem. The important idea is that the estimator and scorer explicitly request sample_weight, and cross_validate forwards it through params.

1import numpy as np
2from sklearn import set_config
3from sklearn.datasets import make_classification
4from sklearn.linear_model import LogisticRegression
5from sklearn.metrics import accuracy_score, make_scorer
6from sklearn.model_selection import KFold, cross_validate
7
8X, y = make_classification(random_state=42)
9sample_weight = np.where(y == 1, 2.0, 1.0)
10
11set_config(enable_metadata_routing=True)
12
13estimator = LogisticRegression(max_iter=1000).set_fit_request(sample_weight=True)
14scorer = make_scorer(accuracy_score).set_score_request(sample_weight=True)
15
16results = cross_validate(
17    estimator,
18    X,
19    y,
20    cv=KFold(n_splits=5, shuffle=True, random_state=42),
21    scoring={"weighted_acc": scorer},
22    params={"sample_weight": sample_weight},
23)
24
25print(results["test_weighted_acc"])

That code is explicit about who consumes the weights. This is safer than older patterns because scikit-learn can now reject metadata that was passed but never requested.

Older-Version Workaround

If you are maintaining an older scikit-learn version or using an estimator that does not participate in metadata routing the way you need, write the cross-validation loop yourself. It is more verbose, but it makes the weight flow obvious.

1import numpy as np
2from sklearn.base import clone
3from sklearn.linear_model import LogisticRegression
4from sklearn.metrics import accuracy_score
5from sklearn.model_selection import KFold
6
7X = np.random.randn(100, 4)
8y = (X[:, 0] + X[:, 1] > 0).astype(int)
9sample_weight = np.where(y == 1, 3.0, 1.0)
10
11cv = KFold(n_splits=5, shuffle=True, random_state=42)
12model = LogisticRegression(max_iter=1000)
13fold_scores = []
14
15for train_idx, test_idx in cv.split(X, y):
16    fitted = clone(model)
17    fitted.fit(X[train_idx], y[train_idx], sample_weight=sample_weight[train_idx])
18
19    predictions = fitted.predict(X[test_idx])
20    score = accuracy_score(y[test_idx], predictions, sample_weight=sample_weight[test_idx])
21    fold_scores.append(score)
22
23print(np.mean(fold_scores))

This approach works across versions and makes debugging far easier when weighted metrics matter.

Check Estimator and Metric Support

Not every estimator accepts sample_weight, and not every metric uses it. Before assuming the framework lost your weights, confirm that the estimator’s fit method and the metric you selected both support that argument.

That is especially important with custom scorers. A custom scorer that ignores weights can make a weighted training pipeline look wrong even when the fitting step behaved exactly as intended.

Weighted Fitting vs Weighted Scoring

Weighted fitting changes the model parameters. Weighted scoring changes how you judge predictions. Those are separate choices. Sometimes you want both. Sometimes you want only one.

Be explicit about which behavior you want instead of assuming all weighting choices should match. The clearest bug reports in this area usually come from code that states each intent separately.

Common Pitfalls

• Passing sample_weight to the estimator but not to the scorer can make the results look inconsistent.
• Assuming every estimator supports weighted fitting is unsafe. Check the API.
• Forgetting request configuration in metadata-routing workflows causes routing errors.
• Misaligning weight arrays with train and test indices is a common manual-loop bug.
• Calling the situation "broken" too early often hides the real issue, which is ambiguous weight flow.

Summary

• Sample weights in cross-validation are a routing problem more than a modeling problem.
• In current scikit-learn, use metadata routing with explicit requests and params.
• In older setups, a manual cross-validation loop is the most reliable fallback.
• Confirm that both the estimator and the scoring function actually support sample_weight.
• Decide explicitly whether you want weighted fitting, weighted scoring, or both.