Introduction

If you want feature importance from cross-validation, cross_val_score is not enough. You need to fit a fresh model inside each fold, extract the fitted model's importance values, and then summarize them across folds. That gives you both per-fold importances and a way to judge whether the ranking is stable or highly split-dependent.

Fit the Model Inside the CV Loop

The core workflow is simple:

1. create a cross-validation splitter
2. loop over train and validation indices
3. fit a new estimator on the training split
4. read the fitted estimator's importance measure
5. store the result for later aggregation

For tree-based models, feature_importances_ is the usual source:

1import pandas as pd
2from sklearn.datasets import load_breast_cancer
3from sklearn.ensemble import RandomForestClassifier
4from sklearn.model_selection import StratifiedKFold
5
6X, y = load_breast_cancer(return_X_y=True, as_frame=True)
7cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
8
9rows = []
10
11for fold, (train_idx, test_idx) in enumerate(cv.split(X, y), start=1):
12    X_train = X.iloc[train_idx]
13    y_train = y.iloc[train_idx]
14
15    model = RandomForestClassifier(random_state=42)
16    model.fit(X_train, y_train)
17
18    rows.append(
19        pd.Series(model.feature_importances_, index=X.columns, name=f"fold_{fold}")
20    )
21
22importance_df = pd.DataFrame(rows)
23print(importance_df.head())

Now each row is one fitted model from one fold.

Summarize Stability Across Folds

Once the fold matrix exists, compute the mean and spread of each feature's importance:

1summary = pd.DataFrame({
2    "mean_importance": importance_df.mean(axis=0),
3    "std_importance": importance_df.std(axis=0),
4}).sort_values("mean_importance", ascending=False)
5
6print(summary.head(10))

The mean tells you which features are usually important. The standard deviation tells you whether that importance is stable or highly sensitive to the split.

That second number is often the real reason to do this analysis. A feature with a high average importance but huge fold-to-fold variation should be interpreted more carefully than one that stays consistently near the top.

Use the Right Importance Measure for the Estimator

Not every estimator exposes feature_importances_. Linear models usually use coefficients instead. If you compare coefficient magnitude, scaling matters, so use the same preprocessing inside every fold.

1import numpy as np
2import pandas as pd
3from sklearn.linear_model import LogisticRegression
4from sklearn.pipeline import make_pipeline
5from sklearn.preprocessing import StandardScaler
6
7coef_rows = []
8
9for fold, (train_idx, test_idx) in enumerate(cv.split(X, y), start=1):
10    X_train = X.iloc[train_idx]
11    y_train = y.iloc[train_idx]
12
13    model = make_pipeline(
14        StandardScaler(),
15        LogisticRegression(max_iter=2000)
16    )
17    model.fit(X_train, y_train)
18
19    coef = model.named_steps["logisticregression"].coef_[0]
20    coef_rows.append(pd.Series(np.abs(coef), index=X.columns, name=f"fold_{fold}"))
21
22coef_df = pd.DataFrame(coef_rows)
23print(coef_df.mean().sort_values(ascending=False).head())

For model-agnostic analysis, permutation importance is the usual fallback. It is slower, but it works for models that do not have built-in importance attributes.

1from sklearn.inspection import permutation_importance
2from sklearn.svm import SVC
3
4X_train = X.iloc[train_idx]
5X_test = X.iloc[test_idx]
6y_train = y.iloc[train_idx]
7y_test = y.iloc[test_idx]
8
9model = SVC(kernel="rbf", probability=True)
10model.fit(X_train, y_train)
11
12result = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)
13print(result.importances_mean[:5])

A simple fold matrix also makes plotting straightforward. Once the values are stored in a dataframe, box plots or error bars can reveal whether a feature is consistently important or only occasionally dominant.

Common Pitfalls

The biggest mistake is fitting one model on the full dataset and calling that a cross-validation importance analysis. Per-fold importance means the estimator must be refit separately in each split.

Another issue is treating all importance measures as equivalent. Tree impurity importance, absolute coefficients, and permutation importance reflect different properties and often produce different rankings.

People also often ignore preprocessing. For coefficient-based models, scaling changes magnitude dramatically, so importance numbers are only comparable if the pipeline is consistent inside each fold.

Finally, do not over-read tiny rank swaps. Correlated features can trade places across folds without meaning the model is unstable in a practically important way.

Summary

• Fit a fresh estimator inside each cross-validation fold.
• Extract per-fold importance values from the fitted model, not from a single global fit.
• Aggregate mean and variability to understand both ranking and stability.
• Use an importance method that matches the estimator type.
• Treat cross-validated feature importance as a diagnostic tool, not as an absolute truth about the data.