Introduction
The warning "One or more of the test scores are non-finite" in RandomizedSearchCV occurs when a parameter combination produces NaN or inf scores during cross-validation. When tuning max_features for RandomForestClassifier or RandomForestRegressor, this typically happens when max_features is set to a value larger than the actual number of features in the dataset, or when max_features is set to "auto" (deprecated) on a dataset where features drop to zero after preprocessing. The fix is to constrain the max_features search range to valid values for your dataset.
Reproducing the Warning
1from sklearn.ensemble import RandomForestClassifier
2from sklearn.model_selection import RandomizedSearchCV
3from sklearn.datasets import make_classification
4import numpy as np
5
6X, y = make_classification(n_samples=100, n_features=10, random_state=42)
7
8# This parameter grid can cause the warning
9param_dist = {
10 'n_estimators': [50, 100, 200],
11 'max_depth': [5, 10, 20, None],
12 'max_features': [1, 5, 10, 15, 20], # 15 and 20 exceed n_features=10!
13}
14
15rf = RandomForestClassifier(random_state=42)
16search = RandomizedSearchCV(
17 rf, param_dist, n_iter=20, cv=5, scoring='accuracy', random_state=42
18)
19search.fit(X, y)
20# UserWarning: One or more of the test scores are non-finite
When max_features=15 or max_features=20 is sampled, it exceeds the 10 available features, causing the RandomForest to fail or produce degenerate splits that result in NaN scores.
Understanding max_features
1# max_features controls how many features each tree considers per split
2# Valid values:
3# int — exact number of features (must be <= n_features)
4# float — fraction of features (0.0 < x <= 1.0)
5# "sqrt" — sqrt(n_features)
6# "log2" — log2(n_features)
7# None — all features (n_features)
8
9# For a dataset with 10 features:
10# max_features=5 → consider 5 features per split
11# max_features=0.5 → consider 50% = 5 features per split
12# max_features="sqrt" → consider sqrt(10) ≈ 3 features
13# max_features=15 → ERROR: exceeds available features
Fix 1: Constrain max_features to Valid Range
1from sklearn.ensemble import RandomForestClassifier
2from sklearn.model_selection import RandomizedSearchCV
3import numpy as np
4
5X, y = make_classification(n_samples=100, n_features=10, random_state=42)
6n_features = X.shape[1]
7
8# Safe parameter distribution
9param_dist = {
10 'n_estimators': [50, 100, 200, 300],
11 'max_depth': [5, 10, 20, None],
12 'max_features': list(range(1, n_features + 1)), # 1 to 10
13 'min_samples_split': [2, 5, 10],
14 'min_samples_leaf': [1, 2, 4],
15}
16
17search = RandomizedSearchCV(
18 RandomForestClassifier(random_state=42),
19 param_dist,
20 n_iter=50,
21 cv=5,
22 scoring='accuracy',
23 random_state=42,
24 error_score='raise' # Raise instead of warn to see the actual error
25)
26search.fit(X, y)
27print(search.best_params_)
Fix 2: Use Fractional or String Values
1# Use fractions instead of absolute numbers — always valid
2param_dist = {
3 'n_estimators': [50, 100, 200],
4 'max_depth': [5, 10, 20, None],
5 'max_features': [0.1, 0.2, 0.3, 0.5, 0.7, 1.0], # Fractions of n_features
6}
7
8# Or use built-in string options
9param_dist = {
10 'n_estimators': [50, 100, 200],
11 'max_depth': [5, 10, 20, None],
12 'max_features': ['sqrt', 'log2', None], # Always valid
13}
14
15search = RandomizedSearchCV(
16 RandomForestClassifier(random_state=42),
17 param_dist,
18 n_iter=20,
19 cv=5,
20 random_state=42
21)
22search.fit(X, y)
Fractional values (0.0-1.0) are always valid regardless of the number of features, making them the safest choice for max_features in hyperparameter search.
Fix 3: Use error_score to Debug
1# error_score='raise' makes the search fail immediately on bad params
2# instead of silently replacing with NaN
3search = RandomizedSearchCV(
4 RandomForestClassifier(random_state=42),
5 param_dist,
6 n_iter=20,
7 cv=5,
8 error_score='raise' # Fail fast instead of warning
9)
10
11try:
12 search.fit(X, y)
13except Exception as e:
14 print(f"Failed with: {e}")
15 # Now you can see exactly which parameter caused the issue
1# After a search with warnings, inspect which combinations failed
2import pandas as pd
3
4results = pd.DataFrame(search.cv_results_)
5failed = results[results['mean_test_score'].isna()]
6print(failed[['params', 'mean_test_score']])
7# Shows exactly which parameter combinations produced NaN scores
Other Causes of Non-Finite Scores
1# Cause 1: NaN values in the dataset
2X_with_nan = X.copy()
3X_with_nan[0, 0] = np.nan
4# Fix: impute or drop NaN before fitting
5from sklearn.impute import SimpleImputer
6imputer = SimpleImputer(strategy='mean')
7X_clean = imputer.fit_transform(X_with_nan)
8
9# Cause 2: Infinite values
10X_with_inf = X.copy()
11X_with_inf[0, 0] = np.inf
12# Fix: replace inf with NaN then impute
13X_clean = np.where(np.isinf(X_with_inf), np.nan, X_with_inf)
14
15# Cause 3: Zero-variance features after train/test split
16# A feature that is constant in the test fold produces NaN importance
17# Fix: use VarianceThreshold before the search
18from sklearn.feature_selection import VarianceThreshold
19selector = VarianceThreshold(threshold=0.01)
20X_clean = selector.fit_transform(X)
21
22# Cause 4: Custom scorer returning NaN
23from sklearn.metrics import make_scorer
24
25def custom_scorer(y_true, y_pred):
26 if len(set(y_true)) == 1:
27 return float('nan') # Bug: returns NaN for single-class folds
28 return some_metric(y_true, y_pred)
29
30# Fix: handle edge cases in custom scorers
Common Pitfalls
Setting max_features as an integer larger than n_features: max_features=20 on a dataset with 10 features causes an error or degenerate behavior. Always compute n_features = X.shape[1] and constrain the search range to range(1, n_features + 1).
Using deprecated max_features="auto": In scikit-learn 1.1+, "auto" is deprecated and equivalent to "sqrt". Some versions produce warnings that cascade into non-finite scores. Replace "auto" with "sqrt" in your parameter grid.
Not checking for NaN/inf in the input data: RandomizedSearchCV may not catch NaN in training data during some parameter combinations (e.g., when a fold happens to have all valid values). Validate data before running the search with np.any(np.isnan(X)) and np.any(np.isinf(X)).
Using error_score=np.nan (default) and ignoring warnings: The default behavior replaces failed parameter combinations with NaN and continues searching. This hides real errors. Set error_score='raise' during development to catch issues immediately.
Forgetting that preprocessing changes feature count: If your pipeline includes feature selection or PCA before the RandomForest, max_features applies to the reduced feature set, not the original. A max_features=50 that was valid for raw data may exceed the 10 components output by PCA.
Summary
The "non-finite test scores" warning means some parameter combinations produced NaN during cross-validation
For max_features, constrain integer values to range(1, n_features + 1) or use fractions (0.0-1.0)
Use "sqrt", "log2", or None as safe string options for max_features
Set error_score='raise' to debug which parameter combinations fail
Check for NaN and inf in your dataset before running RandomizedSearchCV
Inspect cv_results_ after the search to identify which parameter combinations produced NaN scores
