Introduction
The hidden_layer_sizes parameter in scikit-learn's MLPRegressor defines the neural network architecture — the number of hidden layers and neurons per layer. Tuning this parameter with Optuna requires special handling because it is a tuple of variable length, not a simple scalar. This article shows how to use Optuna's trial suggestions to search over both the number of layers and the number of neurons per layer.
The Challenge
hidden_layer_sizes accepts a tuple like (100,) (one hidden layer with 100 neurons) or (64, 32, 16) (three hidden layers). Optuna's trial.suggest_int() returns a single value, so you need to build the tuple dynamically.
Basic Approach
1import optuna
2from sklearn.neural_network import MLPRegressor
3from sklearn.model_selection import cross_val_score
4from sklearn.datasets import make_regression
5
6X, y = make_regression(n_samples=1000, n_features=20, random_state=42)
7
8def objective(trial):
9 # Number of hidden layers (1 to 4)
10 n_layers = trial.suggest_int('n_layers', 1, 4)
11
12 # Neurons per layer
13 layers = []
14 for i in range(n_layers):
15 n_neurons = trial.suggest_int(f'n_neurons_layer_{i}', 16, 256)
16 layers.append(n_neurons)
17
18 hidden_layer_sizes = tuple(layers)
19
20 model = MLPRegressor(
21 hidden_layer_sizes=hidden_layer_sizes,
22 max_iter=500,
23 random_state=42,
24 )
25
26 scores = cross_val_score(model, X, y, cv=3, scoring='neg_mean_squared_error')
27 return scores.mean()
28
29study = optuna.create_study(direction='maximize')
30study.optimize(objective, n_trials=50)
31
32print(f"Best params: {study.best_params}")
33print(f"Best score: {study.best_value:.4f}")
Full Example with All Hyperparameters
Tune the architecture alongside learning rate, activation, and solver:
1import optuna
2from sklearn.neural_network import MLPRegressor
3from sklearn.model_selection import cross_val_score
4from sklearn.preprocessing import StandardScaler
5from sklearn.pipeline import Pipeline
6from sklearn.datasets import make_regression
7
8X, y = make_regression(n_samples=1000, n_features=20, noise=10, random_state=42)
9
10def objective(trial):
11 # Architecture
12 n_layers = trial.suggest_int('n_layers', 1, 3)
13 layers = tuple(
14 trial.suggest_int(f'n_neurons_{i}', 32, 256, log=True)
15 for i in range(n_layers)
16 )
17
18 # Other hyperparameters
19 activation = trial.suggest_categorical('activation', ['relu', 'tanh', 'logistic'])
20 solver = trial.suggest_categorical('solver', ['adam', 'sgd', 'lbfgs'])
21 alpha = trial.suggest_float('alpha', 1e-5, 1e-1, log=True)
22 learning_rate_init = trial.suggest_float('learning_rate_init', 1e-4, 1e-1, log=True)
23
24 model = Pipeline([
25 ('scaler', StandardScaler()),
26 ('mlp', MLPRegressor(
27 hidden_layer_sizes=layers,
28 activation=activation,
29 solver=solver,
30 alpha=alpha,
31 learning_rate_init=learning_rate_init,
32 max_iter=1000,
33 early_stopping=True,
34 random_state=42,
35 ))
36 ])
37
38 scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_squared_error')
39 return scores.mean()
40
41study = optuna.create_study(direction='maximize')
42study.optimize(objective, n_trials=100, show_progress_bar=True)
43
44# Print results
45print(f"\nBest score: {study.best_value:.4f}")
46for key, value in study.best_params.items():
47 print(f" {key}: {value}")
48
49# Reconstruct the best architecture
50best = study.best_params
51n_layers = best['n_layers']
52best_architecture = tuple(best[f'n_neurons_{i}'] for i in range(n_layers))
53print(f"\nBest architecture: {best_architecture}")
Using Logarithmic Sampling
For neuron counts, logarithmic sampling often works better because the difference between 32 and 64 neurons is more significant than between 224 and 256:
1# Linear sampling (uniform between 16 and 256)
2n_neurons = trial.suggest_int(f'n_neurons_{i}', 16, 256)
3
4# Logarithmic sampling (favors powers of 2 range)
5n_neurons = trial.suggest_int(f'n_neurons_{i}', 16, 256, log=True)
Constraining Architecture Shape
You may want architectures that taper (shrink from input to output) or maintain width:
Tapering Network
1def objective(trial):
2 n_layers = trial.suggest_int('n_layers', 1, 4)
3 max_neurons = trial.suggest_int('max_neurons', 64, 512, log=True)
4
5 # Each layer is smaller than the previous
6 layers = []
7 for i in range(n_layers):
8 neurons = max(16, max_neurons // (2 ** i))
9 layers.append(neurons)
10
11 model = MLPRegressor(hidden_layer_sizes=tuple(layers), max_iter=500)
12 return cross_val_score(model, X, y, cv=3, scoring='neg_mean_squared_error').mean()
Fixed-Width Network
1def objective(trial):
2 n_layers = trial.suggest_int('n_layers', 1, 4)
3 n_neurons = trial.suggest_int('n_neurons', 32, 256, log=True)
4
5 # All layers have the same width
6 layers = tuple([n_neurons] * n_layers)
7
8 model = MLPRegressor(hidden_layer_sizes=layers, max_iter=500)
9 return cross_val_score(model, X, y, cv=3, scoring='neg_mean_squared_error').mean()
Optuna Features for Better Results
Pruning Unpromising Trials
1from optuna.integration import OptunaSearchCV
2
3# Or use pruning with manual callback
4study = optuna.create_study(
5 direction='maximize',
6 pruner=optuna.pruners.MedianPruner(n_startup_trials=10)
7)
Saving and Resuming Studies
1# Save to SQLite
2study = optuna.create_study(
3 study_name='mlp-tuning',
4 storage='sqlite:///optuna_study.db',
5 direction='maximize',
6 load_if_exists=True
7)
Common Pitfalls
Conditional parameters: When n_layers=2, parameters n_neurons_2 and n_neurons_3 are not used. Optuna handles this via conditional sampling, but be aware that the best_params dictionary will contain entries for all suggested parameters across all trials. Reconstruct the architecture using n_layers.
Scaling features: MLPRegressor is sensitive to feature scaling. Always use StandardScaler or MinMaxScaler in a pipeline, otherwise hyperparameter tuning results will be unreliable.
max_iter convergence: Low max_iter causes convergence warnings that make tuning unreliable. Set it high enough (500-1000) or enable early_stopping=True.
Reproducibility: Set random_state on both the model and the study sampler for reproducible results.
Efficient Sampling: Optuna uses Tree-structured Parzen Estimator (TPE) by default, which is effective for this type of mixed search space. Avoid switching to random sampling unless debugging.
Summary
Build hidden_layer_sizes as a tuple by suggesting n_layers first, then n_neurons per layer
Use log=True for neuron count suggestions to better explore the search space
Always scale features with StandardScaler in a pipeline
Reconstruct the best architecture from study.best_params using the n_layers value
Enable early_stopping=True and set sufficient max_iter to avoid convergence issues
