In training neural networks with Keras, one frequently encounters the parameters steps_per_epoch and validation_steps when working with models that utilize data generators or when dealing with large datasets that cannot fit into memory. Setting these parameters correctly is crucial for efficient training and for obtaining meaningful evaluation metrics. Here, we’ll explore how to properly determine these values, backed with technical insights and examples.

Understanding steps_per_epoch

and validation_steps

steps_per_epoch

steps_per_epoch defines the number of batches of samples to be processed before declaring one epoch finished during the training phase. It is a critical parameter when using methods like fit_generator , timeseries_generator , or when a custom generator is utilized.

• Formula:
  • If you're shuffling your dataset on each epoch and you have a dataset size of N and a batch size of B , generally, steps_per_epoch = N / B .
  • If data augmentation or random transformations are applied which increase the dataset size virtually, adjust steps_per_epoch accordingly to balance computational load.

validation_steps

This parameter informs how many batches from the validation generator should be used for validation after each epoch. It is essential when validation data is provided as a generator.

• Formula:
  • For a validation set of size V with a batch size B , validation_steps = V / B .

Practical Considerations

1. Data Size and Memory Constraints:
   • For very large datasets, utilizing a generator with steps_per_epoch allows batching data. This is especially critical when the entire dataset cannot fit into memory.
2. Data Distribution:
   • When data is imbalanced, each epoch should still cover a representative sample by ensuring steps_per_epoch is set correctly to incorporate class balance.
3. Performance and Convergence:
   • A too-small steps_per_epoch may result in overfitting during training as weights are updated more frequently with less data per epoch.
   • Conversely, a large steps_per_epoch can lead to increased training time without significant accuracy improvement.
4. Epoch Definition:
   • The concept of an epoch traditionally means passing through the complete dataset once. Adjust steps_per_epoch and validation_steps to honor this principle whenever feasible.

Example Implementation

Consider a CIFAR-10 dataset used in a small CNN model. Assume the training set has 50,000 samples and validation set has 10,000 samples. If using a batch size of 32, here’s how we calculate:

• Dynamic Adjustments:
  • During training, monitor accuracy and loss. Adjust steps_per_epoch and validation_steps if the network appears overfitting or underfitting.
• Custom Generators:
  • For bespoke data processing pipelines, custom Python generators may be used. These should yield tuples (inputs, targets) , or (inputs, targets, sample_weights) , with steps_per_epoch and validation_steps still applicable.
• Environmental Constraints:
  • computational resources (e.g., CPU vs GPU) might also affect the choice of steps_per_epoch significantly in terms of processing speed limitations.