Understanding Memory Usage of Neural Networks with Keras

The memory consumption of neural networks during training and inference is a critical aspect that affects performance, scalability, and the feasibility of deploying models on various hardware platforms. In this article, we delve into how memory is utilized by neural networks implemented using Keras, a popular high-level deep learning framework. We will examine how model parameters, intermediate data, batch sizes, and memory optimization techniques play a role in the overall memory usage.

Key Factors Affecting Memory Usage

1. Model Parameters: • The primary consumers of memory are the model parameters or weights. Each parameter, usually stored as a 32-bit floating-point number, requires approximately 4 bytes. For a network with millions of parameters, the memory cost can be substantial.
2. Activations: • During the forward pass, neural networks store intermediate activations needed for backpropagation. These activations can occupy a large portion of the memory, especially in deep architectures.
3. Gradients: • The gradients computed for each parameter during backpropagation also require memory. Like weights and activations, gradients are usually stored as 32-bit floats.
4. Batch Size: • The batch size determines how many examples are processed simultaneously. This impacts memory usage since more data is being loaded into memory at once.
5. Model Architecture: • Different layers have varying memory consumptions — convolutional layers, for instance, generally require more memory than fully connected layers due to the way activations are convolved and stored.

Memory Usage Calculation Example

Consider a simple neural network with one fully connected layer, a single hidden layer, and an output layer. Let’s calculate the memory usage:

• Input shape: (batch_size, 256) • Hidden layer: 128 neurons • Output layer: 10 neurons • Weights and Biases: • First Layer: $(256 \times 128 + 128) \times 4 \text{ bytes}$ • Second Layer: $(128 \times 64 + 64) \times 4 \text{ bytes}$ • Output Layer: $(64 \times 10 + 10) \times 4 \text{ bytes}$ • Activations: • Calculated similarly and depend on batch size. • Trains models by trading compute for memory. Intermediate activations are discarded and recomputed during backpropagation to save memory. • Using mixed precision (e.g., `float16` instead of `float32`) to reduce memory usage. TensorFlow and Keras support this through the `tf.keras.mixed_precision` API. • Allows memory for activations to be reused after they are no longer needed in backpropagation. • Techniques to reduce the model size and hence, memory footprint. Pruning removes redundant weights, while quantization reduces the precision of the weights. • Reducing batch size can aid in fitting models into memory, albeit at the potential cost of slower convergence rates.