Introduction

Batch normalization (Batch Norm) has become an integral component of deep learning models, improving training speed and performance stability. By normalizing the inputs to each layer, batch normalization mitigates the problem of internal covariate shift, leading to models that converge faster and generalize better. However, using batch normalization introduces complexities, especially when it comes to restoring and deploying models. This article provides a comprehensive look into using batch normalization when restoring models, exploring technical intricacies, practical considerations, and useful tips.

Understanding Batch Normalization

Batch normalization is primarily used to improve the performance and stability of neural networks. It normalizes the output from a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation. To allow the network to learn optimal representations, trainable parameters $\gamma$ (scale) and $\beta$ (shift) are introduced.

Mathematically Defined

Given an input batch $x = (x_1, ... , x_m)$, batch normalization transforms these inputs as follows:

1. Compute the mean: $$\mu_{\text{batch}} = \frac{1}{m} \sum_{i=1}^{m} x_i$$
2. Compute the variance: $$\sigma_{\text{batch}}^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu_{\text{batch}})^2$$
3. Normalize: $$\hat{x}_i = \frac{x_i - \mu_{\text{batch}}}{\sqrt{\sigma_{\text{batch}}^2 + \epsilon}}$$
4. Scale and shift: $$y_i = \gamma \hat{x}_i + \beta$$

• Where $\epsilon$ is a small constant to avoid division by zero.

Using Batch Norm When Restoring Models

Key Considerations

1. Track Running Statistics: During training, the network keeps track of running mean and variance, which are critical for the normalization of data during inference.
2. Training vs. Inference: During training, batch normalization uses the statistics of the current mini-batch. In contrast, during inference, it uses the curated running statistics from the training phase to ensure stability and consistency.
3. Restoration Process: When restoring a model that includes batch normalization:
   • Restore model weights including $\gamma$ and $\beta$.
   • Restore running statistics — the running mean and variance.
   • Ensure to set the network from training mode to inference mode. Many deep learning frameworks like TensorFlow and PyTorch offer functions to toggle between these modes (`model.eval()` in PyTorch, for example).

Framework-Specific Implementation

Here's how batch normalization is handled in two popular frameworks during model restoration:

PyTorch Example