Introduction

In Caffe, BatchNorm is usually not used as a stand-alone replacement for everything that batch normalization means in papers or newer frameworks. In practice, you typically place a BatchNorm layer after a convolution or inner-product layer and then follow it with a Scale layer so the network can learn the equivalent of gamma and beta.

The Normal Caffe Pattern

Caffe's BatchNorm layer handles normalization statistics, but it does not usually provide the learnable scale and shift parameters by itself in the way many people expect. That is why Caffe examples commonly pair it with Scale.

A standard block looks like this:

1layer {
2  name: "conv1"
3  type: "Convolution"
4  bottom: "data"
5  top: "conv1"
6  convolution_param {
7    num_output: 64
8    kernel_size: 3
9    stride: 1
10    pad: 1
11    bias_term: false
12  }
13}
14
15layer {
16  name: "bn1"
17  type: "BatchNorm"
18  bottom: "conv1"
19  top: "conv1"
20  batch_norm_param {
21    use_global_stats: false
22    eps: 1e-5
23  }
24}
25
26layer {
27  name: "scale1"
28  type: "Scale"
29  bottom: "conv1"
30  top: "conv1"
31  scale_param {
32    bias_term: true
33  }
34}
35
36layer {
37  name: "relu1"
38  type: "ReLU"
39  bottom: "conv1"
40  top: "conv1"
41}

The order is usually convolution, normalization, scaling, then activation.

Why Scale Is Important

Batch normalization has two phases:

• Normalize using batch or running statistics
• Learn an affine transform after normalization

In Caffe, the affine part is commonly implemented by the Scale layer with bias_term: true. If you skip Scale, the network loses the learnable post-normalization rescaling step, which often hurts training.

That is also why many Caffe model definitions disable the bias term in the preceding convolution. Once batch normalization and scaling are in place, the extra convolution bias becomes unnecessary.

Training Versus Inference

During training, batch normalization should use statistics computed from the current mini-batch. During inference, it should use the running estimates accumulated during training.

In practice, that means:

• Training prototxt files usually use use_global_stats: false
• Test or deploy prototxt files use use_global_stats: true

If you evaluate a model with training-mode statistics by mistake, predictions may jump around based on batch composition instead of reflecting the learned moving averages.

Parameter Handling in Solver Definitions

Another Caffe-specific detail is that the internal blobs used by BatchNorm are not trained the same way as ordinary weights. Many models explicitly freeze them with zero learning-rate multipliers:

1layer {
2  name: "bn1"
3  type: "BatchNorm"
4  bottom: "conv1"
5  top: "conv1"
6  param { lr_mult: 0 }
7  param { lr_mult: 0 }
8  param { lr_mult: 0 }
9}

This pattern prevents the solver from trying to update running-statistics blobs as if they were normal trainable parameters.

When to Insert Batch Normalization

The most common use is after Convolution or InnerProduct, before the nonlinearity. In older Caffe architectures, you add it selectively to deeper parts of the network where optimization is unstable or sensitive to initialization.

For example, a fully connected block follows the same idea:

1layer {
2  name: "fc1"
3  type: "InnerProduct"
4  bottom: "pool5"
5  top: "fc1"
6  inner_product_param {
7    num_output: 256
8    bias_term: false
9  }
10}

Then add BatchNorm, Scale, and ReLU in the same sequence.

Common Pitfalls

• Using BatchNorm without a following Scale layer and then wondering why training quality drops.
• Leaving use_global_stats in the wrong mode for inference.
• Keeping a bias term on the previous layer when batch normalization plus scaling already covers that role.
• Treating BatchNorm blobs like ordinary trainable weights in the solver.

Summary

• In Caffe, BatchNorm is usually paired with Scale to recover learnable scale and shift.
• The usual placement is after convolution or fully connected layers, before activation.
• Training uses batch statistics; inference should use global running statistics.
• Many models set the preceding layer's bias_term to false.
• Getting the train versus test configuration right is just as important as adding the layer itself.