How to perform mean subtraction and normalization with Tensorflow

TensorFlow

mean subtraction

normalization

deep learning

data preprocessing

How to perform mean subtraction and normalization with Tensorflow

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Deep Learning TensorFlow & Keras Data Science

Introduction

Mean subtraction and normalization are essential preprocessing steps for neural networks. They center the data around zero and scale features to comparable ranges, which helps gradient descent converge faster and prevents features with large values from dominating the learning process. TensorFlow provides several ways to implement these operations.

Why Normalize?

Without normalization, features on different scales cause problems:

python

1# Raw features: vastly different scales
2# age: [20, 30, 40]        (range ~20)
3# salary: [50000, 80000]    (range ~30000)
4# Gradient descent oscillates because salary dominates the loss

After normalization, all features have similar scales, enabling smoother optimization.

Mean Subtraction

Center data by subtracting the mean of each feature:

python

1import tensorflow as tf
2import numpy as np
3
4# Sample data: (batch_size, features)
5data = tf.constant([[1.0, 200.0], [2.0, 400.0], [3.0, 600.0]])
6
7# Compute mean per feature (along axis 0)
8mean = tf.reduce_mean(data, axis=0)
9# mean = [2.0, 400.0]
10
11# Subtract mean
12centered = data - mean
13# [[-1.0, -200.0], [0.0, 0.0], [1.0, 200.0]]

Normalization Methods

Min-Max Normalization (Scale to [0, 1])

python

1data = tf.constant([[1.0, 200.0], [2.0, 400.0], [3.0, 600.0]])
2
3min_val = tf.reduce_min(data, axis=0)
4max_val = tf.reduce_max(data, axis=0)
5
6normalized = (data - min_val) / (max_val - min_val)
7# [[0.0, 0.0], [0.5, 0.5], [1.0, 1.0]]

Z-Score Normalization (Standardization)

python

1mean = tf.reduce_mean(data, axis=0)
2std = tf.math.reduce_std(data, axis=0)
3
4standardized = (data - mean) / std
5# Each feature now has mean ≈ 0 and std ≈ 1

Per-Image Normalization

Common for image data:

python

1# Normalize each image independently
2image = tf.random.uniform([224, 224, 3], 0, 255)
3
4# TensorFlow built-in
5normalized = tf.image.per_image_standardization(image)
6# Result: mean ≈ 0, std ≈ 1 for each image
7
8# Manual
9mean = tf.reduce_mean(image)
10std = tf.math.reduce_std(image)
11normalized = (image - mean) / tf.maximum(std, 1.0 / tf.sqrt(tf.cast(tf.size(image), tf.float32)))

Using tf.keras.layers.Normalization

The Keras Normalization layer adapts to data and applies z-score normalization:

python

1import tensorflow as tf
2
3# Create the layer
4normalizer = tf.keras.layers.Normalization(axis=-1)
5
6# Adapt to training data (computes mean and variance)
7train_data = np.array([[1.0, 200.0], [2.0, 400.0], [3.0, 600.0]])
8normalizer.adapt(train_data)
9
10# Use in a model
11model = tf.keras.Sequential([
12    normalizer,
13    tf.keras.layers.Dense(64, activation='relu'),
14    tf.keras.layers.Dense(1)
15])
16
17# The normalizer automatically applies mean subtraction and scaling
18predictions = model.predict(train_data)

Using tf.keras.layers.Rescaling

For simple scaling (e.g., pixel values):

python

1# Scale pixel values from [0, 255] to [0, 1]
2rescale = tf.keras.layers.Rescaling(scale=1./255)
3
4model = tf.keras.Sequential([
5    rescale,
6    tf.keras.layers.Conv2D(32, 3, activation='relu'),
7    # ...
8])
9
10# Or scale to [-1, 1]
11rescale = tf.keras.layers.Rescaling(scale=1./127.5, offset=-1)

ImageNet-Style Normalization

Pre-trained models (ResNet, VGG, etc.) expect ImageNet normalization:

python

1# ImageNet mean and std per channel (RGB)
2IMAGENET_MEAN = tf.constant([0.485, 0.456, 0.406])
3IMAGENET_STD = tf.constant([0.229, 0.224, 0.225])
4
5def imagenet_normalize(image):
6    """Normalize image for ImageNet pre-trained models."""
7    image = tf.cast(image, tf.float32) / 255.0
8    return (image - IMAGENET_MEAN) / IMAGENET_STD
9
10# Or use the built-in preprocess_input
11from tensorflow.keras.applications.resnet50 import preprocess_input
12normalized = preprocess_input(image)  # handles normalization automatically

Batch Normalization During Training

Batch normalization normalizes activations within the network during training:

python

1model = tf.keras.Sequential([
2    tf.keras.layers.Dense(128),
3    tf.keras.layers.BatchNormalization(),
4    tf.keras.layers.Activation('relu'),
5    tf.keras.layers.Dense(64),
6    tf.keras.layers.BatchNormalization(),
7    tf.keras.layers.Activation('relu'),
8    tf.keras.layers.Dense(10, activation='softmax')
9])

Using tf.data Pipeline

Apply normalization in the data pipeline:

python

1def normalize_fn(image, label):
2    image = tf.cast(image, tf.float32) / 255.0
3    mean = tf.constant([0.485, 0.456, 0.406])
4    std = tf.constant([0.229, 0.224, 0.225])
5    image = (image - mean) / std
6    return image, label
7
8dataset = tf.data.Dataset.from_tensor_slices((images, labels))
9dataset = dataset.map(normalize_fn).batch(32).prefetch(tf.data.AUTOTUNE)

Common Pitfalls

Test Data Consistency: It is critical to apply the same mean subtraction and normalization parameters (mean, min, max, std deviation) calculated from the training data to the test and validation datasets. Never compute statistics from test data.
Feature-Wise Normalization: Each feature should be normalized independently for optimal performance. Do not compute a single mean/std across all features.
Batch Normalization: During model training, TensorFlow offers Batch Normalization as an additional normalization technique that normalizes intermediate activations, which is different from input normalization.
Division by zero: When the standard deviation is zero (constant feature), division fails. Add a small epsilon: (data - mean) / (std + 1e-7).
Integer overflow: Image data stored as uint8 overflows when subtracting the mean. Cast to float32 first: tf.cast(image, tf.float32).
Training vs inference: BatchNormalization behaves differently during training and inference. Always set training=True/False appropriately or use model.fit() / model.predict() which handle it automatically.

Summary

Mean subtraction centers data around zero; normalization scales features to comparable ranges
Use tf.keras.layers.Normalization with .adapt() for automatic z-score normalization
Use tf.keras.layers.Rescaling for simple value scaling (e.g., pixel values)
For pre-trained models, use the model's preprocess_input function for correct normalization
Always compute normalization statistics from training data and apply them to test/validation data

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

How to perform mean subtraction and normalization with Tensorflow

ML System Design practice on Codemia

Introduction

Why Normalize?

Mean Subtraction

Normalization Methods

Min-Max Normalization (Scale to [0, 1])

Z-Score Normalization (Standardization)

Per-Image Normalization

Using tf.keras.layers.Normalization

Using tf.keras.layers.Rescaling

ImageNet-Style Normalization

Batch Normalization During Training

Using tf.data Pipeline

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

Track what you have practised

ML System Design practice on Codemia