Introduction

Transfer learning works best when the input pipeline is as disciplined as the model code. A strong pretrained backbone can still perform badly if images are resized inconsistently, normalized the wrong way, or fed through a training pipeline that differs from inference.

Build the Dataset With Explicit Shapes and Labels

For many image-classification tasks, image_dataset_from_directory is the easiest reliable starting point. It creates a tf.data pipeline with labels inferred from folder names.

1import tensorflow as tf
2
3IMAGE_SIZE = (224, 224)
4BATCH_SIZE = 32
5SEED = 42
6
7train_ds = tf.keras.utils.image_dataset_from_directory(
8    "data/train",
9    image_size=IMAGE_SIZE,
10    batch_size=BATCH_SIZE,
11    label_mode="int",
12    shuffle=True,
13    seed=SEED,
14)
15
16val_ds = tf.keras.utils.image_dataset_from_directory(
17    "data/val",
18    image_size=IMAGE_SIZE,
19    batch_size=BATCH_SIZE,
20    label_mode="int",
21    shuffle=False,
22)

The important part is consistency. Folder names become class labels, so any directory rename changes the label mapping. Save that mapping with the model if you need stable serving behavior later.

Match Preprocessing to the Backbone

Pretrained models are not interchangeable at the input stage. MobileNet, EfficientNet, and ResNet families expect different value ranges or preprocessing logic. Use the preprocessing function that matches the backbone you load.

1from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2, preprocess_input
2
3AUTOTUNE = tf.data.AUTOTUNE
4
5train_ds = train_ds.map(
6    lambda images, labels: (preprocess_input(images), labels),
7    num_parallel_calls=AUTOTUNE,
8)
9val_ds = val_ds.map(
10    lambda images, labels: (preprocess_input(images), labels),
11    num_parallel_calls=AUTOTUNE,
12)

Using generic scaling when the model expects a specific preprocessing function is one of the easiest ways to quietly reduce transfer-learning quality.

Freeze the Backbone First

A common training pattern is to freeze the pretrained base model at the beginning and train only a small classification head. That lets the top layers adapt to the new dataset before you start updating deeper pretrained weights.

1base_model = MobileNetV2(
2    include_top=False,
3    weights="imagenet",
4    input_shape=(224, 224, 3),
5)
6base_model.trainable = False
7
8model = tf.keras.Sequential(
9    [
10        base_model,
11        tf.keras.layers.GlobalAveragePooling2D(),
12        tf.keras.layers.Dropout(0.2),
13        tf.keras.layers.Dense(5, activation="softmax"),
14    ]
15)
16
17model.compile(
18    optimizer=tf.keras.optimizers.Adam(1e-3),
19    loss="sparse_categorical_crossentropy",
20    metrics=["accuracy"],
21)

This stage is usually faster and more stable than jumping straight into full fine-tuning.

Fine-Tune Gradually and With a Lower Learning Rate

After the classification head has learned something useful, unfreeze part of the backbone and continue training with a much smaller learning rate.

1base_model.trainable = True
2
3for layer in base_model.layers[:-30]:
4    layer.trainable = False
5
6model.compile(
7    optimizer=tf.keras.optimizers.Adam(1e-5),
8    loss="sparse_categorical_crossentropy",
9    metrics=["accuracy"],
10)

The smaller learning rate matters because pretrained features are valuable. If you fine-tune too aggressively, the model can forget what it learned from the original large dataset.

Add Data Augmentation Carefully

Augmentation improves generalization, but only when the transformations reflect realistic variation in the target data. Extreme rotations or color shifts can fight the assumptions built into the pretrained model and the real deployment environment.

1augmentation = tf.keras.Sequential(
2    [
3        tf.keras.layers.RandomFlip("horizontal"),
4        tf.keras.layers.RandomRotation(0.05),
5        tf.keras.layers.RandomZoom(0.1),
6    ]
7)
8
9train_ds = train_ds.map(
10    lambda images, labels: (augmentation(images, training=True), labels),
11    num_parallel_calls=AUTOTUNE,
12)

For many transfer-learning tasks, modest augmentation beats aggressive augmentation because it expands the dataset without destroying the signal the pretrained backbone already understands.

Keep Training and Inference Pipelines Identical

One of the most expensive mistakes is training with one resize and preprocessing path, then serving with another. If the deployed code decodes images differently or skips the model-specific preprocessing step, your validation metrics will not match real behavior.

A good habit is to keep preprocessing in one reusable function and use it in both training and inference code. Also save the class names alongside the model artifact so prediction indices can be interpreted correctly later.

Common Pitfalls

The most common problems are preprocessing mismatch, inconsistent image size, and over-aggressive fine-tuning. Teams also forget to measure data-pipeline throughput, which leads to slow training where the GPU waits idle while the CPU decodes images. Another frequent issue is tracking only overall accuracy while missing class imbalance or poor per-class recall.

Summary

• Build TensorFlow image datasets with explicit image size, label mode, and shuffle behavior.
• Use the preprocessing function that matches the pretrained backbone.
• Freeze the base model first, then fine-tune selectively with a lower learning rate.
• Apply realistic augmentation instead of maximizing distortion.
• Keep training and inference preprocessing identical so evaluation matches deployment.