Introduction

Fine-tuning InceptionV3 in Keras means starting from ImageNet-trained weights and adapting them to your own dataset. The most reliable workflow is to train a new classifier head first with the backbone frozen, then unfreeze only part of the model and continue with a much smaller learning rate.

Why Two-Stage Training Works

InceptionV3 already knows many useful visual features such as edges, textures, and object parts. If you unfreeze the whole network immediately on a small dataset, random gradients from the new task can damage those pretrained features.

A safer process is:

1. freeze the pretrained base
2. train only the new classification head
3. unfreeze upper layers
4. fine-tune gently with a low learning rate

That is the standard transfer-learning pattern in Keras.

Build the Input Pipeline

InceptionV3 expects images resized to 299 x 299 and preprocessed with the model-specific preprocessing function.

1import tensorflow as tf
2from tensorflow import keras
3
4IMG_SIZE = (299, 299)
5BATCH_SIZE = 32
6
7train_ds = keras.utils.image_dataset_from_directory(
8    "data/train",
9    image_size=IMG_SIZE,
10    batch_size=BATCH_SIZE,
11    label_mode="int",
12)
13
14val_ds = keras.utils.image_dataset_from_directory(
15    "data/val",
16    image_size=IMG_SIZE,
17    batch_size=BATCH_SIZE,
18    label_mode="int",
19)
20
21preprocess = keras.applications.inception_v3.preprocess_input
22AUTOTUNE = tf.data.AUTOTUNE
23
24train_ds = train_ds.map(lambda x, y: (preprocess(tf.cast(x, tf.float32)), y)).prefetch(AUTOTUNE)
25val_ds = val_ds.map(lambda x, y: (preprocess(tf.cast(x, tf.float32)), y)).prefetch(AUTOTUNE)

Do not skip the preprocessing step. Transfer learning works best when the new inputs are normalized the way the pretrained model expects.

Train a New Head with the Base Frozen

1num_classes = 5
2
3base_model = keras.applications.InceptionV3(
4    include_top=False,
5    weights="imagenet",
6    input_shape=(299, 299, 3),
7)
8base_model.trainable = False
9
10inputs = keras.Input(shape=(299, 299, 3))
11x = base_model(inputs, training=False)
12x = keras.layers.GlobalAveragePooling2D()(x)
13x = keras.layers.Dropout(0.3)(x)
14outputs = keras.layers.Dense(num_classes, activation="softmax")(x)
15model = keras.Model(inputs, outputs)
16
17model.compile(
18    optimizer=keras.optimizers.Adam(1e-3),
19    loss="sparse_categorical_crossentropy",
20    metrics=["accuracy"],
21)
22
23model.fit(train_ds, validation_data=val_ds, epochs=5)

The important detail is training=False when calling the frozen base model. That is especially helpful around batch normalization behavior.

Unfreeze the Top Layers Carefully

After the classifier head has learned something useful, unfreeze only the upper part of the backbone.

1base_model.trainable = True
2
3for layer in base_model.layers[:-60]:
4    layer.trainable = False
5
6model.compile(
7    optimizer=keras.optimizers.Adam(1e-5),
8    loss="sparse_categorical_crossentropy",
9    metrics=["accuracy"],
10)
11
12callbacks = [
13    keras.callbacks.EarlyStopping(monitor="val_loss", patience=3, restore_best_weights=True),
14    keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.2, patience=2),
15]
16
17model.fit(train_ds, validation_data=val_ds, epochs=15, callbacks=callbacks)

The low learning rate is crucial. Fine-tuning is not a second round of ordinary training. It is a delicate adjustment of pretrained weights.

Batch Normalization Caution

InceptionV3 contains batch normalization layers. Those layers can destabilize fine-tuning if they are updated too aggressively, especially on small datasets.

That is why many practitioners:

• keep most of the network frozen
• use a low learning rate
• keep the base model called with training=False during some fine-tuning setups

If validation performance suddenly collapses after unfreezing, batch normalization behavior is one of the first things to inspect.

Add Realistic Augmentation

Small datasets often overfit quickly, so light augmentation is useful:

1augment = keras.Sequential([
2    keras.layers.RandomFlip("horizontal"),
3    keras.layers.RandomRotation(0.1),
4    keras.layers.RandomZoom(0.1),
5])

Use augmentation that matches the domain. Aggressive transforms can hurt more than they help if they create unrealistic samples.

Common Pitfalls

One common mistake is fine-tuning the whole model from the start. That often destroys pretrained features before the new classifier head has stabilized.

Another issue is using too large a learning rate after unfreezing. Even a rate that was fine for the classifier head can be too large for the pretrained backbone.

A third pitfall is forgetting model-specific preprocessing. InceptionV3 transfer learning assumes inputs are normalized the expected way.

Summary

• Fine-tune InceptionV3 in two stages: frozen backbone first, partial unfreeze second.
• Resize inputs to 299 x 299 and use InceptionV3 preprocessing.
• Use a much smaller learning rate during the unfreeze stage.
• Be careful with batch normalization behavior during fine-tuning.
• Add realistic augmentation and monitor validation metrics closely.