Introduction

Fine-tuning Inception in TensorFlow is a standard transfer-learning workflow: start from ImageNet weights, replace the classification head, train the new head, and only then unfreeze part of the base model for a lower-learning-rate tuning pass. That approach gives you most of the benefit of a large pretrained model without paying the full cost of training from scratch.

The main mistake people make is unfreezing too much too early. Fine-tuning works best when you separate feature extraction from gradual adaptation.

Load the Base Model Correctly

For Keras-based TensorFlow code, InceptionV3 is the usual starting point. You typically drop the original top classifier and keep the convolutional base:

1import tensorflow as tf
2
3base_model = tf.keras.applications.InceptionV3(
4    include_top=False,
5    weights="imagenet",
6    input_shape=(299, 299, 3),
7)
8base_model.trainable = False

Two details matter here:

• Inception expects 299 x 299 input by default
• You should use the matching preprocessing function, not generic pixel scaling only

For datasets built from files, apply the Inception preprocessing step:

1train_ds = tf.keras.utils.image_dataset_from_directory(
2    "data/train",
3    image_size=(299, 299),
4    batch_size=32,
5)
6
7preprocess = tf.keras.applications.inception_v3.preprocess_input
8train_ds = train_ds.map(lambda x, y: (preprocess(x), y))

Add a New Classification Head

Once the base is frozen, attach a small task-specific head:

1inputs = tf.keras.Input(shape=(299, 299, 3))
2x = base_model(inputs, training=False)
3x = tf.keras.layers.GlobalAveragePooling2D()(x)
4x = tf.keras.layers.Dropout(0.2)(x)
5outputs = tf.keras.layers.Dense(5, activation="softmax")(x)
6
7model = tf.keras.Model(inputs, outputs)
8model.compile(
9    optimizer=tf.keras.optimizers.Adam(1e-3),
10    loss="sparse_categorical_crossentropy",
11    metrics=["accuracy"],
12)

That first training phase is not really "fine-tuning" yet. It is feature extraction, because only the new head learns.

Train the Head Before Unfreezing

Train the frozen-base model first:

history = model.fit(train_ds, epochs=5)

This lets the top classifier adapt to your classes while preserving the pretrained visual features in the base model. If you skip this step and unfreeze immediately, gradients from a random new classifier can damage useful pretrained representations.

Unfreeze Only the Top Part of Inception

After the head has stabilized, unfreeze a small upper portion of the base model and continue with a lower learning rate:

1base_model.trainable = True
2
3for layer in base_model.layers[:-50]:
4    layer.trainable = False
5
6model.compile(
7    optimizer=tf.keras.optimizers.Adam(1e-5),
8    loss="sparse_categorical_crossentropy",
9    metrics=["accuracy"],
10)
11
12history_finetune = model.fit(train_ds, epochs=5)

The exact cutoff depends on your dataset size and similarity to ImageNet. If the new task is close to the original one and you have enough data, you can fine-tune more layers. If data is limited, unfreezing too much often leads to overfitting.

Why the Lower Learning Rate Matters

Once pretrained layers are unfrozen, the optimizer can modify valuable weights that were learned on a huge dataset. A large learning rate can erase that knowledge quickly.

That is why the second phase usually uses a learning rate around 1e-5 or similarly small values. Fine-tuning is about small corrections, not large jumps.

Validate During Fine-Tuning

Always keep a validation set separate from the tuning loop. Fine-tuning can improve training accuracy very quickly while hurting generalization, so callbacks such as early stopping and model checkpointing are often worth adding once the basic workflow is working.

Common Pitfalls

• Forgetting to use preprocess_input for Inception-style inputs.
• Unfreezing the whole base model immediately.
• Keeping the original high learning rate during the fine-tuning phase.
• Using too little data and unfreezing too many layers, which encourages overfitting.
• Assuming fine-tuning is always necessary. Sometimes a frozen base plus a new head already performs well enough.

Summary

• Start with pretrained InceptionV3 and include_top=False.
• Freeze the base model first and train only the new classification head.
• Use the correct Inception preprocessing pipeline.
• Unfreeze only part of the upper network for the second phase.
• Lower the learning rate significantly during fine-tuning.