Introduction

When you want Inception v3 to recognize new categories, you usually do not retrain the whole network from scratch. The standard approach is transfer learning: keep the pretrained convolutional base, replace the classifier head, train the new head on your dataset, and then fine-tune part of the base model if needed.

Start with a Dataset Organized by Class

The simplest workflow uses one directory per class. For example, a dataset could contain data/flowers/daisy, data/flowers/rose, and data/flowers/tulip. TensorFlow can read this structure directly and infer the class labels.

Inception v3 expects images resized to 299 x 299, so make sure the input pipeline uses that shape and applies the correct preprocessing function.

1import tensorflow as tf
2from tensorflow import keras
3
4image_size = (299, 299)
5batch_size = 32
6
7train_ds = keras.utils.image_dataset_from_directory(
8    "data/flowers",
9    validation_split=0.2,
10    subset="training",
11    seed=42,
12    image_size=image_size,
13    batch_size=batch_size,
14)
15
16val_ds = keras.utils.image_dataset_from_directory(
17    "data/flowers",
18    validation_split=0.2,
19    subset="validation",
20    seed=42,
21    image_size=image_size,
22    batch_size=batch_size,
23)
24
25num_classes = len(train_ds.class_names)
26preprocess = keras.applications.inception_v3.preprocess_input
27
28train_ds = train_ds.map(lambda x, y: (preprocess(tf.cast(x, tf.float32)), y))
29val_ds = val_ds.map(lambda x, y: (preprocess(tf.cast(x, tf.float32)), y))
30
31train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
32val_ds = val_ds.prefetch(tf.data.AUTOTUNE)

Replace the Original Classifier Head

The original Inception v3 top layers were trained for ImageNet classes. To add your own categories, load the pretrained base with include_top=False, then attach a new output layer sized for your dataset.

1base_model = keras.applications.InceptionV3(
2    include_top=False,
3    weights="imagenet",
4    input_shape=(299, 299, 3),
5)
6base_model.trainable = False
7
8inputs = keras.Input(shape=(299, 299, 3))
9x = base_model(inputs, training=False)
10x = keras.layers.GlobalAveragePooling2D()(x)
11x = keras.layers.Dropout(0.2)(x)
12outputs = keras.layers.Dense(num_classes, activation="softmax")(x)
13
14model = keras.Model(inputs, outputs)
15
16model.compile(
17    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
18    loss="sparse_categorical_crossentropy",
19    metrics=["accuracy"],
20)
21
22model.fit(train_ds, validation_data=val_ds, epochs=5)

At this stage, only the new classification head trains. This lets the model learn the mapping from the pretrained visual features to your new categories without immediately disturbing the useful weights learned from ImageNet.

Fine-Tune After the Head Has Learned

If validation accuracy plateaus and you have enough data, unfreeze part of the base model and continue training with a much smaller learning rate.

1base_model.trainable = True
2
3for layer in base_model.layers[:-30]:
4    layer.trainable = False
5
6model.compile(
7    optimizer=keras.optimizers.Adam(learning_rate=1e-5),
8    loss="sparse_categorical_crossentropy",
9    metrics=["accuracy"],
10)
11
12model.fit(train_ds, validation_data=val_ds, epochs=3)

The small learning rate is critical. Fine-tuning changes pretrained weights that already contain strong general-purpose image features. Updating them too aggressively often destroys useful representations and makes the model perform worse.

How This Relates to TensorFlow 1

Older TensorFlow 1 examples often retrained only the final layer using graph-based code or the original TensorFlow image retraining scripts. The training strategy was the same: keep most pretrained weights, add a new classifier, and optimize that classifier on your classes. The modern tf.keras API shown here is easier to maintain and is the recommended approach for current projects.

Common Pitfalls

The first common problem is using the wrong image size or skipping preprocess_input. Inception v3 was trained with a specific input pipeline, so mismatched preprocessing can noticeably reduce accuracy.

Another issue is unfreezing the entire model too early. If you start fine-tuning before the new head has learned anything useful, the base network may drift and training becomes unstable.

Small datasets can also create misleadingly high training accuracy with poor validation results. Data augmentation, class balancing, and early stopping often matter more than another extra epoch.

Finally, verify that the class folders are correct. A mislabeled directory name can silently train the model on the wrong labels and waste hours of debugging.

Summary

• Use transfer learning instead of training Inception v3 from scratch for a small custom dataset.
• Resize images to 299 x 299 and apply Inception v3 preprocessing.
• Load the pretrained base with include_top=False, then add a new Dense layer for your categories.
• Train the new classifier head first, then fine-tune upper layers with a very small learning rate.
• Check labels, preprocessing, and dataset quality before blaming the model architecture.