Introduction

Transfer learning with VGG weights is useful even when your custom CNN does not exactly match VGG architecture. Keras can load compatible layers and skip unmatched ones if you configure loading correctly. The key is to keep naming and tensor shapes predictable, then verify which layers were actually initialized.

Start with a Compatible Backbone

If possible, build your model from a VGG backbone and attach custom heads. This is the least error-prone path.

1import tensorflow as tf
2from tensorflow.keras import layers, models
3
4base = tf.keras.applications.VGG16(
5    include_top=False,
6    weights="imagenet",
7    input_shape=(224, 224, 3),
8)
9
10x = layers.GlobalAveragePooling2D()(base.output)
11x = layers.Dense(128, activation="relu")(x)
12out = layers.Dense(5, activation="softmax")(x)
13
14model = models.Model(inputs=base.input, outputs=out)

Here convolutional layers already contain pretrained weights, while new dense layers are randomly initialized.

Loading Partial Weights by Name

For custom architecture files, use name-based loading and skip mismatches.

# model is already built
model.load_weights("vgg_weights.h5", by_name=True, skip_mismatch=True)

This loads only layers with matching names and compatible tensor shapes. Non-matching layers are left at initial values.

Name matching requires stable layer names across source and destination models. If names differ, expected layers will be skipped.

Verify What Was Loaded

Do not assume transfer worked. Inspect layers and parameters before training.

for layer in model.layers:
    if layer.weights:
        print(layer.name, [w.shape for w in layer.get_weights()])

For a known transferred layer, inspect weight statistics:

1import numpy as np
2
3layer = model.get_layer("block1_conv1")
4kernel = layer.get_weights()[0]
5print("mean:", float(np.mean(kernel)), "std:", float(np.std(kernel)))

If values look random across expected pretrained layers, mapping likely failed.

Freezing and Fine-Tuning Strategy

A stable transfer workflow is staged training:

1. freeze backbone and train new head.
2. unfreeze top blocks.
3. continue with smaller learning rate.

1for layer in base.layers:
2    layer.trainable = False
3
4model.compile(
5    optimizer=tf.keras.optimizers.Adam(1e-3),
6    loss="sparse_categorical_crossentropy",
7    metrics=["accuracy"],
8)
9
10# ... train head first ...
11
12for layer in base.layers[-4:]:
13    layer.trainable = True
14
15model.compile(
16    optimizer=tf.keras.optimizers.Adam(1e-5),
17    loss="sparse_categorical_crossentropy",
18    metrics=["accuracy"],
19)

Staged unfreezing is usually more stable than training the whole network immediately.

Keep Preprocessing Consistent

Using VGG weights but incorrect preprocessing can erase transfer-learning benefits. Ensure input normalization matches the pretrained model.

from tensorflow.keras.applications.vgg16 import preprocess_input

x_batch = preprocess_input(x_batch)

If your input pipeline differs from VGG expectations, feature distributions shift and early layers become less useful.

Layer Naming Best Practices

When building custom models intended for partial weight loading:

• set explicit layer names for shared blocks.
• avoid accidental duplicate names.
• keep architecture changes localized to head or final blocks.

Example with named layer:

inputs = layers.Input(shape=(224, 224, 3), name="image")
x = layers.Conv2D(64, 3, padding="same", activation="relu", name="block1_conv1")(inputs)

Stable naming improves reproducibility across experiments.

Persist Transfer State for Reproducibility

After successful partial loading, save a checkpoint immediately so future runs do not depend on repeated mapping steps.

model.save("transfer_initialized.keras")

Keeping an explicit transfer-initialized artifact makes experiment comparison easier and helps debug whether performance differences came from data changes or weight-loading changes.

Common Pitfalls

• Expecting classifier head weights to load when output class count differs.
• Forgetting by_name=True and skip_mismatch=True in partial-load scenarios.
• Ignoring skipped-layer warnings and assuming full transfer occurred.
• Using mismatched preprocessing for pretrained backbones.
• Unfreezing too many layers too early and destabilizing training.

Summary

• You can reuse VGG weights even when your CNN is not an exact copy.
• Load by name and skip mismatches to transfer compatible layers safely.
• Verify loaded layers instead of assuming transfer succeeded.
• Use staged freeze and unfreeze fine-tuning for stable optimization.
• Keep layer names and preprocessing consistent across experiments.