Introduction

To continue training from a Keras checkpoint, you need to know what was saved. If the checkpoint stores only weights, rebuild the model and load the weights; if it stores the full model, load the model object and continue calling fit.

Save Weights Only and Resume Later

This is a common pattern when you want control over model construction in code.

1import numpy as np
2from tensorflow import keras
3
4
5def build_model():
6    model = keras.Sequential([
7        keras.layers.Input(shape=(4,)),
8        keras.layers.Dense(8, activation="relu"),
9        keras.layers.Dense(1, activation="sigmoid"),
10    ])
11    model.compile(
12        optimizer="adam",
13        loss="binary_crossentropy",
14        metrics=["accuracy"],
15    )
16    return model
17
18
19x = np.random.rand(200, 4).astype("float32")
20y = (x.sum(axis=1) > 2).astype("float32")
21
22model = build_model()
23
24checkpoint = keras.callbacks.ModelCheckpoint(
25    "checkpoint.weights.h5",
26    save_weights_only=True,
27    save_best_only=False,
28)
29
30model.fit(x, y, epochs=3, callbacks=[checkpoint], verbose=0)

Later, to continue training:

model = build_model()
model.load_weights("checkpoint.weights.h5")
model.fit(x, y, epochs=5, initial_epoch=3, verbose=0)

The important requirement is that build_model() must recreate the same architecture. If the layer shapes or names do not match, weight loading will fail.

Save the Full Model If You Want More State Back

If you save the full model instead of weights only, Keras can restore the architecture and optimizer state too.

1checkpoint = keras.callbacks.ModelCheckpoint(
2    "checkpoint.keras",
3    save_weights_only=False,
4    save_best_only=False,
5)

Resume like this:

1from tensorflow import keras
2
3model = keras.models.load_model("checkpoint.keras")
4model.fit(x, y, epochs=5, initial_epoch=3, verbose=0)

This is often the simplest way to continue training because you do not need to rebuild the model manually.

Understand initial_epoch

initial_epoch does not load anything by itself. It only tells fit which epoch number to treat as the starting point for logs and callbacks.

If you previously trained through epoch 3, then a continuation call such as this is appropriate:

1model.fit(
2    x,
3    y,
4    epochs=10,
5    initial_epoch=3,
6)

That means Keras continues from epoch index 3 up to 9. Without initial_epoch, the training still runs, but the epoch numbering and some callback behaviors may be misleading.

Best Checkpoint Versus Last Checkpoint

Be careful with save_best_only=True. That setting saves the best checkpoint according to the monitored metric, not necessarily the most recent training state.

1checkpoint = keras.callbacks.ModelCheckpoint(
2    "best.keras",
3    monitor="val_loss",
4    mode="min",
5    save_best_only=True,
6)

If you resume from this file, you are continuing from the best saved model so far, not automatically from the interrupted last epoch. That is fine when you want the best weights, but it is different from exact crash recovery.

For exact training recovery in long jobs, many teams also use callbacks designed for fault tolerance, not just model selection.

A Practical Rule

Choose one of these patterns:

• weights only if you prefer explicit model construction in code
• full model if you want the easiest continuation path

In either case, keep the architecture, preprocessing, and label encoding consistent between the original run and the resumed run. A checkpoint cannot fix a changed input pipeline.

Common Pitfalls

• Calling load_weights on a model whose architecture no longer matches the saved weights.
• Expecting initial_epoch to restore training state. It only affects fit bookkeeping.
• Using save_best_only=True and assuming the file is the latest epoch rather than the best monitored checkpoint.
• Forgetting to compile the rebuilt model before further training when using weights-only restoration.
• Changing preprocessing or class order between runs and blaming the checkpoint when resumed metrics look wrong.

Summary

• To continue training, first identify whether the checkpoint saved weights only or the full model.
• With weights-only checkpoints, rebuild the same model and call load_weights.
• With full-model checkpoints, use keras.models.load_model and continue fit.
• 'initial_epoch is for correct epoch numbering, not for loading the checkpoint itself.'
• 'save_best_only=True resumes from the best saved checkpoint, not necessarily the most recent one.'