Introduction

In the TensorFlow Object Detection API, training resumes from checkpoint files, not from the SavedModel serving directory itself. The main job is to point pipeline.config at a compatible checkpoint prefix, set the checkpoint type correctly, and keep the model architecture aligned with the weights you are restoring.

Distinguish Checkpoints from SavedModel Exports

This topic gets confusing because an exported model package may contain several artifacts. For training, the Object Detection API restores from checkpoint files such as ckpt-0, ckpt-12, or whatever latest checkpoint exists. A SavedModel export is for inference and serving.

That means the important distinction is:

• use checkpoint files for training or fine-tuning
• use the exported SavedModel for inference

If an export directory includes a checkpoint/ folder, you still restore from the checkpoint prefix inside that folder, not from the saved_model directory.

Arrange the Workspace Clearly

A clean layout reduces mistakes when you edit pipeline.config.

1workspace/
2  annotations/
3    label_map.pbtxt
4    train.record
5    val.record
6  models/
7    my_ssd/
8      pipeline.config
9  exported/
10    my_ssd/
11      checkpoint/
12        ckpt-0.data-00000-of-00001
13        ckpt-0.index
14      saved_model/

Keeping your editable config under models/my_ssd/ and your export artifacts under a separate folder makes it much easier to reason about what is used for training versus serving.

Configure pipeline.config Correctly

Three settings are the most important during fine-tuning:

• 'num_classes'
• 'fine_tune_checkpoint'
• 'fine_tune_checkpoint_type'

A minimal example looks like this:

1model {
2  ssd {
3    num_classes: 3
4  }
5}
6
7train_config {
8  batch_size: 8
9  fine_tune_checkpoint: "workspace/exported/my_ssd/checkpoint/ckpt-0"
10  fine_tune_checkpoint_type: "detection"
11}
12
13train_input_reader {
14  label_map_path: "workspace/annotations/label_map.pbtxt"
15  tf_record_input_reader {
16    input_path: "workspace/annotations/train.record"
17  }
18}
19
20eval_input_reader {
21  label_map_path: "workspace/annotations/label_map.pbtxt"
22  tf_record_input_reader {
23    input_path: "workspace/annotations/val.record"
24  }
25}

The fine_tune_checkpoint value should point to the checkpoint prefix such as ckpt-0. Do not point it at the containing folder only, and do not point it at the .index file alone.

fine_tune_checkpoint_type should match the weights you are restoring. If you are starting from a pretrained detection model, "detection" is usually the right value.

Launch Training and Resume Safely

Training normally runs through model_main_tf2.py.

1python model_main_tf2.py \
2  --pipeline_config_path=workspace/models/my_ssd/pipeline.config \
3  --model_dir=workspace/models/my_ssd \
4  --alsologtostderr

Watch the startup logs. They should show that variables were restored from the checkpoint. If many variables are skipped unexpectedly, treat that as a configuration problem until you verify otherwise.

To resume your own interrupted run, rerun the trainer with the same model_dir.

python model_main_tf2.py \
  --pipeline_config_path=workspace/models/my_ssd/pipeline.config \
  --model_dir=workspace/models/my_ssd

In that case, the API restores from the latest checkpoint in model_dir, which is different from the original fine-tune checkpoint you started from.

Export Only After Training

Once training completes, export a serving artifact for inference.

1python exporter_main_v2.py \
2  --input_type image_tensor \
3  --pipeline_config_path workspace/models/my_ssd/pipeline.config \
4  --trained_checkpoint_dir workspace/models/my_ssd \
5  --output_directory workspace/exported/my_ssd_final

A quick sanity check helps confirm the export is usable:

1import tensorflow as tf
2
3saved_model = tf.saved_model.load("workspace/exported/my_ssd_final/saved_model")
4infer = saved_model.signatures["serving_default"]
5print(infer.structured_outputs.keys())

That validates the serving artifact without confusing it with training state.

When an Exported Checkpoint Is Usable

If someone says they want to train from an exported model checkpoint, the precise answer is: yes, if they mean the actual checkpoint files packaged alongside the export and those checkpoints are compatible with the target architecture. No, if they mean the SavedModel serving directory alone.

That wording matters because many failed training runs are caused by pointing the fine-tune path at the wrong artifact rather than by a deeper model issue.

Common Pitfalls

The biggest mistake is confusing the SavedModel export with the checkpoint prefix. They serve different purposes.

Another issue is forgetting to update num_classes, label map paths, or input record paths when adapting a checkpoint to a new dataset. The model may restore successfully but still be configured for the wrong task.

Be careful with checkpoint type as well. A mismatch between model architecture and restore type can silently skip variables and degrade training quality.

Finally, if the label semantics changed significantly, do not try to continue a run blindly. Start a fresh fine-tuning experiment with a clear configuration instead.

Summary

• Fine-tuning uses checkpoint files, not the SavedModel directory.
• Point fine_tune_checkpoint at a checkpoint prefix such as ckpt-0.
• Set fine_tune_checkpoint_type and num_classes correctly in pipeline.config.
• Resume interrupted training from model_dir, not from a new export path.
• Export a SavedModel only after training is complete.