Introduction

Evaluating a TensorFlow Object Detection model is not just about running a script. You need the right pipeline config, a valid checkpoint directory, and an evaluation dataset that matches the label map and preprocessing settings used during training. The confusing part is that eval.py is a legacy workflow, while newer versions of the Object Detection API recommend model_main.py for eval-only runs.

Know Which Evaluation Entry Point You Are Using

In the TensorFlow Models repository, the old evaluation script lives under object_detection/legacy/eval.py. Its own source code marks it as deprecated and points users to object_detection/model_main.py instead. That distinction matters because many older tutorials still show eval.py, while newer setups are built around the estimator-based model_main.py interface.

A good rule is:

• if you are maintaining an older TensorFlow 1 style pipeline, legacy/eval.py may still match your project
• if you are using the newer Object Detection API layout, use model_main.py in eval-only mode

If you mix those workflows, flags and directory expectations often stop matching.

Make Sure the Required Inputs Exist

Before running evaluation, verify these pieces:

• a pipeline config with valid eval_config and eval_input_reader
• a checkpoint directory containing trained checkpoints
• a label map that matches the training job
• TFRecord evaluation data referenced by the pipeline config

The pipeline config is especially important because the Object Detection API reads evaluation behavior from it, including which metrics set to use and where evaluation data comes from.

A typical config fragment looks like this:

1eval_config: {
2  num_examples: 500
3  metrics_set: "coco_detection_metrics"
4}
5
6eval_input_reader: {
7  label_map_path: "/path/to/label_map.pbtxt"
8  tf_record_input_reader {
9    input_path: "/path/to/eval.record"
10  }
11}

If the evaluation record or label map path is wrong, the script may launch but never produce meaningful metrics.

Run the Legacy eval.py Job

For an older TensorFlow 1 style Object Detection API setup, the legacy command looks like this:

1python object_detection/legacy/eval.py \
2  --logtostderr \
3  --pipeline_config_path=training/faster_rcnn.config \
4  --checkpoint_dir=training \
5  --eval_dir=eval \
6  --run_once

What those flags do:

• 'pipeline_config_path points to the train-eval pipeline config'
• 'checkpoint_dir points to the directory containing model checkpoints'
• 'eval_dir is where evaluation summaries are written'
• 'run_once tells the evaluator to process one checkpoint pass and exit'

Without --run_once, the evaluator may keep watching for new checkpoints, which is useful during active training but confusing if you only want one batch of metrics.

Use model_main.py for the Newer Flow

In newer TensorFlow Models workflows, the recommended eval-only command uses model_main.py with checkpoint_dir set.

1python object_detection/model_main.py \
2  --pipeline_config_path=training/faster_rcnn.config \
3  --model_dir=eval_output \
4  --checkpoint_dir=training \
5  --run_once

This tells the binary to operate in evaluation-only mode. The checkpoint is read from training, and the evaluation summaries are written to eval_output.

If you omit checkpoint_dir, model_main.py switches into train-and-evaluate mode instead of eval-only mode.

Read the Results in TensorBoard

Both workflows typically write scalar summaries that you inspect with TensorBoard.

tensorboard --logdir=eval_output

Or for the legacy layout:

tensorboard --logdir=eval

Look for metrics such as mean Average Precision, localization quality, and class-level performance. A completed run that writes no useful metrics usually points back to config problems, label-map mismatches, or empty evaluation data.

Use run_once While Debugging

During setup, run_once is the safest mode because it gives you a deterministic one-shot result. Continuous evaluation is useful later when training jobs produce checkpoints over time, but it can hide setup mistakes because the process appears alive even when it is waiting or silently failing to find usable checkpoints.

Once the job works once, you can remove run_once if you want the evaluator to monitor the training directory continuously.

Common Pitfalls

The biggest mistake is following an old eval.py tutorial with a newer Object Detection API checkout and wondering why the workflow feels inconsistent. Another common issue is pointing checkpoint_dir at the wrong folder, such as an exported inference model instead of the training checkpoint directory. Label-map mismatches are also frequent and can ruin metrics even when the script itself runs. Finally, people often forget that model_main.py becomes eval-only only when checkpoint_dir is provided, otherwise it behaves as a training entry point.

Summary

• 'legacy/eval.py still exists for older workflows, but the TensorFlow Models repo points newer users to model_main.py.'
• Evaluation needs a valid pipeline config, evaluation dataset, label map, and checkpoint directory.
• Use --run_once while debugging so the evaluation job exits after one pass.
• Use TensorBoard to inspect the metrics written by the evaluation run.
• Keep the training-generation workflow and evaluation-generation workflow aligned so flags and configs match.