Introduction

In TensorFlow Estimator, training and evaluation are separate phases with different input pipelines and different runtime behavior. You usually control when evaluation happens by combining TrainSpec, EvalSpec, and train_and_evaluate, or by writing your own loop that calls train() and evaluate() at the exact intervals you want.

Separate the Data Pipelines First

Estimator expects training and evaluation to use different input functions because those phases should behave differently. Training data is commonly shuffled and repeated. Evaluation data should usually be deterministic and finite.

1import tensorflow as tf
2
3
4def train_input_fn():
5    features = {"x": tf.constant([[1.0], [2.0], [3.0], [4.0]])}
6    labels = tf.constant([0, 0, 1, 1])
7    ds = tf.data.Dataset.from_tensor_slices((features, labels))
8    return ds.shuffle(4).repeat().batch(2)
9
10
11def eval_input_fn():
12    features = {"x": tf.constant([[1.5], [2.5]])}
13    labels = tf.constant([0, 1])
14    ds = tf.data.Dataset.from_tensor_slices((features, labels))
15    return ds.batch(2)

That separation is the first and most important control. If you reuse the same endlessly repeating training dataset for evaluation, your evaluation phase stops meaning what you think it means.

Use TrainSpec and EvalSpec for Managed Scheduling

The standard Estimator orchestration API is tf.estimator.train_and_evaluate. It takes a training spec and an evaluation spec that describe how long to train and how often evaluation is allowed to run.

1import tensorflow as tf
2
3feature_columns = [tf.feature_column.numeric_column("x")]
4
5estimator = tf.estimator.DNNClassifier(
6    feature_columns=feature_columns,
7    hidden_units=[8, 8],
8    n_classes=2,
9)
10
11train_spec = tf.estimator.TrainSpec(
12    input_fn=train_input_fn,
13    max_steps=200
14)
15
16eval_spec = tf.estimator.EvalSpec(
17    input_fn=eval_input_fn,
18    steps=None,
19    start_delay_secs=0,
20    throttle_secs=30
21)
22
23tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

Important controls here are:

• 'max_steps, which caps training progress'
• 'start_delay_secs, which delays the first evaluation'
• 'throttle_secs, which sets the minimum time between evaluations'

That means evaluation is not triggered on every batch. It runs according to the orchestration policy, checkpoint creation, and the throttling rules you specify.

Use mode Inside model_fn to Change Behavior

If you are writing a custom Estimator, the runtime tells your model_fn whether it is training, evaluating, or predicting through the mode argument.

1def model_fn(features, labels, mode):
2    inputs = features["x"]
3    logits = tf.keras.layers.Dense(2)(inputs)
4    classes = tf.argmax(logits, axis=1)
5
6    if mode == tf.estimator.ModeKeys.PREDICT:
7        return tf.estimator.EstimatorSpec(
8            mode=mode,
9            predictions={"class_ids": classes}
10        )
11
12    loss = tf.reduce_mean(
13        tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits)
14    )
15
16    if mode == tf.estimator.ModeKeys.TRAIN:
17        optimizer = tf.compat.v1.train.AdamOptimizer(0.01)
18        train_op = optimizer.minimize(
19            loss,
20            global_step=tf.compat.v1.train.get_global_step()
21        )
22        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
23
24    metrics = {
25        "accuracy": tf.compat.v1.metrics.accuracy(labels=labels, predictions=classes)
26    }
27    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=metrics)

This is how the same model definition can behave differently across phases. Training-only logic, such as optimizer steps, belongs under TRAIN. Evaluation metrics belong under EVAL.

Use a Manual Loop for Exact Timing

If you need evaluation after a precise number of training steps rather than a time-based throttle, a manual loop is often clearer than train_and_evaluate.

1for cycle in range(5):
2    estimator.train(input_fn=train_input_fn, steps=100)
3    metrics = estimator.evaluate(input_fn=eval_input_fn)
4    print(f"cycle={cycle}", metrics)

This pattern is useful when you want predictable step-based cadence, custom early stopping, or reporting that does not fit the default orchestration flow.

Estimator is a legacy TensorFlow API in many modern codebases, so if you are starting from scratch you may prefer Keras. But when you are working inside Estimator, this split between managed orchestration and explicit looping is the main way to control train-versus-eval timing.

Common Pitfalls

The most common mistake is using the same input function for training and evaluation. A training dataset often repeats forever and shuffles data, which makes evaluation unstable or meaningless.

Another issue is misunderstanding throttle_secs. It controls minimum wall-clock time between evaluation runs in the managed workflow, not the number of batches between evaluations.

Developers also sometimes forget to branch on mode inside model_fn. If training-only behavior leaks into evaluation, metrics no longer represent true inference-time behavior.

Finally, if you need exact evaluation intervals, do not force that requirement into train_and_evaluate. A small explicit loop is often more honest and easier to maintain.

Summary

• Use separate input functions for training and evaluation.
• 'TrainSpec and EvalSpec control managed train-and-evaluate scheduling.'
• 'start_delay_secs and throttle_secs affect when evaluation is allowed to run.'
• Branch on mode inside model_fn so training and evaluation behave correctly.
• If you need exact step-based control, call train() and evaluate() yourself in a loop.