Introduction

Evaluating on validation data more than once during training is a normal requirement for model selection, early stopping, and debugging. The key is to build a validation input pipeline that can be consumed repeatedly, because a one-pass queue or exhausted iterator will not magically reset itself.

The Core Idea

Training and validation serve different purposes, so they should usually have separate input pipelines. During training, the pipeline can shuffle and repeat forever. During validation, the pipeline should be deterministic and restartable so the model can scan the same examples at regular intervals.

In modern TensorFlow, the cleanest path is to let Keras run validation on a schedule:

1import tensorflow as tf
2
3train_ds = (
4    tf.data.Dataset.from_tensor_slices((x_train, y_train))
5    .shuffle(1000)
6    .batch(32)
7    .repeat()
8)
9
10val_ds = (
11    tf.data.Dataset.from_tensor_slices((x_val, y_val))
12    .batch(32)
13)
14
15model = tf.keras.Sequential([
16    tf.keras.layers.Dense(64, activation="relu"),
17    tf.keras.layers.Dense(1, activation="sigmoid"),
18])
19
20model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
21
22model.fit(
23    train_ds,
24    steps_per_epoch=100,
25    epochs=10,
26    validation_data=val_ds,
27    validation_freq=1,
28)

Keras recreates the validation pass each time, so you do not need to manage queue state manually.

Manual Evaluation Inside A Custom Training Loop

If you use tf.GradientTape, keep validation in a separate loop that iterates over a dataset from the beginning each time:

1import tensorflow as tf
2
3loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()
4optimizer = tf.keras.optimizers.Adam()
5train_acc = tf.keras.metrics.SparseCategoricalAccuracy()
6val_acc = tf.keras.metrics.SparseCategoricalAccuracy()
7
8
9for epoch in range(5):
10    train_acc.reset_state()
11    for xb, yb in train_ds.take(100):
12        with tf.GradientTape() as tape:
13            logits = model(xb, training=True)
14            loss = loss_fn(yb, logits)
15        grads = tape.gradient(loss, model.trainable_variables)
16        optimizer.apply_gradients(zip(grads, model.trainable_variables))
17        train_acc.update_state(yb, logits)
18
19    val_acc.reset_state()
20    for xb, yb in val_ds:
21        logits = model(xb, training=False)
22        val_acc.update_state(yb, logits)
23
24    print(
25        f"epoch={epoch} "
26        f"train_acc={train_acc.result().numpy():.4f} "
27        f"val_acc={val_acc.result().numpy():.4f}"
28    )

The important detail is that val_ds is not an already-consumed queue object in the TensorFlow 1 sense. Each for loop creates a fresh iteration over the dataset.

If You Still Use TensorFlow 1 Queue-Style Code

Older TensorFlow code often used queue runners or initializable iterators. In that model, validation can fail after the first pass because the queue becomes empty. The fix is to reinitialize the validation iterator before each evaluation:

1import tensorflow as tf
2
3dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val)).batch(32)
4iterator = tf.compat.v1.data.make_initializable_iterator(dataset)
5features, labels = iterator.get_next()
6
7logits = model(features)
8accuracy = tf.reduce_mean(
9    tf.cast(tf.equal(tf.argmax(logits, axis=1), tf.cast(labels, tf.int64)), tf.float32)
10)
11
12with tf.compat.v1.Session() as sess:
13    sess.run(tf.compat.v1.global_variables_initializer())
14
15    for step in range(1000):
16        sess.run(train_op)
17
18        if step % 100 == 0:
19            sess.run(iterator.initializer)
20            scores = []
21            while True:
22                try:
23                    scores.append(sess.run(accuracy))
24                except tf.errors.OutOfRangeError:
25                    break
26            print("validation accuracy:", sum(scores) / len(scores))

The reset happens with sess.run(iterator.initializer). Without that line, the validation pipeline is exhausted after the first full evaluation.

Common Pitfalls

The most common problem is sharing one repeating training pipeline for validation. Validation should not shuffle unpredictably or repeat forever unless you also provide a strict validation_steps value and know exactly what subset you are measuring.

Another issue is forgetting to reset metrics. If you call update_state repeatedly across epochs without reset_state, the numbers blend old and new evaluations together and become misleading.

In TensorFlow 1 code, queue runners and iterators need explicit lifecycle management. If your validation queue stops producing values, it is usually not a modeling bug but an input-pipeline state bug.

Also watch for dropout and batch normalization. Validation should run with training=False, otherwise the reported metrics reflect training behavior rather than inference behavior.

Summary

• Validation can run many times during training as long as its input pipeline is restartable.
• With Keras fit, validation_data and validation_freq handle repeated evaluation cleanly.
• In custom loops, iterate over a separate validation dataset from the beginning each time.
• In TensorFlow 1 style code, reinitialize the iterator or queue before every validation pass.
• Reset metrics and use training=False so validation results stay meaningful.