Introduction

tf.data is the standard way to build efficient input pipelines in TensorFlow. It helps you scale from tiny local experiments to large training jobs without rewriting your data logic. This guide shows how to build training and validation datasets correctly and wire them into model.fit.

Build a Dataset from In-Memory Arrays

A good starting point is from_tensor_slices, then split into train and validation sets using deterministic slicing.

1import tensorflow as tf
2import numpy as np
3
4# Reproducible toy dataset
5rng = np.random.default_rng(7)
6features = rng.normal(size=(1000, 10)).astype("float32")
7labels = (features[:, 0] + features[:, 1] > 0).astype("int32")
8
9full_ds = tf.data.Dataset.from_tensor_slices((features, labels))
10full_ds = full_ds.shuffle(buffer_size=1000, seed=7, reshuffle_each_iteration=False)
11
12val_size = 200
13val_ds = full_ds.take(val_size)
14train_ds = full_ds.skip(val_size)
15
16train_ds = train_ds.batch(32).prefetch(tf.data.AUTOTUNE)
17val_ds = val_ds.batch(32).prefetch(tf.data.AUTOTUNE)

Using reshuffle_each_iteration=False preserves a stable split across runs. That is useful when comparing experiments.

Add Preprocessing with map

Most real projects need preprocessing. Keep it in your dataset pipeline so training and evaluation stay consistent.

1def preprocess(x, y):
2    x = tf.clip_by_value(x, -3.0, 3.0)
3    x = (x - tf.reduce_mean(x)) / (tf.math.reduce_std(x) + 1e-6)
4    return x, y
5
6train_ds = train_ds.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)
7val_ds = val_ds.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)

For image data, this is where you decode files, resize images, and optionally apply augmentations only on training data.

Train with Separate Training and Validation Datasets

Once datasets are prepared, use validation_data in model.fit.

1model = tf.keras.Sequential([
2    tf.keras.layers.Input(shape=(10,)),
3    tf.keras.layers.Dense(32, activation="relu"),
4    tf.keras.layers.Dense(1, activation="sigmoid")
5])
6
7model.compile(
8    optimizer=tf.keras.optimizers.Adam(1e-3),
9    loss=tf.keras.losses.BinaryCrossentropy(),
10    metrics=[tf.keras.metrics.BinaryAccuracy(name="acc")]
11)
12
13history = model.fit(
14    train_ds,
15    validation_data=val_ds,
16    epochs=8,
17    verbose=2
18)

Validation metrics now represent generalization on held-out data, not performance on the training subset.

Pipeline Performance Techniques

tf.data performance usually depends on order and buffering. A common pattern is:

1. shuffle
2. map
3. batch
4. prefetch

Add cache if data fits memory or if preprocessing is expensive and deterministic.

1train_ds = (
2    tf.data.Dataset.from_tensor_slices((features, labels))
3    .shuffle(1000, seed=7)
4    .map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)
5    .batch(32)
6    .cache()
7    .prefetch(tf.data.AUTOTUNE)
8)

If your dataset comes from files, consider interleave for parallel reads. For distributed training, ensure each worker sees a suitable shard.

Working with Validation Split from Keras Utilities

For image folders, tf.keras.utils.image_dataset_from_directory supports validation split directly.

1train_images = tf.keras.utils.image_dataset_from_directory(
2    "data/flowers",
3    validation_split=0.2,
4    subset="training",
5    seed=7,
6    image_size=(224, 224),
7    batch_size=32
8)
9
10val_images = tf.keras.utils.image_dataset_from_directory(
11    "data/flowers",
12    validation_split=0.2,
13    subset="validation",
14    seed=7,
15    image_size=(224, 224),
16    batch_size=32
17)
18
19train_images = train_images.prefetch(tf.data.AUTOTUNE)
20val_images = val_images.prefetch(tf.data.AUTOTUNE)

The shared seed and matching split settings are essential to avoid overlap.

Validate Dataset Cardinality and Epoch Behavior

Input bugs often come from unknown dataset size or accidental infinite repetition. For bounded datasets, inspect cardinality so you can reason about steps_per_epoch and validation coverage.

1train_count = tf.data.experimental.cardinality(train_ds).numpy()
2val_count = tf.data.experimental.cardinality(val_ds).numpy()
3print(\"Train batches:\", train_count)
4print(\"Validation batches:\", val_count)

If you call repeat() on training data, either leave steps_per_epoch explicit or remove repeat for small local experiments. Otherwise, training may never end as expected. Keep validation finite so each epoch reports comparable metrics.

Common Pitfalls

• Applying random augmentations to validation data. Validation should reflect real inference conditions.
• Reshuffling differently while splitting. If split logic is inconsistent, train and validation samples can overlap.
• Forgetting prefetch, causing the model to wait on input pipeline work.
• Using tiny shuffle buffers. Small buffers reduce randomness and may bias batches.
• Caching before random transforms when you intended fresh augmentation each epoch.

Summary

• Use take and skip or directory split utilities to build explicit train and validation datasets.
• Keep preprocessing in tf.data pipelines for consistency and reproducibility.
• Feed validation data through validation_data in model.fit.
• Optimize pipelines with map, batch, cache, and prefetch in sensible order.
• Protect split integrity so your validation metrics remain trustworthy.