Introduction

A typical TensorFlow input pipeline has two separate jobs: serialize raw examples into TFRecord files, then parse those records back into tensors and batch them efficiently. If you keep those steps distinct, the pipeline becomes easier to debug and scale.

Decide What One Example Looks Like

Before writing TFRecords, define the structure of a single example clearly. Suppose each sample has:

• a feature vector of floats
• one integer label

NumPy setup:

1import numpy as np
2
3x = np.array([
4    [0.1, 0.2, 0.3],
5    [1.0, 1.1, 1.2],
6    [2.0, 2.1, 2.2],
7], dtype=np.float32)
8
9y = np.array([0, 1, 0], dtype=np.int64)

Each row in x will become one TFRecord example.

Write NumPy Data to TFRecords

TensorFlow stores structured examples as serialized protocol buffers. A common pattern is to convert each NumPy row into a tf.train.Example.

1import tensorflow as tf
2import numpy as np
3
4
5def float_feature(values):
6    return tf.train.Feature(float_list=tf.train.FloatList(value=values))
7
8
9def int64_feature(value):
10    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
11
12
13def make_example(features, label):
14    example = tf.train.Example(
15        features=tf.train.Features(
16            feature={
17                "x": float_feature(features),
18                "y": int64_feature(label),
19            }
20        )
21    )
22    return example.SerializeToString()
23
24
25x = np.array([
26    [0.1, 0.2, 0.3],
27    [1.0, 1.1, 1.2],
28    [2.0, 2.1, 2.2],
29], dtype=np.float32)
30
31y = np.array([0, 1, 0], dtype=np.int64)
32
33with tf.io.TFRecordWriter("train.tfrecord") as writer:
34    for features, label in zip(x, y):
35        writer.write(make_example(features, label))

That creates a TFRecord file where each record is one serialized example.

Parse Records Back into Tensors

Reading TFRecords is the mirror image of writing them. You define a schema and parse each serialized record.

1import tensorflow as tf
2
3feature_spec = {
4    "x": tf.io.FixedLenFeature([3], tf.float32),
5    "y": tf.io.FixedLenFeature([], tf.int64),
6}
7
8
9def parse_example(serialized):
10    parsed = tf.io.parse_single_example(serialized, feature_spec)
11    return parsed["x"], parsed["y"]

The shape [3] matches the length of each feature vector that was written earlier.

Generate Batches with tf.data

Now build the input pipeline.

1import tensorflow as tf
2
3dataset = tf.data.TFRecordDataset(["train.tfrecord"])
4dataset = dataset.map(parse_example)
5dataset = dataset.shuffle(100)
6dataset = dataset.batch(2)
7dataset = dataset.prefetch(tf.data.AUTOTUNE)
8
9for batch_x, batch_y in dataset:
10    print(batch_x)
11    print(batch_y)

This is the standard modern TensorFlow approach. map, batch, and prefetch are the core pieces.

Variable-Length Data Needs a Different Schema

If your NumPy data does not have a fixed shape, you usually serialize the array as raw bytes or use variable-length features instead of FixedLenFeature.

For fixed-size tabular data, FixedLenFeature is the simplest and fastest choice. Do not make the schema more complicated than the data requires.

Separate Train, Validation, and Test Files

A practical project usually writes separate TFRecord files for different data splits.

# train.tfrecord
# valid.tfrecord
# test.tfrecord

That keeps your pipeline explicit and avoids mixing split logic into runtime data loading.

Why Use TFRecords at All

TFRecords are helpful when:

• the dataset is large
• training reads many examples repeatedly
• you want a stable binary format for TensorFlow pipelines
• preprocessing should happen once, not every epoch from raw text or CSV

For very small datasets, plain NumPy arrays or in-memory tensors may be simpler. TFRecords become more valuable as scale and pipeline complexity increase.

Common Pitfalls

The biggest mistake is writing one schema and parsing with another. If the field name, dtype, or shape changes between the write and read step, parsing will fail.

Another mistake is storing everything as strings or bytes when fixed-size numeric features would be simpler and faster.

Developers also sometimes forget that batching happens after parsing. The parser should describe one example, not the entire batch.

Finally, do not skip verification. After writing TFRecords, always read a few examples back and confirm the tensors look correct before starting model training.

Summary

• Write TFRecords by serializing one example at a time with tf.train.Example.
• Parse them with a matching schema using tf.io.parse_single_example.
• Build batches with tf.data.TFRecordDataset, map, batch, and prefetch.
• Keep the record schema aligned exactly between the write and read steps.
• Use TFRecords when dataset size and training repetition justify a binary input pipeline.