Introduction

Variable-length features are common in TensorFlow pipelines. A text example may have a different token count than the next one, and one recommendation event may contain three item IDs while another contains twenty. The usual way to store this in TFRecords is with list features, then parse them with tf.io.VarLenFeature and batch them with padding or ragged tensors.

Write and Parse Variable-Length Lists

The TFRecord format stores each example independently, so variable-length lists fit naturally. Here is a complete example that writes two examples with different list sizes and reads them back through the Dataset API.

1import tensorflow as tf
2
3
4def int_feature(values):
5    return tf.train.Feature(int64_list=tf.train.Int64List(value=values))
6
7
8examples = [
9    tf.train.Example(features=tf.train.Features(feature={
10        "item_ids": int_feature([10, 11, 12]),
11        "label": int_feature([1]),
12    })),
13    tf.train.Example(features=tf.train.Features(feature={
14        "item_ids": int_feature([20, 21]),
15        "label": int_feature([0]),
16    })),
17]
18
19path = "variable_lists.tfrecord"
20with tf.io.TFRecordWriter(path) as writer:
21    for example in examples:
22        writer.write(example.SerializeToString())
23
24
25feature_spec = {
26    "item_ids": tf.io.VarLenFeature(tf.int64),
27    "label": tf.io.FixedLenFeature([], tf.int64),
28}
29
30
31def parse_record(serialized):
32    parsed = tf.io.parse_single_example(serialized, feature_spec)
33    item_ids = tf.sparse.to_dense(parsed["item_ids"])
34    label = parsed["label"]
35    return item_ids, label
36
37
38dataset = tf.data.TFRecordDataset(path).map(parse_record)
39for item_ids, label in dataset:
40    print(item_ids.numpy(), label.numpy())

The key detail is VarLenFeature. That tells TensorFlow the list length may differ between records. The parser returns a SparseTensor, because each example may have a different number of values.

Dense, Sparse, or Ragged

After parsing, you have three main options:

1. Keep the feature sparse if the downstream model accepts sparse inputs.
2. Convert it to a dense tensor with tf.sparse.to_dense().
3. Convert it to a ragged tensor if variable-length structure should be preserved more explicitly.

Dense conversion is simple, but batching dense tensors requires padding. That is usually fine for sequence models.

1dataset = tf.data.TFRecordDataset(path).map(parse_record)
2dataset = dataset.padded_batch(
3    2,
4    padded_shapes=([None], []),
5    padding_values=(tf.constant(0, tf.int64), tf.constant(0, tf.int64)),
6)
7
8for item_ids, labels in dataset:
9    print(item_ids.numpy())
10    print(labels.numpy())

The padded result for the sample data would be a rectangular batch where shorter lists are filled with zeros. That makes it easy to feed the data into embedding layers, recurrent models, or custom training loops.

If zeros are meaningful values in your domain, choose another padding token and make sure the model knows to mask it or ignore it.

When to Use SequenceExample

Some pipelines use tf.train.SequenceExample for sequential data. That can be useful when you have context features plus per-step feature lists. But if your record simply contains one variable-length list field, a normal Example plus VarLenFeature is often easier to read and maintain.

The important design decision is not the container type alone. It is whether the downstream parser and model can preserve the meaning of sequence length, order, and padding.

For performance, keep the input pipeline streaming and avoid loading the whole file at once:

1dataset = (
2    tf.data.TFRecordDataset(path)
3    .map(parse_record, num_parallel_calls=tf.data.AUTOTUNE)
4    .shuffle(1000)
5    .padded_batch(32, padded_shapes=([None], []))
6    .prefetch(tf.data.AUTOTUNE)
7)

That pattern scales much better for real training jobs than manually reading records into Python lists.

Common Pitfalls

The most common mistake is using tf.io.FixedLenFeature for data that is not actually fixed length. That either fails during parsing or forces you to pad at write time, which makes the format less flexible.

Another mistake is forgetting that VarLenFeature produces a sparse tensor. Code that expects a dense tensor will break until you explicitly convert it or adapt the model.

Batching is another frequent source of confusion. Variable-length examples cannot form a normal dense batch without padding. Use padded_batch() or keep the data ragged if the model supports it.

Finally, be deliberate about padding values. A bad padding token can leak into training and hurt accuracy if the model treats padded positions like real data.

Summary

• Store variable-length lists in TFRecords as list features inside each Example.
• Parse them with tf.io.VarLenFeature, not FixedLenFeature.
• 'VarLenFeature returns a SparseTensor, which you can keep sparse or convert to dense.'
• Use padded_batch() when batching sequences of different lengths.
• Choose padding values and masking behavior carefully so the model does not learn from padding.