Introduction

A Python generator and a TensorFlow tensor solve different problems. A generator yields items lazily over time, while a tensor is an in-memory value with a concrete shape and dtype. That means there is no universal one-step conversion from "generator" to "tensor" without deciding whether you want to materialize all values or keep the data streaming.

For small datasets, you can exhaust the generator into a list and call tf.convert_to_tensor. For training pipelines or large inputs, the right bridge is usually tf.data.Dataset.from_generator, not a single tensor.

Materialize Small Generators Into a Tensor

If the generator is small enough to fit comfortably in memory, convert it to a list first:

1import tensorflow as tf
2
3
4def gen_rows(n):
5    for i in range(n):
6        yield [float(i), float(i + 1), float(i + 2)]
7
8
9rows = list(gen_rows(5))
10tensor = tf.convert_to_tensor(rows, dtype=tf.float32)
11
12print(tensor)
13print(tensor.shape)

This works because by the time tf.convert_to_tensor runs, the data is no longer a generator. It is already a fully materialized Python collection.

That is fine for toy examples, small config tables, and debugging. It is the wrong approach for large or unbounded streams.

Use tf.data.Dataset.from_generator for Streaming

If the reason you used a generator was memory efficiency or pipeline flexibility, keep it lazy and wrap it as a dataset:

1import tensorflow as tf
2
3
4def sample_generator():
5    for i in range(10):
6        features = [float(i), float(i + 1)]
7        label = i % 2
8        yield features, label
9
10
11signature = (
12    tf.TensorSpec(shape=(2,), dtype=tf.float32),
13    tf.TensorSpec(shape=(), dtype=tf.int32),
14)
15
16dataset = tf.data.Dataset.from_generator(
17    sample_generator,
18    output_signature=signature,
19)
20
21dataset = dataset.batch(4).prefetch(tf.data.AUTOTUNE)
22
23for batch_x, batch_y in dataset.take(1):
24    print(batch_x.shape, batch_y.shape)

This is the normal pattern for input pipelines that feed model.fit, custom training loops, or batched preprocessing.

output_signature Is the Critical Part

Most generator-related TensorFlow errors come from an incorrect output_signature. The signature must describe exactly what each yielded element looks like:

• the number of outputs
• the dtype of each output
• the shape or partial shape of each output

If the generator yields floats but the signature says integers, or yields vectors of the wrong length, the pipeline will fail at runtime.

That is why checking one sample early is useful:

g = sample_generator()
x, y = next(g)
print(x, y)

Inspecting one item often reveals shape or dtype mismatches before a longer training run does.

Variable-Length Data Needs Padding or Ragged Handling

If the generator yields sequences of different lengths, you usually cannot batch them directly into dense tensors. One common fix is padded_batch:

1import tensorflow as tf
2
3
4def sequence_generator():
5    yield [1, 2]
6    yield [3, 4, 5]
7    yield [6]
8
9
10dataset = tf.data.Dataset.from_generator(
11    sequence_generator,
12    output_signature=tf.TensorSpec(shape=(None,), dtype=tf.int32),
13)
14
15dataset = dataset.padded_batch(2, padded_shapes=[None])
16
17for batch in dataset.take(1):
18    print(batch)

Without padding or another variable-length strategy, TensorFlow cannot form regular batches from uneven records.

Keep Heavy Processing Out of the Python Generator

Generators are flexible, but Python-side preprocessing can become the throughput bottleneck. A common pattern is:

1. use the generator for basic record retrieval
2. move heavier transforms into dataset operations such as map
3. batch and prefetch to overlap input and compute

For example:

1dataset = dataset.map(
2    lambda x, y: (x / 255.0, y),
3    num_parallel_calls=tf.data.AUTOTUNE,
4).prefetch(tf.data.AUTOTUNE)

That usually scales better than doing all preprocessing in plain Python before TensorFlow ever sees the data.

Common Pitfalls

The biggest mistake is trying to convert a huge generator straight into a tensor by calling list(generator) on it. That defeats the whole point of using a generator and can exhaust memory.

Another common issue is getting output_signature wrong. Shape and dtype mismatches are the main source of runtime errors when using Dataset.from_generator.

Developers also forget that generators are consumed. Once you iterate one, you may need to create a fresh generator instance rather than reusing the exhausted one.

Finally, variable-length samples need a deliberate batching strategy. Dense batching assumes consistent shapes unless you use padding or another representation.

Summary

• A generator is lazy, while a tensor is materialized, so conversion depends on whether you want in-memory data or streaming input.
• Use tf.convert_to_tensor(list(generator)) only for small datasets.
• Use tf.data.Dataset.from_generator for scalable streaming pipelines.
• Define output_signature carefully so TensorFlow knows the yielded shapes and dtypes.
• Use padding or another variable-length strategy before batching uneven samples.