Introduction

If model.fit_generator is much slower than model.fit, the bottleneck is usually not the optimizer or the network itself. The slowdown almost always comes from how batches are produced, transferred, decoded, or augmented before the model sees them.

Why fit_generator can be slower

Historically, model.fit worked with in-memory arrays and model.fit_generator worked with Python generators. In modern Keras, fit_generator is effectively folded into model.fit, but the performance difference still matters because Python-driven input pipelines behave very differently from preloaded arrays.

With in-memory arrays, the model can consume batches with very little overhead:

1history = model.fit(
2    x_train,
3    y_train,
4    batch_size=64,
5    epochs=5,
6    validation_data=(x_val, y_val),
7)

If the data is already in RAM as NumPy arrays, there is no per-batch file read, image decode, Python loop, or user-defined augmentation function in the hot path.

By contrast, a generator may do some or all of the following every batch:

• read files from disk
• decode images
• resize or normalize data
• allocate new arrays
• run Python code under the GIL
• wait for the next batch before the GPU can continue

That is why training can become several times slower even with the same model.

A slow generator pattern

The following style is common and often underperforms:

1def batch_generator(file_paths, labels, batch_size):
2    while True:
3        for start in range(0, len(file_paths), batch_size):
4            batch_paths = file_paths[start:start + batch_size]
5            batch_labels = labels[start:start + batch_size]
6
7            images = []
8            for path in batch_paths:
9                img = load_and_preprocess_image(path)
10                images.append(img)
11
12            yield np.array(images), np.array(batch_labels)

This works, but it does all preprocessing synchronously in Python. If the model finishes a training step before the next batch is ready, the accelerator sits idle.

Better options: Sequence and tf.data

If you need Python-side loading, keras.utils.Sequence is safer than a bare generator because it is indexable and works better with multiprocessing.

1import math
2import numpy as np
3from tensorflow import keras
4
5class MySequence(keras.utils.Sequence):
6    def __init__(self, x, y, batch_size):
7        self.x = x
8        self.y = y
9        self.batch_size = batch_size
10
11    def __len__(self):
12        return math.ceil(len(self.x) / self.batch_size)
13
14    def __getitem__(self, index):
15        start = index * self.batch_size
16        end = start + self.batch_size
17        return self.x[start:end], self.y[start:end]
18
19sequence = MySequence(x_train, y_train, batch_size=64)
20history = model.fit(sequence, epochs=5)

Even better, use tf.data when possible:

1import tensorflow as tf
2
3dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
4dataset = dataset.shuffle(10000).batch(64).prefetch(tf.data.AUTOTUNE)
5
6history = model.fit(dataset, epochs=5)

prefetch lets the input pipeline prepare future batches while the current batch is training, which is one of the easiest wins for throughput.

When the generator overhead dominates

The gap is especially noticeable when the model is small or the hardware is fast. If each training step takes only a few milliseconds, input overhead becomes a large percentage of total step time.

For example, a compact CNN on cached images may train quickly, so repeated Python-level augmentation can dominate runtime. On the other hand, a very large transformer may hide some generator cost because the compute phase is much longer.

That is why the right question is not "Why is fit_generator always slower?" but "Is my input pipeline slower than my model?"

Practical ways to speed it up

Several fixes usually help:

• move from a plain generator to Sequence or tf.data
• precompute expensive preprocessing steps
• store data in a format optimized for sequential reads
• use parallel mapping and prefetching
• avoid Python loops in per-sample transforms when vectorization is possible

A typical image pipeline with tf.data might look like this:

1def preprocess(path, label):
2    image = tf.io.read_file(path)
3    image = tf.image.decode_jpeg(image, channels=3)
4    image = tf.image.resize(image, (224, 224))
5    image = tf.cast(image, tf.float32) / 255.0
6    return image, label
7
8dataset = tf.data.Dataset.from_tensor_slices((train_paths, train_labels))
9dataset = dataset.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)
10dataset = dataset.batch(64).prefetch(tf.data.AUTOTUNE)

This usually outperforms a Python generator because TensorFlow can pipeline and parallelize the work more effectively.

Common Pitfalls

• Comparing in-memory arrays against a generator that reads files from disk and expecting equal speed.
• Using a Python generator for work that tf.data can pipeline more efficiently.
• Measuring model speed without checking whether the GPU is waiting on input.
• Keeping expensive augmentations in the training loop when they could be cached or vectorized.
• Treating fit_generator as a separate modern API. In current Keras, model.fit already handles generator-like inputs.

Summary

• Generator-based training is often slower because input preparation happens batch by batch.
• The slowdown is usually caused by Python, disk I/O, decoding, or preprocessing overhead.
• 'keras.utils.Sequence is safer than a bare generator for batch loading.'
• 'tf.data with batching and prefetching is usually the best long-term pipeline.'
• The smaller and faster the model, the more visible input pipeline overhead becomes.