Introduction

If Keras training gets slower every time you repeat an experiment, the problem is usually not that gradient descent somehow becomes inherently slower. More often, the process is accumulating state around the model: extra graphs, callbacks, file handles, datasets, or GPU memory pressure. The fix is to isolate each training run and measure where the slowdown is actually happening.

The Most Common Cause: Rebuilding Models Without Clearing State

A frequent pattern is training many models inside a loop while leaving the TensorFlow backend state alive. That can accumulate graph objects and memory allocations.

1import gc
2import tensorflow as tf
3from tensorflow import keras
4
5
6def make_model():
7    model = keras.Sequential([
8        keras.layers.Dense(64, activation="relu", input_shape=(20,)),
9        keras.layers.Dense(1, activation="sigmoid"),
10    ])
11    model.compile(optimizer="adam", loss="binary_crossentropy")
12    return model
13
14for run in range(5):
15    tf.keras.backend.clear_session()
16    model = make_model()
17    x = tf.random.normal((1000, 20))
18    y = tf.cast(tf.random.uniform((1000, 1)) > 0.5, tf.float32)
19    model.fit(x, y, epochs=3, batch_size=32, verbose=0)
20    del model
21    gc.collect()

clear_session() is the first thing to try when repeated experiments grow slower over time.

Distinguish Epoch Slowdown from Run-to-Run Slowdown

There are two different symptoms.

• Each later epoch inside one training run gets slower.
• Each new call to fit() in a repeated experiment loop gets slower.

The second case usually points to leaked state or a pipeline issue. The first case can come from callbacks, logging, dataset shuffling overhead, checkpointing, or CPU throttling.

Do not treat them as the same bug.

Check the Input Pipeline Before Blaming the Model

If the model trains quickly at the start and then slows down, data loading may be the real bottleneck. Python generators, disk reads, on-the-fly augmentation, and non-prefetched tf.data pipelines can all cause unstable throughput.

1import tensorflow as tf
2
3
4def make_dataset(x, y, batch_size=32):
5    ds = tf.data.Dataset.from_tensor_slices((x, y))
6    ds = ds.shuffle(1000)
7    ds = ds.batch(batch_size)
8    ds = ds.cache()
9    ds = ds.prefetch(tf.data.AUTOTUNE)
10    return ds

Caching and prefetching often help more than changing model architecture when the real issue is input starvation.

Watch Callbacks and Disk I/O

Training can degrade when you save a checkpoint, TensorBoard log, or CSV log too often. Writing to a slow disk on every epoch or every batch adds overhead that looks like model slowdown.

A practical test is to temporarily disable callbacks and compare timings.

1history = model.fit(
2    train_ds,
3    epochs=5,
4    verbose=0,
5)

If training becomes stable without callbacks, the model was not the problem.

GPU Memory Pressure Can Cause Secondary Slowdowns

On GPU systems, repeated training runs can fragment memory or push the process into a less efficient allocation pattern. Even if the code does not crash with an out-of-memory error, runtime can degrade as memory becomes tighter.

Useful checks include:

• monitoring GPU memory during repeated runs
• reducing batch size
• making sure old models are deleted before new ones are built
• avoiding unnecessary copies of training arrays

If you train many candidate models in one process, memory hygiene matters.

Use Timing Around the Real Steps

Instead of guessing, time the pieces separately.

1import time
2
3start = time.perf_counter()
4history = model.fit(train_ds, epochs=3, verbose=0)
5end = time.perf_counter()
6print(f"fit time: {end - start:.2f} seconds")

You can apply the same idea to dataset creation, model construction, evaluation, and checkpoint saving. Once you know which phase is growing, the fix becomes much clearer.

Reuse Data, Not Model State

For repeated experiments, it is reasonable to reuse preprocessed tensors or cached datasets. It is usually a mistake to reuse model state unless you are intentionally fine-tuning from a previous checkpoint.

That distinction matters. Reusing data improves speed. Reusing hidden backend state often creates the slowdown you are trying to diagnose.

Common Pitfalls

• Rebuilding many models in one process without calling tf.keras.backend.clear_session().
• Measuring only total runtime instead of timing model build, data loading, and callbacks separately.
• Using a slow Python data generator and assuming the neural network is the bottleneck.
• Saving checkpoints or logs too frequently.
• Ignoring GPU memory growth because the process still technically finishes.

Summary

• Repeated Keras slowdowns usually come from accumulated state, data pipelines, or I/O overhead.
• Clear the backend session between repeated experiment runs.
• Time the individual phases before changing model architecture.
• Optimize the input pipeline with caching and prefetching when data loading is the bottleneck.
• Remove unnecessary callbacks and manage GPU memory carefully in long experiment loops.