Introduction

It is entirely possible for a Keras LSTM to run faster on CPU than GPU, especially for small models or workloads that do not hit the optimized GPU path. GPUs win by doing a lot of work in parallel, but they also pay launch overhead, data-movement cost, and kernel-selection cost. If the recurrent workload is small or poorly configured for the fast path, the CPU can finish sooner.

Small Sequence Models Often Do Not Feed the GPU Well

An LSTM has sequential dependencies across time steps. Within each step there is matrix math, but the sequence dimension still limits how much work can run in parallel compared with a large convolution or transformer block.

If the batch size, hidden size, or sequence length is small, GPU overhead may dominate the useful work. The CPU, especially with optimized BLAS libraries, can be very competitive in that regime.

A simple benchmark skeleton shows the pattern.

1import time
2import tensorflow as tf
3
4x = tf.random.normal((64, 20, 16))
5model = tf.keras.Sequential([
6    tf.keras.layers.LSTM(32),
7    tf.keras.layers.Dense(1)
8])
9
10model(x)
11
12start = time.time()
13for _ in range(100):
14    _ = model(x, training=False)
15print(time.time() - start)

If you repeat that with larger batch sizes and longer sequences, the GPU often improves relative to the CPU because there is finally enough work to amortize the overhead.

The Fast GPU Path Has Requirements

Historically, Keras and TensorFlow achieve the best LSTM GPU performance when the layer configuration matches the fused implementation that maps well to cuDNN-style kernels. If your settings fall off that path, TensorFlow may use a slower generic implementation.

Common reasons include custom activations, unusual recurrent settings, or shapes that prevent the optimized kernel from being used efficiently. In other words, “running on GPU” does not automatically mean “using the best GPU implementation”.

For many mainstream cases, a standard configuration is the safest starting point.

1layer = tf.keras.layers.LSTM(
2    units=128,
3    activation="tanh",
4    recurrent_activation="sigmoid",
5    dropout=0.0,
6    recurrent_dropout=0.0,
7    unroll=False,
8)

This does not guarantee peak speed by itself, but it avoids some settings that commonly disable the most efficient execution path.

Input Pipeline and Transfer Overhead Matter

If each batch must be prepared on the CPU and copied to the GPU, a small model may spend more time waiting on transfer and scheduling than on recurrent math. That becomes more obvious when:

• batches are tiny,
• the model is shallow,
• or inference is called many times with small arrays.

This is why quick notebook tests sometimes report a slower GPU even though a production training run with larger batches would favor the GPU strongly.

CPUs Are Better Than People Assume for RNN-Sized Workloads

Modern CPUs have strong vector instructions, good cache behavior for medium-sized tensors, and optimized math libraries. An LSTM that is too small to saturate the GPU may actually fit very well into CPU cache and thread scheduling.

So the right mental model is not “GPU is always faster”. It is “GPU is faster when the workload is large and aligned with the optimized kernels”. Recurrent models cross that threshold later than many developers expect.

Measure the Whole Pipeline, Not Only the Layer

Another trap is timing the forward pass without considering warm-up, graph tracing, input preparation, and device synchronization. A fair benchmark should include warm-up and consistent device placement.

1@tf.function
2def predict_step(batch):
3    return model(batch, training=False)
4
5for _ in range(10):
6    predict_step(x)
7
8start = time.time()
9for _ in range(100):
10    predict_step(x)
11print(time.time() - start)

Without warm-up, the first few calls can include tracing overhead that distorts comparisons.

What Usually Helps the GPU

If you want the GPU to win, the usual levers are:

• larger batch sizes,
• longer sequences,
• larger hidden dimensions,
• optimized LSTM settings,
• and an input pipeline that keeps the device fed.

If your use case is low-latency inference on small batches, the CPU may remain the better choice. That is not a failure. It is just the actual workload profile.

Common Pitfalls

• Assuming device choice alone determines performance without checking model size and batch shape.
• Using an LSTM configuration that misses the optimized GPU execution path.
• Benchmarking without warm-up or with significant host-to-device transfer overhead.
• Expecting sequence models to parallelize like large convolutional workloads.
• Optimizing for theoretical throughput when the real requirement is small-batch latency.

Summary

• Keras LSTM can be faster on CPU when the workload is too small to amortize GPU overhead.
• GPU performance depends heavily on using the optimized recurrent execution path.
• Batch size, sequence length, and hidden size strongly affect the result.
• Measure warm-up, input transfer, and full pipeline behavior, not only one call.
• Choose the device for the real workload profile, not for the stereotype that GPU always wins.