Introduction

If inference runs on the CPU even though a GPU is installed, the problem is usually not the predict method itself. It is almost always one of four things: the framework cannot see the GPU, the model or tensors were placed on the CPU, some operations do not have GPU kernels, or the workload is too small to show obvious GPU utilization. The fix is to verify device visibility first, then confirm that both the model and the actual prediction inputs are on the device you expect.

Check Whether the Framework Really Sees the GPU

Do not rely on the fact that nvidia-smi shows a device. Your ML framework must be built with GPU support and linked to compatible drivers and CUDA libraries.

A quick TensorFlow check:

import tensorflow as tf

print(tf.config.list_physical_devices('GPU'))

A quick PyTorch check:

1import torch
2
3print(torch.cuda.is_available())
4print(torch.cuda.device_count())
5if torch.cuda.is_available():
6    print(torch.cuda.get_device_name(0))

If these checks fail, prediction will stay on the CPU no matter what your code looks like.

In PyTorch, Move Both Model and Inputs

In PyTorch, a GPU-visible environment is still not enough. The model and input tensors must live on the same CUDA device.

1import torch
2import torch.nn as nn
3
4model = nn.Sequential(
5    nn.Linear(10, 16),
6    nn.ReLU(),
7    nn.Linear(16, 2)
8)
9
10device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
11model.to(device)
12model.eval()
13
14x = torch.randn(4, 10).to(device)
15with torch.no_grad():
16    y = model(x)
17
18print(y.device)

A common failure mode is moving the model to CUDA during training, then creating new prediction tensors on the CPU during inference. In that case PyTorch usually raises a device mismatch error, but wrappers can obscure it.

In TensorFlow, Check Placement and Unsupported Ops

TensorFlow places many operations automatically, but not all operations have a GPU kernel. You can inspect placement decisions with device logging.

1import tensorflow as tf
2
3tf.debugging.set_log_device_placement(True)
4
5model = tf.keras.Sequential([
6    tf.keras.layers.Dense(16, activation='relu', input_shape=(10,)),
7    tf.keras.layers.Dense(2)
8])
9
10x = tf.random.normal((4, 10))
11y = model.predict(x, verbose=0)
12print(y.shape)

If logs show CPU placement, that can be legitimate. Some preprocessing steps, string operations, control-flow patterns, or custom layers may not run on the GPU.

Small Predictions Often Look Like CPU Work

Another subtle issue is workload size. A tiny inference batch can finish so quickly that GPU usage barely registers. Kernel launch overhead and data transfer can dominate, making the GPU appear idle.

This is especially common when:

• you call predict one sample at a time
• preprocessing runs on the CPU before every call
• the model is small compared with transfer overhead
• the monitoring tool samples utilization too slowly

To test this, increase the batch size and time the run.

1import time
2import torch
3import torch.nn as nn
4
5model = nn.Sequential(nn.Linear(1024, 2048), nn.ReLU(), nn.Linear(2048, 10)).cuda().eval()
6x = torch.randn(4096, 1024, device='cuda')
7
8torch.cuda.synchronize()
9start = time.time()
10with torch.no_grad():
11    _ = model(x)
12torch.cuda.synchronize()
13print("elapsed:", time.time() - start)

This kind of test is more informative than watching utilization while predicting a single row.

Watch for Hidden CPU Steps

In end-to-end pipelines, .predict() may be only one part of the work. Tokenization, image decoding, feature assembly, Pandas transformations, and postprocessing are usually CPU tasks.

That means your overall prediction service can appear CPU-bound even if the model forward pass uses the GPU correctly. Profiling is the right tool here, not guesswork.

For TensorFlow, use the profiler. For PyTorch, use the profiler or insert targeted timers around preprocessing, model execution, and postprocessing separately.

Common Pitfalls

A common mistake is checking only whether a GPU exists on the machine, not whether the framework can use it.

Another mistake is moving the model to the GPU but leaving inference inputs on the CPU, or converting tensors back to NumPy too early.

People also often expect strong GPU utilization from tiny batches or tiny models. In those cases the CPU may genuinely be competitive.

Finally, unsupported operations can force parts of the graph onto the CPU. Custom layers and preprocessing code are frequent culprits.

Summary

• If prediction stays on the CPU, first verify that TensorFlow or PyTorch actually sees the GPU
• In PyTorch, both the model and the input tensors must be on the same CUDA device
• In TensorFlow, placement is automatic, but unsupported operations can still run on the CPU
• Tiny inference workloads may not show meaningful GPU utilization even when GPU execution is correct
• Pipeline preprocessing often dominates runtime and is usually CPU-bound
• Profile the full inference path instead of assuming the predict call alone explains performance