Introduction

If GPU utilization sits near 0% during training, the GPU is usually waiting for something else rather than doing useful work. The most common causes are an input pipeline bottleneck, batches that are too small, CPU-bound preprocessing, or the fact that tools like nvidia-smi sample utilization so coarsely that short bursts of GPU work look like idleness.

First: Do Not Trust One Snapshot Blindly

nvidia-smi is useful, but it reports sampled utilization. Deep learning workloads often run in bursts:

• CPU prepares a batch,
• data is transferred,
• a short GPU kernel runs,
• then the GPU waits again.

If the kernel bursts are short, a coarse utilization sample can show 0% even though the GPU is active in spikes. So the first step is to observe a time series, not a single number.

Also watch:

• batch time,
• data-loader time,
• host CPU usage,
• and GPU memory usage.

Low utilization with high memory occupancy tells a different story than low utilization with almost no allocated memory.

The Most Common Cause: Data Pipeline Starvation

If the CPU cannot prepare batches fast enough, the GPU runs briefly and then waits for the next batch.

In PyTorch, a better data loader often helps immediately:

1from torch.utils.data import DataLoader
2
3loader = DataLoader(
4    dataset,
5    batch_size=128,
6    shuffle=True,
7    num_workers=4,
8    pin_memory=True,
9    persistent_workers=True,
10)

Then move tensors non-blockingly:

for inputs, targets in loader:
    inputs = inputs.cuda(non_blocking=True)
    targets = targets.cuda(non_blocking=True)

If CPU-side image decoding, augmentation, or disk reads dominate the step time, the GPU cannot stay busy.

Batch Size Can Be Too Small

A very small batch may not contain enough work to keep the GPU occupied efficiently.

For example:

• tiny models,
• tiny inputs,
• and batch size 1 or 2

can produce kernels so small that launch overhead and synchronization dominate the step.

If memory allows, increasing batch size is often the simplest way to raise utilization. The right batch size depends on your model and optimizer, but if the GPU is nearly empty and the batch is tiny, that is an obvious place to experiment.

Check That the Model Is Really on the GPU

This sounds basic, but partial CPU fallback is common in real code. In PyTorch:

1device = "cuda"
2model = model.to(device)
3
4for inputs, targets in loader:
5    inputs = inputs.to(device)
6    targets = targets.to(device)

If some tensors or custom operations stay on CPU, the training loop can bounce between host and device in a way that destroys throughput.

The same idea applies in TensorFlow: verify that the heavy compute path is really placed on the GPU rather than silently falling back to CPU.

Watch for Expensive Python or CPU Work in the Loop

Another common pattern is doing too much non-GPU work per step:

• logging every batch,
• converting tensors to NumPy,
• frequent .item() calls,
• expensive metric calculations on CPU,
• or Python-side augmentation inside the training loop.

Each of those can introduce synchronization or host-side overhead that leaves the GPU idle.

For example, calling .item() repeatedly forces synchronization:

loss_value = loss.item()

That is fine occasionally, but doing it excessively inside a tight loop can slow training more than people expect.

Small Models May Never Saturate a Big GPU

Sometimes nothing is "wrong." If the model is small and the workload is light, a large GPU may simply be overprovisioned. A tiny CNN or MLP can train correctly while showing low utilization because there just is not enough arithmetic to keep the device busy.

In that case, the fix is not pipeline tuning so much as realistic expectations about workload size and hardware scale.

Practical Profiling Beats Guessing

A good diagnostic routine is:

1. measure step time,
2. separate data-loading time from forward/backward time,
3. increase batch size if memory allows,
4. profile CPU preprocessing,
5. verify device placement,
6. and only then optimize the slowest stage.

Framework profilers are more informative than guessing from utilization alone.

Common Pitfalls

The biggest pitfall is treating one nvidia-smi snapshot as proof that the GPU is not being used. Short bursts of work can be missed by coarse sampling.

Another mistake is blaming the model first when the real bottleneck is data loading, decoding, or augmentation on CPU.

Developers also often keep batch sizes too small or synchronize too frequently with logging and .item() calls.

Finally, if the model is genuinely tiny, low utilization may just mean the GPU is much larger than the workload requires.

Summary

• Low GPU utilization usually means the GPU is waiting, not that it is broken.
• Input-pipeline starvation is the most common cause.
• Small batches, CPU fallback, and excessive synchronization also hurt utilization.
• 'nvidia-smi can understate bursty workloads, so profile over time rather than trusting one sample.'
• Diagnose with step timing and pipeline profiling before changing the model blindly.