Introduction

In PyTorch, "GPU available" and "your code is actually using the GPU" are not the same thing. The correct check is usually a combination of three things: CUDA availability, the device of your tensors and model, and whether the operations you care about are happening on that device.

Check Whether CUDA Is Available

The first check is whether the installed PyTorch build can see a CUDA-capable device at all.

1import torch
2
3print(torch.cuda.is_available())
4print(torch.cuda.device_count())
5print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else "no GPU")

If torch.cuda.is_available() is False, your code is not going to use the NVIDIA GPU through CUDA, no matter how the rest of the script is written.

This check confirms that PyTorch can detect the GPU. It does not confirm that your tensors or model are actually running there.

Check the Device of a Tensor or Model

The next step is to inspect the device of the objects doing the work.

For a tensor:

1import torch
2
3device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
4x = torch.randn(3, 3).to(device)
5
6print(x.device)

For a model:

1import torch
2import torch.nn as nn
3
4model = nn.Linear(4, 2)
5device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
6model = model.to(device)
7
8print(next(model.parameters()).device)

If the model parameters are still on cpu, then the forward pass is not running on the GPU even if CUDA is installed.

Make Sure Inputs and Model Match

A very common mistake is moving the model to CUDA but leaving the input tensors on the CPU, or the reverse. PyTorch expects them to be on compatible devices.

1import torch
2import torch.nn as nn
3
4model = nn.Linear(4, 2).to("cuda")
5inputs = torch.randn(5, 4).to("cuda")
6output = model(inputs)
7
8print(output.device)

If both model and inputs are on cuda, the output should also report a CUDA device.

That is often the most practical runtime confirmation: inspect the device of the tensors involved in the actual computation you care about.

A Minimal End-to-End Check

A small working script is often the fastest sanity check:

1import torch
2import torch.nn as nn
3
4if not torch.cuda.is_available():
5    print("CUDA is not available")
6else:
7    device = torch.device("cuda")
8    model = nn.Linear(10, 1).to(device)
9    x = torch.randn(8, 10, device=device)
10    y = model(x)
11
12    print("Model device:", next(model.parameters()).device)
13    print("Input device:", x.device)
14    print("Output device:", y.device)

If all three print a CUDA device, then the computation is actually happening on the GPU.

Use Monitoring When You Need Extra Confirmation

If you want confirmation outside Python, use nvidia-smi while the program is running:

nvidia-smi

That can show GPU memory usage and active processes. It is useful when debugging whether a training job really moved to the GPU or is just sitting on the CPU.

Still, nvidia-smi is a secondary check. In code, the most reliable test is still the device of the tensors and parameters involved in the computation.

Common Pitfalls

• Checking only torch.cuda.is_available() and assuming that means the model is already using the GPU.
• Moving the model to CUDA but forgetting to move the inputs.
• Moving inputs to CUDA but leaving the model on the CPU.
• Running on Apple Silicon or another non-CUDA backend and expecting torch.cuda checks to apply.
• Looking at the device too early, before the tensors or model have actually been moved.

Summary

• First check whether CUDA is available with torch.cuda.is_available().
• Then inspect the device of the actual tensors and model parameters.
• Model, inputs, and outputs should all report a CUDA device if the computation is on the GPU.
• A small end-to-end forward pass is the best practical sanity check.
• 'nvidia-smi is useful as an extra confirmation, but the code-level device checks matter most.'