Introduction

Running a container with GPU access is a Docker host configuration problem first and a Compose syntax problem second. If the NVIDIA container toolkit is installed correctly, Docker Compose can pass GPU devices into the service; if the host is not ready, no Compose file will fix it.

Verify the Host Before Touching Compose

Before editing docker-compose.yml, confirm that plain Docker can see the GPU. This isolates driver and toolkit problems from Compose problems.

docker run --rm --gpus all nvidia/cuda:12.3.2-base-ubuntu22.04 nvidia-smi

If that command fails, fix the host first. Common prerequisites are:

• a working NVIDIA driver on the machine
• the NVIDIA Container Toolkit installed
• a recent Docker Engine that supports --gpus

Only move to Compose after nvidia-smi works in a normal docker run.

Modern Compose Configuration

On current Docker setups, the preferred approach is to request GPU devices explicitly. A Compose service can declare a device reservation so the container gets access to the NVIDIA driver stack.

1services:
2  trainer:
3    image: nvidia/cuda:12.3.2-base-ubuntu22.04
4    command: nvidia-smi
5    deploy:
6      resources:
7        reservations:
8          devices:
9            - driver: nvidia
10              count: 1
11              capabilities: [gpu]

Then start the service:

docker compose up

If your environment supports GPU reservations in Compose, the container should print the GPU details and exit successfully.

When you need every available GPU instead of a single device, replace count: 1 with count: all.

Legacy runtime: nvidia

Older Compose examples often show a service-level runtime setting:

1services:
2  trainer:
3    image: nvidia/cuda:11.8.0-base-ubuntu22.04
4    runtime: nvidia
5    command: nvidia-smi

This format existed before Docker standardized the --gpus workflow. It can still appear in older projects, but it depends heavily on the Docker and Compose versions installed on the host. For new setups, treat it as a compatibility path rather than the default recommendation.

That distinction matters because many failures come from copying an old blog post into a newer Docker environment and assuming the syntax is universal.

Check the Container After Startup

Once the service is up, verify that the container can really use the GPU rather than assuming the Compose file worked.

docker compose ps
docker compose exec trainer nvidia-smi

For machine learning workloads, you can also check from inside the application runtime. For example, with Python and PyTorch:

1import torch
2
3print("CUDA available:", torch.cuda.is_available())
4print("Device count:", torch.cuda.device_count())

If torch.cuda.is_available() returns False, the issue is still in the container runtime path, not in your training code.

Keep the Compose File Focused

It is tempting to change several things at once: base image, runtime syntax, environment variables, and application command. Resist that. Start with a minimal service that only runs nvidia-smi. Once that works, add your actual image and command. This staged approach gives you a clean checkpoint and prevents application bugs from being confused with GPU runtime bugs.

For example, this is a good progression:

1. Run docker run --gpus all ... nvidia-smi
2. Run minimal Compose service with nvidia-smi
3. Add your real image
4. Add your application command

Each step narrows the problem space.

Common Pitfalls

• Editing Compose before verifying docker run --gpus all works wastes time. Fix host drivers and toolkit issues first.
• Using runtime: nvidia on a newer setup can fail because many current environments expect device requests instead of legacy runtime configuration.
• Assuming the container has GPU access because it started successfully is misleading. Always run nvidia-smi or an application-level CUDA check.
• Mixing application debugging with runtime debugging makes failures harder to isolate. Start with a minimal CUDA image and a trivial command.
• Forgetting that Compose behavior depends on Docker and toolkit versions leads to copy-paste errors. Match the syntax to the environment you actually have.

Summary

• Confirm the host GPU setup with docker run --gpus all ... nvidia-smi first.
• Prefer explicit GPU device reservations in Compose for modern Docker environments.
• Treat runtime: nvidia as a legacy compatibility option, not the default approach.
• Validate access inside the running container with nvidia-smi or a CUDA-aware library.
• Debug in stages so you can separate host, runtime, and application issues.