Introduction

TensorFlow usually detects all visible GPUs by default, but that does not mean your model automatically runs across all of them efficiently. Detection, memory allocation, and multi-GPU execution are related topics, and they are often confused.

The practical answer is: TensorFlow can see all available GPUs unless you hide some of them, but using them all for one training job normally requires an explicit distribution strategy.

What TensorFlow Does Automatically

When GPU-enabled TensorFlow starts, it typically enumerates visible GPU devices.

import tensorflow as tf

print(tf.config.list_physical_devices("GPU"))

If multiple GPUs are installed and visible through CUDA, TensorFlow will usually list them all.

Historically, TensorFlow would also try to allocate a large amount of memory on visible GPUs unless configured otherwise. That behavior is about memory management, not about automatic data-parallel training.

Does One Model Automatically Use All GPUs

Not by default in the sense most people mean.

A normal model script like this:

model.fit(dataset)

does not automatically become a correct multi-GPU training setup just because several GPUs exist.

To train across multiple GPUs intentionally, use a distribution strategy such as MirroredStrategy.

1strategy = tf.distribute.MirroredStrategy()
2
3with strategy.scope():
4    model = tf.keras.Sequential([
5        tf.keras.layers.Dense(64, activation="relu"),
6        tf.keras.layers.Dense(10)
7    ])
8    model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")

This is the explicit step that tells TensorFlow to replicate the model and coordinate work across GPUs.

Controlling Which GPUs Are Visible

You can restrict TensorFlow to selected GPUs.

gpus = tf.config.list_physical_devices("GPU")
tf.config.set_visible_devices(gpus[:1], "GPU")

Or at the environment level:

CUDA_VISIBLE_DEVICES=0 python train.py

This is useful for debugging, shared machines, and avoiding accidental contention with other jobs.

Memory Growth and Allocation

Even when TensorFlow is not actively training on all GPUs, it may still reserve GPU memory in ways that surprise users.

To make allocation more incremental, enable memory growth.

for gpu in tf.config.list_physical_devices("GPU"):
    tf.config.experimental.set_memory_growth(gpu, True)

This is often a good default on development machines where multiple processes may share the same hardware.

What Often Causes Confusion

People often observe this sequence:

1. TensorFlow lists multiple GPUs
2. GPU memory appears to be touched
3. training still is not actually scaling across all devices

That happens because visibility and memory management are not the same thing as coordinated multi-device execution.

TensorFlow can know about many GPUs without automatically parallelizing your training in the way you expect.

Quick Diagnostic Pattern

A practical debugging pattern is to print visible GPUs, enable memory growth if needed, and then watch actual device utilization while running a small training step. That separates device discovery from real workload placement.

On many machines, this makes it obvious whether the issue is visibility, memory configuration, or missing distribution strategy.

Common Pitfalls

The biggest mistake is assuming that detected GPUs automatically imply multi-GPU training. Usually they do not.

Another common issue is confusing memory allocation behavior with actual compute utilization.

People also forget to use a distribution strategy and then wonder why only one GPU seems busy during training.

Finally, if a machine has many GPUs but you want deterministic debugging, limit visible devices explicitly instead of trusting defaults.

Summary

• TensorFlow usually detects all visible GPUs by default.
• Detection is not the same thing as automatically using all GPUs for one training job.
• Multi-GPU training typically requires a strategy such as MirroredStrategy.
• GPU visibility can be restricted with TensorFlow APIs or CUDA_VISIBLE_DEVICES.
• Memory allocation behavior can be tuned with memory growth settings.
• Separate the questions of device discovery, memory use, and actual parallel execution.