Introduction

Not in the way many people expect. In current TensorFlow, all discovered GPUs are visible by default unless you restrict them, and TensorFlow may reserve memory on all visible GPUs. But that does not mean your model is automatically training across all GPUs.

Visibility Is Not the Same as Multi-GPU Training

TensorFlow makes a distinction between devices being visible to the runtime and a model actually being distributed across those devices.

If you simply write ordinary TensorFlow or Keras code on a machine with several GPUs, TensorFlow can place operations on a GPU automatically. In many straightforward cases, work ends up on GPU:0 unless you explicitly configure a strategy or manual device placement.

So the short answer is:

• all GPUs are usually visible by default
• memory may be reserved on all visible GPUs
• computation is not automatically spread across all GPUs for training

What Happens by Default

This small example shows the available physical GPUs.

import tensorflow as tf

print(tf.config.list_physical_devices("GPU"))

If TensorFlow can see four GPUs, that does not mean a normal model.fit(...) call will use four replicas automatically. For true single-machine multi-GPU training, you typically use tf.distribute.MirroredStrategy.

1import tensorflow as tf
2from tensorflow import keras
3
4strategy = tf.distribute.MirroredStrategy()
5
6with strategy.scope():
7    model = keras.Sequential([
8        keras.layers.Input(shape=(10,)),
9        keras.layers.Dense(32, activation="relu"),
10        keras.layers.Dense(1)
11    ])
12    model.compile(optimizer="adam", loss="mse")

That strategy creates one replica per visible GPU and synchronizes updates across them.

Why People Think TensorFlow Uses Everything Automatically

The confusion comes from memory behavior. TensorFlow documentation explains that visible GPUs are visible to the runtime by default, and the runtime may map nearly all memory on each visible GPU unless you enable memory growth or restrict visibility.

That can make it look like TensorFlow is fully using every GPU, when in reality it may only be reserving memory while your actual model runs on one device or under a specific distribution policy.

How to Restrict TensorFlow to Specific GPUs

If you want TensorFlow to use only one GPU, set visible devices before the runtime initializes them.

1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4if gpus:
5    tf.config.set_visible_devices(gpus[0], "GPU")
6    tf.config.set_memory_growth(gpus[0], True)
7
8print(tf.config.list_logical_devices("GPU"))

This is useful on shared machines or when you want reproducible experiments on a fixed device.

Manual Device Placement Is Different Again

You can also place specific operations on a specific GPU.

1import tensorflow as tf
2
3with tf.device("/GPU:0"):
4    a = tf.random.normal((2000, 2000))
5    b = tf.random.normal((2000, 2000))
6    c = tf.matmul(a, b)
7
8print(c.device)

Manual placement tells TensorFlow where to run an operation, but it still does not create data-parallel multi-GPU training by itself.

Recommended Mental Model

Think of TensorFlow GPU behavior in three layers:

1. device discovery
2. device visibility and memory policy
3. actual distribution of computation

Beginners often look only at the first layer and assume the other two are automatic. They are not.

Common Pitfalls

A common mistake is reading GPU memory usage in nvidia-smi and assuming all GPUs are actively training the model. Memory reservation does not prove distributed execution.

Another mistake is calling tf.config.set_visible_devices after TensorFlow has already initialized GPUs. That raises a runtime error because visibility must be configured early.

Finally, if you want multi-GPU training, do not rely on default placement heuristics. Use tf.distribute.MirroredStrategy or another explicit strategy.

Summary

• TensorFlow usually makes all discovered GPUs visible by default.
• Visible does not mean your model automatically trains on all GPUs.
• TensorFlow may reserve memory on all visible GPUs, which can be misleading.
• Use tf.distribute.MirroredStrategy for normal single-host multi-GPU training.
• Restrict visibility and memory policy explicitly when you need predictable GPU usage.