Introduction

Most MirroredStrategy errors come from one of four causes: TensorFlow cannot see the expected devices, variables are created outside the strategy scope, the input pipeline is not shaped correctly for distributed execution, or non-distributed code leaks into the training step. The fix is usually structural rather than a one-line flag change.

Core Sections

Create the model inside the strategy scope

The most common mistake is building variables before entering strategy.scope(). With mirrored training, TensorFlow needs to create mirrored variables that belong to the distribution strategy.

1import tensorflow as tf
2
3strategy = tf.distribute.MirroredStrategy()
4
5with strategy.scope():
6    model = tf.keras.Sequential([
7        tf.keras.layers.Dense(64, activation="relu"),
8        tf.keras.layers.Dense(10, activation="softmax")
9    ])
10    model.compile(
11        optimizer="adam",
12        loss="sparse_categorical_crossentropy",
13        metrics=["accuracy"],
14    )

If the model, optimizer, or certain variables are created outside the scope, TensorFlow may throw scope or variable placement errors.

Confirm TensorFlow actually sees the devices

If you expected multiple GPUs and TensorFlow sees only one or none, MirroredStrategy cannot behave the way you intended.

import tensorflow as tf

print(tf.config.list_physical_devices("GPU"))

If this list is empty, the problem is below the strategy layer. Check the TensorFlow build, CUDA or ROCm setup, driver versions, and whether the runtime environment exposes the GPUs at all.

Batch the dataset correctly

Distributed training splits each global batch across replicas. If batching is missing or inconsistent, training can fail or behave strangely.

train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_ds = train_ds.shuffle(10000).batch(128).prefetch(tf.data.AUTOTUNE)

The batch size above is the global batch size. TensorFlow divides it across replicas automatically under MirroredStrategy.

Small or irregular batches can trigger shape issues, especially near the end of the dataset. In some cases, drop_remainder=True helps create stable per-replica shapes.

Keep custom training logic distribution-safe

If you are using a custom training loop, all replica-aware computation needs to run through the strategy correctly. That includes per-replica loss reduction and gradient application.

For many projects, the simplest path is to let model.fit() handle the training loop under the strategy. Custom loops are powerful, but they also expose more ways to misuse per-replica values. When an error disappears under model.fit() but reappears in a manual loop, that is a strong sign the bug is in replica handling rather than in the model itself.

Watch for incompatible stateful patterns

Errors also appear when code assumes a single device context. Examples include manually placing tensors on one GPU, storing Python-side state that does not align with replicas, or mixing distribution strategy code with old graph/session-era assumptions.

When debugging, simplify first:

• remove custom callbacks not essential to the issue
• reduce to one dataset and one model
• verify the code works on one device
• then add MirroredStrategy back incrementally

That isolates whether the failure is truly distribution-specific. It also makes stack traces shorter, which helps distinguish a real strategy misuse from a secondary error that only appears once distributed execution amplifies it.

Common Pitfalls

• Building the model or optimizer outside strategy.scope(), which creates variables that are not mirrored correctly.
• Assuming TensorFlow can use multiple GPUs before verifying that the runtime actually detects them.
• Feeding an unbatched or shape-unstable dataset into distributed training.
• Writing custom training logic that ignores replica reduction or per-device value handling.
• Debugging everything at once instead of first confirming the same model works on a single device without the distribution strategy.

Summary

• 'MirroredStrategy errors are usually about device visibility, variable scope, or dataset structure.'
• Create the model and optimizer inside strategy.scope().
• Verify the GPUs are visible before blaming the strategy API.
• Batch the dataset consistently for distributed execution.
• Start from a minimal single-device working example and add distribution back in steps.