Introduction

TensorFlow often tries to reserve most or all visible GPU memory up front so it can avoid later fragmentation and allocation overhead. That behavior is fast for single-process workloads, but it is frustrating on shared machines because one process can appear to monopolize the device. The usual fixes are enabling memory growth or imposing an explicit memory limit.

Let TensorFlow Grow Memory On Demand

In modern TensorFlow, the most common solution is memory growth.

1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4if gpus:
5    tf.config.experimental.set_memory_growth(gpus[0], True)

With memory growth enabled, TensorFlow allocates GPU memory incrementally instead of grabbing everything immediately.

This is often the best default on development machines where multiple GPU processes may coexist.

Set A Hard Memory Limit

If you want a stricter cap, create a logical device configuration.

1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4if gpus:
5    tf.config.set_logical_device_configuration(
6        gpus[0],
7        [tf.config.LogicalDeviceConfiguration(memory_limit=4096)]
8    )

This limits TensorFlow to a fixed amount of memory, here 4096 MB on the first GPU.

Use this when you need predictable sharing rather than just friendlier behavior.

The Configuration Must Happen Early

GPU memory settings must be applied before TensorFlow initializes the GPU runtime.

That means the configuration should run immediately after importing TensorFlow and before any model creation or tensor allocation that touches the GPU.

If TensorFlow has already initialized the device, changing the setting later usually raises an error.

TensorFlow 1.x Session-Based Equivalent

Older TensorFlow 1.x style code used session config.

1import tensorflow as tf
2
3config = tf.compat.v1.ConfigProto()
4config.gpu_options.allow_growth = True
5session = tf.compat.v1.Session(config=config)

Or to reserve only a fraction:

config.gpu_options.per_process_gpu_memory_fraction = 0.5

That style still matters if you maintain legacy graph-session code.

Environment Variable Option

A useful environment-based switch is:

export TF_FORCE_GPU_ALLOW_GROWTH=true

This can help in environments where modifying application code is inconvenient, such as notebooks, wrappers, or third-party scripts.

Why You Might Still Want Full Preallocation

Preallocating most memory is not purely bad. For dedicated training jobs on isolated GPUs, it can reduce fragmentation and produce more stable performance.

So the right choice depends on whether the GPU is:

• shared among multiple users or processes
• dedicated to one long-lived training run

Development and multi-tenant systems usually prefer friendlier allocation. Dedicated training boxes may not care.

Multiple GPUs Need Explicit Handling

If several GPUs are visible, configure the one or ones you intend to use. It is often wise to combine memory policy with device visibility controls so the process does not accidentally touch the wrong GPU.

Growth Versus Hard Caps

Memory growth and explicit limits solve different problems. Growth is polite because it avoids grabbing everything immediately, but it can still expand later. A hard cap is stricter and is the better choice when several long-lived workloads must share one GPU predictably.

Common Pitfalls

The biggest mistake is configuring memory growth after TensorFlow has already initialized the GPU. Another is assuming allow_growth and hard limits do the same thing; they do not. Developers also sometimes forget that TensorFlow 1.x and 2.x use different APIs for this problem. Finally, if another framework or process already occupies most of the GPU, TensorFlow memory settings cannot create capacity that is already gone.

Summary

• Use memory growth when you want TensorFlow to allocate GPU memory gradually.
• Use a logical device memory limit when you want a strict cap.
• Apply the configuration before TensorFlow initializes the GPU.
• Legacy TensorFlow 1.x code uses session config instead of the newer device APIs.
• The best setting depends on whether the GPU is shared or dedicated.