Introduction

Limiting TensorFlow CPU and memory usage usually means controlling three different things: thread parallelism, GPU memory behavior, and the size of the data or model you feed into the runtime. The right knob depends on whether you are trying to reduce host CPU contention, cap GPU memory growth, or stop the process from exhausting RAM.

Limiting CPU Usage

TensorFlow exposes thread settings for CPU execution:

1import tensorflow as tf
2
3tf.config.threading.set_intra_op_parallelism_threads(2)
4tf.config.threading.set_inter_op_parallelism_threads(2)

intra_op limits threads used within a single op. inter_op limits how many ops run in parallel. Lower values reduce CPU pressure, which is useful on shared machines or when TensorFlow is competing with other services.

These settings do not act like a perfect CPU quota, but they are the main built-in controls for CPU-side concurrency.

Set them as early as possible in process startup. TensorFlow and the libraries beneath it may initialize thread pools quickly, and late changes can lead to confusing results when you profile the application.

Limiting GPU Memory

On GPU, the common concern is memory preallocation. TensorFlow can be told to grow memory usage gradually:

1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4for gpu in gpus:
5    tf.config.experimental.set_memory_growth(gpu, True)

That prevents TensorFlow from grabbing all visible GPU memory immediately.

If you need a harder cap, configure a logical device memory limit:

1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4if gpus:
5    tf.config.set_logical_device_configuration(
6        gpus[0],
7        [tf.config.LogicalDeviceConfiguration(memory_limit=2048)]
8    )

This example limits the logical device to about 2 GB.

Order matters here. Memory-growth and logical-device settings must be applied before TensorFlow fully initializes the GPU runtime. If you allocate tensors first and configure the device later, TensorFlow will normally reject the change.

RAM Usage Is Often a Pipeline Problem

Host memory pressure often comes from batch size, dataset caching, and model size rather than from TensorFlow alone. If you cache a huge dataset in memory or batch more samples than the machine can hold comfortably, thread limits will not solve the problem.

Practical levers include:

• smaller batch sizes
• removing unnecessary dataset caching
• using lower-precision dtypes where appropriate
• reducing model width or sequence length

A restrained tf.data pipeline often matters more than thread tuning:

1import tensorflow as tf
2
3dataset = tf.data.Dataset.range(1_000_000)
4dataset = dataset.shuffle(buffer_size=1000)
5dataset = dataset.batch(32)
6dataset = dataset.prefetch(tf.data.AUTOTUNE)

This keeps only a modest shuffle window in memory. By contrast, a massive shuffle buffer or an eager cache() on a large dataset can consume gigabytes of RAM before model execution becomes the bottleneck.

Environment-Level Limits

Sometimes the best control is outside TensorFlow. Containers, cgroups, and job schedulers can enforce hard CPU and memory caps for the process. That is often more reliable when you need operational guarantees rather than just "be a little less aggressive."

In other words, TensorFlow knobs help shape behavior, but operating-system limits enforce policy.

That separation is important in production. TensorFlow configuration is good for tuning cooperative behavior on a shared machine. If you need hard safety limits, use container or scheduler controls as the outer boundary.

Common Pitfalls

• Trying to solve RAM exhaustion only with thread settings.
• Confusing GPU memory growth with an actual hard memory cap.
• Forgetting that data pipelines can consume large amounts of RAM.
• Applying GPU memory settings after the runtime has already initialized the device.
• Setting thread counts too low and then wondering why training throughput collapses.
• Expecting TensorFlow alone to provide perfect process-level resource isolation.

Summary

• Use thread settings to reduce CPU-side parallelism.
• Use GPU memory growth or logical device limits to control GPU memory behavior.
• Reduce batch size and dataset memory pressure when RAM is the real issue.
• Process-level limits may need OS or container tooling, not just TensorFlow APIs.
• Match the control mechanism to the actual resource problem you are seeing.