TensorFlow
GPU Memory
Deep Learning
Machine Learning
Memory Management

Is there a way of determining how much GPU memory is in use by TensorFlow?

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Introduction

Yes, but there are two different questions hiding inside that sentence. You might want to know how much memory TensorFlow's allocator is actively using inside your program, or you might want to know how much GPU memory the whole process has reserved from the system.

Those numbers are related but not identical. That is why TensorFlow APIs and nvidia-smi can report different values without either one being wrong.

TensorFlow Runtime Memory Info

For programmatic inspection inside your Python process, TensorFlow can report allocator memory information:

python
1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4for gpu in gpus:
5    tf.config.experimental.set_memory_growth(gpu, True)
6
7x = tf.random.normal((4096, 4096))
8y = tf.matmul(x, x)
9
10info = tf.config.experimental.get_memory_info("GPU:0")
11print(info)
12print("Current MB:", info["current"] / 1024 / 1024)
13print("Peak MB:", info["peak"] / 1024 / 1024)

The current value reflects memory currently tracked by TensorFlow's allocator for that device. The peak value is useful for estimating worst-case usage during a run.

This is usually the best choice when:

  • you want to log memory use during training
  • you are comparing different model configurations
  • you need metrics inside the program itself

System-Level Monitoring with nvidia-smi

If you want to see what the operating environment sees, use nvidia-smi:

bash
nvidia-smi

Or refresh continuously:

bash
watch -n 1 nvidia-smi

This shows per-process GPU memory usage and is especially useful when multiple jobs share one GPU. It answers the operational question: how much memory is this process occupying on the device from the system's point of view.

Why the Numbers Can Differ

This is the part that confuses most people. TensorFlow often uses a caching allocator. That means memory may be reserved from the GPU and kept available for reuse even if a particular tensor is no longer active.

So:

  • TensorFlow API reports allocator state
  • 'nvidia-smi reports process-level reservation and context overhead'

You may see nvidia-smi report a larger number because it includes:

  • CUDA context setup
  • reserved but currently reusable memory
  • allocator behavior outside the narrow "current tensor usage" view

These are different measurement layers, not contradictory facts.

Preventing Aggressive Preallocation

Older TensorFlow setups often grabbed most of the GPU memory at startup. A common way to avoid that is enabling memory growth:

python
1import tensorflow as tf
2
3gpus = tf.config.list_physical_devices("GPU")
4for gpu in gpus:
5    tf.config.experimental.set_memory_growth(gpu, True)

With memory growth enabled, TensorFlow tends to request memory as needed rather than reserving almost everything immediately. That makes coexistence with other GPU processes much easier.

Checking Multiple GPUs

If your system has more than one GPU, inspect them separately:

python
1import tensorflow as tf
2
3for index, gpu in enumerate(tf.config.list_physical_devices("GPU")):
4    name = f"GPU:{index}"
5    info = tf.config.experimental.get_memory_info(name)
6    print(name, info)

This is helpful in distributed training or multi-GPU experiments where one device may be overloaded while another is underused.

A Practical Logging Pattern

You can log memory during training or after large operations:

python
1def log_gpu_memory(device_name="GPU:0"):
2    info = tf.config.experimental.get_memory_info(device_name)
3    current_mb = info["current"] / 1024 / 1024
4    peak_mb = info["peak"] / 1024 / 1024
5    print(f"{device_name}: current={current_mb:.2f} MB peak={peak_mb:.2f} MB")
6
7
8log_gpu_memory()

This is often enough to compare batch sizes, model widths, or mixed-precision settings without leaving the training script.

Common Pitfalls

The biggest pitfall is assuming nvidia-smi and TensorFlow should show the exact same number. They measure different layers of memory usage.

Another issue is querying memory before any GPU work has happened. If the model has not run yet, the numbers may be small or uninformative.

Developers also sometimes forget that memory growth must be configured before TensorFlow initializes the GPU runtime. If set too late, it may have no effect.

Finally, if multiple processes share a GPU, TensorFlow-only metrics tell you nothing about the other jobs. In that case, system tools such as nvidia-smi are mandatory.

Summary

  • Use TensorFlow memory APIs for in-process measurements such as current and peak allocator usage.
  • Use nvidia-smi for system-level per-process GPU memory monitoring.
  • The two numbers can differ because TensorFlow caches memory and nvidia-smi includes reservation overhead.
  • Enable memory growth when you want TensorFlow to avoid grabbing most of the GPU upfront.
  • For real debugging, use both views together rather than relying on only one.

Course
Intermediate
27 lessons
15 hours
DSA Fundamentals

Master algorithmic patterns and data structures through hands-on LeetCode-style problems - from arrays and hashing to dynamic programming and advanced graphs.

View the course
ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design