Is there a way of determining how much GPU memory is in use by TensorFlow?
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Yes, but there are two different questions hiding inside that sentence. You might want to know how much memory TensorFlow's allocator is actively using inside your program, or you might want to know how much GPU memory the whole process has reserved from the system.
Those numbers are related but not identical. That is why TensorFlow APIs and nvidia-smi can report different values without either one being wrong.
TensorFlow Runtime Memory Info
For programmatic inspection inside your Python process, TensorFlow can report allocator memory information:
The current value reflects memory currently tracked by TensorFlow's allocator for that device. The peak value is useful for estimating worst-case usage during a run.
This is usually the best choice when:
- you want to log memory use during training
- you are comparing different model configurations
- you need metrics inside the program itself
System-Level Monitoring with nvidia-smi
If you want to see what the operating environment sees, use nvidia-smi:
Or refresh continuously:
This shows per-process GPU memory usage and is especially useful when multiple jobs share one GPU. It answers the operational question: how much memory is this process occupying on the device from the system's point of view.
Why the Numbers Can Differ
This is the part that confuses most people. TensorFlow often uses a caching allocator. That means memory may be reserved from the GPU and kept available for reuse even if a particular tensor is no longer active.
So:
- TensorFlow API reports allocator state
- '
nvidia-smireports process-level reservation and context overhead'
You may see nvidia-smi report a larger number because it includes:
- CUDA context setup
- reserved but currently reusable memory
- allocator behavior outside the narrow "current tensor usage" view
These are different measurement layers, not contradictory facts.
Preventing Aggressive Preallocation
Older TensorFlow setups often grabbed most of the GPU memory at startup. A common way to avoid that is enabling memory growth:
With memory growth enabled, TensorFlow tends to request memory as needed rather than reserving almost everything immediately. That makes coexistence with other GPU processes much easier.
Checking Multiple GPUs
If your system has more than one GPU, inspect them separately:
This is helpful in distributed training or multi-GPU experiments where one device may be overloaded while another is underused.
A Practical Logging Pattern
You can log memory during training or after large operations:
This is often enough to compare batch sizes, model widths, or mixed-precision settings without leaving the training script.
Common Pitfalls
The biggest pitfall is assuming nvidia-smi and TensorFlow should show the exact same number. They measure different layers of memory usage.
Another issue is querying memory before any GPU work has happened. If the model has not run yet, the numbers may be small or uninformative.
Developers also sometimes forget that memory growth must be configured before TensorFlow initializes the GPU runtime. If set too late, it may have no effect.
Finally, if multiple processes share a GPU, TensorFlow-only metrics tell you nothing about the other jobs. In that case, system tools such as nvidia-smi are mandatory.
Summary
- Use TensorFlow memory APIs for in-process measurements such as current and peak allocator usage.
- Use
nvidia-smifor system-level per-process GPU memory monitoring. - The two numbers can differ because TensorFlow caches memory and
nvidia-smiincludes reservation overhead. - Enable memory growth when you want TensorFlow to avoid grabbing most of the GPU upfront.
- For real debugging, use both views together rather than relying on only one.
Related reading
- Is there a way to check if mxnet uses my gpu?
- Is there a way to determine where messages came from in a Kafka topic?
- Is there a way to impose a constraint in tensor flow, could I enforce some rule along the way?
- Is there a way to set up a multi-hidden layer neural network with the mlp method in the caret package?
- Is there a way to get tensorflow tf.Print output to appear in Jupyter Notebook output
- Is there a way to stack two tensorflow datasets?
- Is there a way to choose the k nearest neighbors in scikits learn with a user defined distance metric?
- Is there a way to make this Python kNN function more efficient?

DSA Fundamentals
Master algorithmic patterns and data structures through hands-on LeetCode-style problems - from arrays and hashing to dynamic programming and advanced graphs.
View the courseML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.