Introduction

Running inference for multiple TensorFlow models on one GPU can improve utilization, but it can also create unstable latency if memory and scheduling are unmanaged. The fastest setup on paper is not always the most predictable in production. A practical design balances throughput, tail latency, and memory isolation.

Start with Controlled GPU Memory Behavior

If each model process pre-allocates most GPU memory, later models can fail to load or become heavily throttled. Enable memory growth before loading models.

1import tensorflow as tf
2
3physical_gpus = tf.config.list_physical_devices('GPU')
4for gpu in physical_gpus:
5    tf.config.experimental.set_memory_growth(gpu, True)
6
7print('GPUs:', physical_gpus)

Do this at process startup before creating any model or tensor. If memory growth is configured too late, TensorFlow can raise runtime errors.

Load Models Once and Reuse Them

Repeated loading during request handling creates severe overhead and memory churn. Keep loaded models in memory and run inference through stable call paths.

1import tensorflow as tf
2import numpy as np
3
4model_a = tf.keras.models.load_model('model_a')
5model_b = tf.keras.models.load_model('model_b')
6
7@tf.function
8def infer_a(x):
9    return model_a(x, training=False)
10
11@tf.function
12def infer_b(x):
13    return model_b(x, training=False)
14
15x = tf.constant(np.random.rand(32, 224, 224, 3), dtype=tf.float32)
16print(infer_a(x).shape)
17print(infer_b(x).shape)

The tf.function wrapper reduces Python overhead and helps stabilize repeated calls.

Choose Concurrency Model by Workload

For lightweight workloads, one process with queues and worker threads can be enough. For stricter isolation, separate processes per model may be safer.

Single-process queue pattern:

1import queue
2import threading
3
4work_q = queue.Queue()
5
6
7def worker():
8    while True:
9        item = work_q.get()
10        if item is None:
11            break
12        model_name, batch = item
13        if model_name == 'a':
14            _ = infer_a(batch)
15        else:
16            _ = infer_b(batch)
17        work_q.task_done()
18
19threads = [threading.Thread(target=worker, daemon=True) for _ in range(2)]
20for t in threads:
21    t.start()

This can raise throughput, but measure carefully. Too many worker threads can increase contention rather than speed.

Batch Strategically for Better Throughput

GPU inference efficiency usually improves with batching, but oversized batches can harm latency and memory.

A practical plan:

• define target latency budget
• test batch sizes such as 1, 8, 16, 32
• track p50 and p99 latency
• choose batch size by service goal, not by raw throughput alone

Basic timing harness:

1import time
2
3for batch_size in [1, 8, 16, 32]:
4    x = tf.random.uniform((batch_size, 224, 224, 3), dtype=tf.float32)
5    t0 = time.perf_counter()
6    _ = infer_a(x)
7    t1 = time.perf_counter()
8    print(batch_size, 'latency_ms', round((t1 - t0) * 1000, 2))

Repeat over many iterations and include warm-up calls to avoid first-run bias.

Isolate Critical Models When Needed

If one model serves latency-critical endpoints and another serves bulk traffic, sharing one process may cause unpredictable delays. In those cases, isolate models at process or service level and control request routing externally.

Common architecture choices:

• one service per model with shared GPU
• one service per model with dedicated GPU slice or device
• model server framework with scheduling controls

Isolation often reduces peak throughput slightly but improves operational stability.

Observability and Guardrails

Without metrics, parallel inference tuning becomes guesswork. At minimum, track:

• per-model request rate
• per-model p50 and p99 latency
• GPU memory usage
• GPU utilization
• inference error rate

Also set backpressure behavior. If queues grow beyond threshold, reject or shed low-priority traffic instead of allowing indefinite queue growth.

Common Pitfalls

A common pitfall is loading models before setting memory growth, then wondering why one model monopolizes GPU memory.

Another issue is maximizing batch size without latency limits, causing poor interactive response times.

Developers also over-thread inference on a single GPU and assume more threads always mean more throughput.

Ignoring warm-up effects is another frequent mistake. First request can be much slower than steady state.

Finally, avoid tuning with average latency only. Tail latency often decides user experience.

Summary

• Configure GPU memory behavior before model load.
• Keep models loaded and reuse inference functions.
• Pick concurrency model based on latency and isolation needs.
• Tune batch size with both throughput and tail latency metrics.
• Add observability and queue guardrails for production stability.