Practice Now
TensorFlow
concurrent sessions
machine learning
deep learning
Python programming

Running multiple tensorflow sessions concurrently

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Deep LearningTensorFlow & KerasMachine LearningPython

Introduction

Running multiple TensorFlow sessions concurrently is useful for parallel model training, hyperparameter search, and serving multiple models. In TensorFlow 1.x, each tf.Session can run in its own thread or process with separate graphs. In TensorFlow 2.x, eager execution removed the session API, but you can still run multiple models concurrently using Python threading, multiprocessing, or tf.distribute strategies. GPU memory management is the main challenge — multiple sessions competing for GPU memory without proper configuration will crash with out-of-memory errors.

TensorFlow 1.x: Multiple Sessions

python
1import tensorflow.compat.v1 as tf
2tf.disable_eager_execution()
3
4# Session 1: simple computation
5graph1 = tf.Graph()
6with graph1.as_default():
7    a = tf.constant(2.0)
8    b = tf.constant(3.0)
9    result1 = tf.multiply(a, b)
10
11# Session 2: different computation
12graph2 = tf.Graph()
13with graph2.as_default():
14    x = tf.constant(10.0)
15    y = tf.constant(5.0)
16    result2 = tf.add(x, y)
17
18# Run concurrently with threads
19import threading
20
21results = {}
22
23def run_session(graph, op, name):
24    with tf.Session(graph=graph) as sess:
25        results[name] = sess.run(op)
26
27t1 = threading.Thread(target=run_session, args=(graph1, result1, "multiply"))
28t2 = threading.Thread(target=run_session, args=(graph2, result2, "add"))
29
30t1.start()
31t2.start()
32t1.join()
33t2.join()
34
35print(results)  # {'multiply': 6.0, 'add': 15.0}

Each session must have its own tf.Graph. Using the same graph across sessions without synchronization causes race conditions.

GPU Memory Configuration

By default, TensorFlow allocates all available GPU memory to the first session. Configure memory growth to allow multiple sessions to share the GPU:

python
1import tensorflow as tf
2
3# Option 1: Allow memory growth (allocate as needed)
4gpus = tf.config.experimental.list_physical_devices('GPU')
5for gpu in gpus:
6    tf.config.experimental.set_memory_growth(gpu, True)
7
8# Option 2: Limit memory per session (TF 1.x)
9import tensorflow.compat.v1 as tf1
10config = tf1.ConfigProto()
11config.gpu_options.per_process_gpu_memory_fraction = 0.4  # 40% of GPU memory
12session = tf1.Session(config=config)
13
14# Option 3: Set memory limit (TF 2.x)
15gpus = tf.config.experimental.list_physical_devices('GPU')
16if gpus:
17    tf.config.set_logical_device_configuration(
18        gpus[0],
19        [tf.config.LogicalDeviceConfiguration(memory_limit=2048)]  # 2 GB
20    )

TensorFlow 2.x: Concurrent Model Training

python
1import tensorflow as tf
2import threading
3
4def train_model(model_name, units, epochs):
5    model = tf.keras.Sequential([
6        tf.keras.layers.Dense(units, activation='relu', input_shape=(10,)),
7        tf.keras.layers.Dense(1)
8    ])
9    model.compile(optimizer='adam', loss='mse')
10
11    # Generate dummy data
12    import numpy as np
13    X = np.random.randn(1000, 10).astype('float32')
14    y = np.random.randn(1000, 1).astype('float32')
15
16    model.fit(X, y, epochs=epochs, verbose=0)
17    loss = model.evaluate(X, y, verbose=0)
18    print(f"{model_name}: loss = {loss:.4f}")
19
20# Train two models concurrently
21t1 = threading.Thread(target=train_model, args=("ModelA", 32, 10))
22t2 = threading.Thread(target=train_model, args=("ModelB", 64, 10))
23
24t1.start()
25t2.start()
26t1.join()
27t2.join()

Multiprocessing for True Parallelism

Python's GIL limits threading to one CPU thread at a time. For CPU-bound work, use multiprocessing:

python
1import multiprocessing as mp
2import tensorflow as tf
3import numpy as np
4
5def train_in_process(config):
6    # Each process gets its own TensorFlow runtime
7    model = tf.keras.Sequential([
8        tf.keras.layers.Dense(config['units'], activation='relu', input_shape=(10,)),
9        tf.keras.layers.Dense(1)
10    ])
11    model.compile(optimizer='adam', loss='mse')
12
13    X = np.random.randn(1000, 10).astype('float32')
14    y = np.random.randn(1000, 1).astype('float32')
15
16    model.fit(X, y, epochs=config['epochs'], verbose=0)
17    loss = model.evaluate(X, y, verbose=0)
18    return f"{config['name']}: loss = {loss:.4f}"
19
20if __name__ == '__main__':
21    configs = [
22        {'name': 'small', 'units': 16, 'epochs': 5},
23        {'name': 'medium', 'units': 64, 'epochs': 10},
24        {'name': 'large', 'units': 128, 'epochs': 15},
25    ]
26
27    with mp.Pool(3) as pool:
28        results = pool.map(train_in_process, configs)
29
30    for r in results:
31        print(r)

Each process has its own Python interpreter and TensorFlow runtime, avoiding GIL contention.

Using CUDA_VISIBLE_DEVICES

Assign specific GPUs to each process:

python
1import os
2import multiprocessing as mp
3
4def train_on_gpu(gpu_id, model_config):
5    os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)
6    import tensorflow as tf
7    # This process now only sees the assigned GPU
8    model = tf.keras.Sequential([...])
9    model.fit(...)
10
11if __name__ == '__main__':
12    p1 = mp.Process(target=train_on_gpu, args=(0, config_a))
13    p2 = mp.Process(target=train_on_gpu, args=(1, config_b))
14    p1.start()
15    p2.start()
16    p1.join()
17    p2.join()

Set CUDA_VISIBLE_DEVICES before importing TensorFlow in each process.

Common Pitfalls

  • Not configuring GPU memory growth: TensorFlow allocates all GPU memory by default. Two sessions on the same GPU without set_memory_growth(True) or per_process_gpu_memory_fraction causes the second to crash with CUDA_ERROR_OUT_OF_MEMORY.
  • Sharing a graph across threads without synchronization: In TF 1.x, two sessions running operations on the same graph concurrently can corrupt shared state. Each thread should use its own tf.Graph and tf.Session.
  • Setting CUDA_VISIBLE_DEVICES after importing TensorFlow: TensorFlow reads GPU configuration at import time. Setting the environment variable after import tensorflow has no effect. Set it before the import in each subprocess.
  • Using threading for CPU-bound training: Python's GIL serializes CPU-bound threads. For models that train primarily on CPU, use multiprocessing instead of threading for actual parallelism.
  • Ignoring TF 2.x eager execution: In TF 2.x, tf.Session is removed by default. Code that creates tf.Session objects directly will fail. Use tf.compat.v1.Session with disable_eager_execution() for legacy code, or restructure to use Keras model.fit() directly.

Summary

  • TF 1.x supports multiple concurrent sessions — each must have its own tf.Graph
  • TF 2.x uses eager execution — run concurrent models with threading or multiprocessing using Keras APIs
  • Configure GPU memory growth with set_memory_growth(True) to prevent OOM errors when sharing a GPU
  • Use multiprocessing for CPU-bound training to bypass Python's GIL
  • Assign GPUs to processes with CUDA_VISIBLE_DEVICES before importing TensorFlow
  • For distributed training across multiple GPUs, prefer tf.distribute.MirroredStrategy over manual session management
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.