Introduction

In TensorFlow 1.x, tf.control_dependencies() is a context manager that forces operations defined inside it to execute only after specified dependency operations have completed. TensorFlow's dataflow graph normally executes operations in any order that satisfies data dependencies, but some operations — like variable assignments, metric updates, and logging — have no data dependency on each other yet must execute in a specific order. control_dependencies adds explicit execution ordering constraints to the graph without creating data flow edges.

Basic Usage

1import tensorflow as tf
2
3# TF1 style (or tf.compat.v1 in TF2)
4tf.compat.v1.disable_eager_execution()
5
6x = tf.Variable(0)
7assign_op = x.assign(5)
8
9# Without control_dependencies, this read may happen BEFORE the assign
10# With control_dependencies, the read waits until assign_op completes
11with tf.control_dependencies([assign_op]):
12    y = x + 1  # guaranteed to see x = 5, so y = 6
13
14with tf.compat.v1.Session() as sess:
15    sess.run(tf.compat.v1.global_variables_initializer())
16    result = sess.run(y)
17    print(result)  # 6

Operations created inside the with tf.control_dependencies([ops]) block will not execute until all operations in the dependency list have finished. The dependency is on execution, not on the return value.

Ensuring Variable Updates Before Reading

1import tensorflow as tf
2tf.compat.v1.disable_eager_execution()
3
4counter = tf.Variable(0, name="counter")
5increment = counter.assign_add(1)
6
7# Without control_dependencies, read_counter might return 0
8with tf.control_dependencies([increment]):
9    read_counter = tf.identity(counter)  # waits for increment to finish
10
11with tf.compat.v1.Session() as sess:
12    sess.run(tf.compat.v1.global_variables_initializer())
13    for _ in range(5):
14        val = sess.run(read_counter)
15        print(val)  # 1, 2, 3, 4, 5

tf.identity() is a common pattern — it creates a new operation (a copy) inside the control dependency block, ensuring the dependency is enforced. Without it, there is no new operation to attach the dependency to.

Training Loop: Apply Gradients After Update Ops

1import tensorflow as tf
2tf.compat.v1.disable_eager_execution()
3
4# Batch normalization has update ops for moving mean/variance
5x = tf.compat.v1.placeholder(tf.float32, [None, 10])
6training = tf.compat.v1.placeholder(tf.bool)
7
8net = tf.compat.v1.layers.batch_normalization(x, training=training)
9logits = tf.compat.v1.layers.dense(net, 2)
10
11loss = tf.reduce_mean(logits)
12optimizer = tf.compat.v1.train.AdamOptimizer()
13
14# Get batch norm update ops
15update_ops = tf.compat.v1.get_collection(tf.compat.v1.GraphKeys.UPDATE_OPS)
16
17# Ensure update_ops run before the train step
18with tf.control_dependencies(update_ops):
19    train_op = optimizer.minimize(loss)
20
21# Now sess.run(train_op) also runs batch norm updates

This is the most common real-world use of control_dependencies. Batch normalization update operations must run each training step, but they have no data dependency on the optimizer. Without control_dependencies, the moving mean and variance would never update.

Multiple Dependencies

1import tensorflow as tf
2tf.compat.v1.disable_eager_execution()
3
4a = tf.Variable(1.0)
5b = tf.Variable(2.0)
6
7update_a = a.assign(10.0)
8update_b = b.assign(20.0)
9
10# Both updates must complete before computing the sum
11with tf.control_dependencies([update_a, update_b]):
12    result = tf.add(a, b)  # guaranteed to see a=10, b=20
13
14with tf.compat.v1.Session() as sess:
15    sess.run(tf.compat.v1.global_variables_initializer())
16    print(sess.run(result))  # 30.0

You can pass multiple operations to the dependency list. All of them must complete before any operation inside the block executes.

TF2 Equivalent: Eager Execution

1import tensorflow as tf
2
3# In TF2 with eager execution, operations run immediately in order
4# No control_dependencies needed
5
6counter = tf.Variable(0)
7counter.assign_add(1)  # runs immediately
8print(counter.numpy())  # 1 — always sees the update
9
10# If using tf.function (graph mode in TF2):
11@tf.function
12def train_step(model, x, y, optimizer):
13    with tf.GradientTape() as tape:
14        predictions = model(x, training=True)
15        loss = tf.reduce_mean(tf.keras.losses.mse(y, predictions))
16    gradients = tape.gradient(loss, model.trainable_variables)
17    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
18    # In TF2 Keras, batch norm updates are handled automatically
19    return loss

TF2's eager execution mode runs operations sequentially, making control_dependencies unnecessary for most code. Inside @tf.function, TensorFlow traces a graph but handles common patterns like batch normalization updates automatically through Keras.

Nesting Control Dependencies

1import tensorflow as tf
2tf.compat.v1.disable_eager_execution()
3
4a = tf.Variable(0)
5b = tf.Variable(0)
6c = tf.Variable(0)
7
8op_a = a.assign(1)
9
10with tf.control_dependencies([op_a]):
11    op_b = b.assign(a + 1)  # waits for op_a, so b = 2
12
13    with tf.control_dependencies([op_b]):
14        op_c = c.assign(b + 1)  # waits for op_b, so c = 3
15
16with tf.compat.v1.Session() as sess:
17    sess.run(tf.compat.v1.global_variables_initializer())
18    sess.run(op_c)
19    print(sess.run([a, b, c]))  # [1, 2, 3]

Nested control_dependencies blocks create a chain: op_a must finish before op_b, and op_b must finish before op_c.

Common Pitfalls

• Not creating a new op inside the block: control_dependencies only applies to operations created inside the with block. If you reference an existing tensor without creating a new operation, the dependency is not enforced. Use tf.identity(tensor) to create a new op that carries the dependency.
• Assuming control_dependencies creates data flow: Control dependencies only enforce execution order. They do not pass data between operations. If operation B reads from a variable that operation A writes, you still need a control dependency to guarantee A completes before B reads.
• Using control_dependencies in TF2 eager mode: In eager mode, operations execute immediately in Python order, so control_dependencies has no effect. It only works in graph mode (tf.compat.v1 or inside @tf.function with explicit graph construction).
• Forgetting batch norm update ops: The most common bug is training a model with batch normalization without using control_dependencies (TF1) or model(x, training=True) (TF2) to ensure moving statistics are updated each step. The model trains but performs poorly at inference.
• Overusing control_dependencies and reducing parallelism: Adding unnecessary dependencies forces sequential execution of operations that could otherwise run in parallel on GPU. Only add dependencies where execution order genuinely matters.

Summary

• tf.control_dependencies([ops]) forces operations inside the block to wait for the listed ops to complete
• Most common use: ensuring batch normalization update ops run during training
• Use tf.identity() inside the block to create a new operation that carries the dependency
• In TF2 eager mode, operations run sequentially by default — control_dependencies is not needed
• Inside @tf.function, Keras handles update ops automatically
• Avoid overuse — unnecessary dependencies reduce GPU parallelism