Introduction

In TensorFlow 1 graph mode, tf.GraphKeys.TRAINABLE_VARIABLES and tf.GraphKeys.UPDATE_OPS refer to very different things. TRAINABLE_VARIABLES holds variables that optimizers are expected to update through gradient descent, while UPDATE_OPS holds operations that should run during training but are not gradient-based parameter updates. The classic example is batch normalization moving statistics.

TRAINABLE_VARIABLES: Parameters Learned by the Optimizer

This collection contains variables marked as trainable. These are typically the weights and biases of your model.

Examples include:

• dense layer kernels
• convolution filters
• bias vectors
• trainable embeddings

A TensorFlow 1 style example:

1import tensorflow as tf
2
3tf.compat.v1.disable_eager_execution()
4
5w = tf.Variable(tf.random.normal([10, 1]), name='w')
6b = tf.Variable(tf.zeros([1]), name='b')
7
8print(tf.compat.v1.get_collection(tf.compat.v1.GraphKeys.TRAINABLE_VARIABLES))

When you call an optimizer's minimize method, TensorFlow computes gradients with respect to these trainable variables and applies updates to them.

UPDATE_OPS: Extra State Updates That Must Also Run

UPDATE_OPS contains graph operations that update internal state but are not themselves trainable variables.

The most famous example is batch normalization. During training, batch norm updates moving averages such as moving mean and moving variance. Those updates are not gradient descent on trainable weights. They are side-effect operations that need to be executed during training.

A simplified graph-mode example:

1import tensorflow as tf
2
3tf.compat.v1.disable_eager_execution()
4
5x = tf.compat.v1.placeholder(tf.float32, shape=[None, 4])
6training = tf.compat.v1.placeholder(tf.bool)
7
8y = tf.compat.v1.layers.batch_normalization(x, training=training)
9
10update_ops = tf.compat.v1.get_collection(tf.compat.v1.GraphKeys.UPDATE_OPS)
11print(update_ops)

Those update ops are usually empty in very simple models and non-empty once layers with internal moving statistics are added.

Why the Difference Matters During Training

If you only minimize the loss and forget to run UPDATE_OPS, batch normalization and similar layers may not update their internal state correctly. The trainable weights still change, but the moving statistics stay stale.

That leads to training or inference behavior that can look mysteriously wrong.

The classic TensorFlow 1 training pattern is:

1import tensorflow as tf
2
3tf.compat.v1.disable_eager_execution()
4
5x = tf.compat.v1.placeholder(tf.float32, shape=[None, 4])
6y_true = tf.compat.v1.placeholder(tf.float32, shape=[None, 1])
7training = tf.compat.v1.placeholder(tf.bool)
8
9h = tf.compat.v1.layers.dense(x, 8, activation=tf.nn.relu)
10h = tf.compat.v1.layers.batch_normalization(h, training=training)
11y_pred = tf.compat.v1.layers.dense(h, 1)
12
13loss = tf.reduce_mean(tf.square(y_true - y_pred))
14optimizer = tf.compat.v1.train.AdamOptimizer(0.001)
15update_ops = tf.compat.v1.get_collection(tf.compat.v1.GraphKeys.UPDATE_OPS)
16
17with tf.control_dependencies(update_ops):
18    train_op = optimizer.minimize(loss)

That control_dependencies block ensures the update ops run whenever train_op runs.

Mental Model

A useful mental model is:

• 'TRAINABLE_VARIABLES are things the optimizer learns'
• 'UPDATE_OPS are side effects that training should execute'

They are related to training, but they do not represent the same kind of graph entity.

One is a collection of variables.

The other is a collection of operations.

TensorFlow 2 Note

This distinction is most visible in TensorFlow 1 graph-mode code. In TensorFlow 2 and modern Keras usage, many of these details are managed automatically by the higher-level training loop, so developers see GraphKeys much less often.

That does not make the concept unimportant. It just means modern APIs hide more of the explicit graph bookkeeping.

Common Pitfalls

A common mistake is assuming UPDATE_OPS are extra trainable parameters. They are not; they are operations.

Another mistake is collecting update ops but never attaching them to the training step in TensorFlow 1 graph mode.

People also sometimes expect every layer to contribute to UPDATE_OPS. Many layers do not. The collection becomes relevant only for layers that maintain extra internal state.

Finally, do not read TensorFlow 1 GraphKeys code as if it were the normal TensorFlow 2 style. The programming model is different.

Summary

• In TensorFlow 1 graph mode, TRAINABLE_VARIABLES and UPDATE_OPS serve different training roles
• 'TRAINABLE_VARIABLES are the optimizer-updated model parameters'
• 'UPDATE_OPS are state-update operations such as batch norm moving-average updates'
• A common TensorFlow 1 training pattern is to run UPDATE_OPS via tf.control_dependencies
• Forgetting UPDATE_OPS can break layers that depend on internal state updates during training
• In TensorFlow 2, these details are often handled automatically, so the collection names appear less often in everyday code