Introduction

TensorFlow optimizers update whole variables, not arbitrary slices inside var_list. If you want to minimize a loss with respect to only some elements of a variable, the usual solutions are to mask the gradient for the frozen elements or to model the trainable subset as a separate variable and rebuild the full tensor from it.

Why var_list Does Not Solve This

Optimizer APIs such as apply_gradients() and older minimize() interfaces expect a list of variables. They can include or exclude entire variables, but they do not let you say “only optimize elements 0 and 3 of this one tensor.”

That means code like this does not exist as a direct built-in feature:

# Conceptual only, not real TensorFlow API
optimizer.minimize(loss, var_list=[weights[0], weights[3]])

Slices of a variable are tensors, not independent trainable variables. To freeze part of a variable, you need to control the gradient or the parameterization.

Mask the Gradient for Frozen Elements

The most direct approach is to compute the gradient for the whole variable and then zero out the positions you do not want to update.

1import tensorflow as tf
2
3w = tf.Variable([1.0, 2.0, 3.0, 4.0], dtype=tf.float32)
4target = tf.constant([0.0, 0.0, 0.0, 0.0], dtype=tf.float32)
5mask = tf.constant([1.0, 0.0, 1.0, 0.0], dtype=tf.float32)
6
7optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)
8
9for step in range(5):
10    with tf.GradientTape() as tape:
11        loss = tf.reduce_sum(tf.square(w - target))
12
13    grad = tape.gradient(loss, w)
14    masked_grad = grad * mask
15    optimizer.apply_gradients([(masked_grad, w)])
16
17    print(f"step={step} w={w.numpy()} loss={loss.numpy():.4f}")

Here, only positions with a 1.0 in the mask move. The other elements keep their previous values because their effective gradient is zero.

This is often the simplest answer when the frozen pattern is fixed.

When Masking Is a Good Fit

Gradient masking works well when:

• the trainable positions are known ahead of time
• you want to keep the variable stored as one tensor
• the optimizer state for the frozen entries does not cause conceptual problems

It is especially convenient for experiments, constrained optimization, or fine-tuning only a subset of coefficients.

However, if the trainable structure is more complex, a separate-variable design can be cleaner.

Use a Separate Trainable Variable for the Mutable Part

If only part of the tensor should ever change, it is often better to represent that part explicitly as its own variable and reconstruct the full tensor inside the loss.

1import tensorflow as tf
2
3base = tf.constant([2.0, 4.0], dtype=tf.float32)
4trainable_part = tf.Variable([1.0, 3.0], dtype=tf.float32)
5indices = tf.constant([[0], [2]], dtype=tf.int32)
6
7optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)
8
9def build_full_vector():
10    initial = tf.constant([0.0, 2.0, 0.0, 4.0], dtype=tf.float32)
11    return tf.tensor_scatter_nd_update(initial, indices, trainable_part)
12
13for step in range(5):
14    with tf.GradientTape() as tape:
15        full_vector = build_full_vector()
16        loss = tf.reduce_sum(tf.square(full_vector))
17
18    grad = tape.gradient(loss, trainable_part)
19    optimizer.apply_gradients([(grad, trainable_part)])
20
21    print(full_vector.numpy(), float(loss))

This pattern makes the trainable subset explicit and often leads to cleaner reasoning about what is actually being optimized.

Which Pattern Should You Prefer

A useful rule is:

• mask gradients when the tensor should stay structurally whole
• split variables when the trainable subset is conceptually separate

Masking is minimal and practical. Splitting variables is often better when frozen versus trainable parts have different meaning in the model.

Be Aware of Optimizer State

With optimizers such as Adam, the optimizer may still maintain slot state for the whole variable even if some entries always receive zero gradients. That is normally fine, but it is another reason the separate-variable design can be cleaner when the frozen subset is permanent and significant.

If you care deeply about optimizer state, explicit variable separation usually reflects the intent better than masking.

Common Pitfalls

• Expecting var_list to accept slices of a variable as if they were standalone trainable variables.
• Zeroing values in the tensor itself instead of zeroing the gradient, which changes the model rather than the update rule.
• Freezing elements conceptually but still mixing trainable and fixed parts in one variable when a separate-variable design would be clearer.
• Forgetting that masked entries still belong to the same variable object and optimizer bookkeeping.
• Applying the mask with the wrong shape or dtype and silently updating the wrong positions.

Summary

• TensorFlow optimizers work at the variable level, not at arbitrary slice level.
• To update only some elements, either mask the gradient or represent the trainable subset as its own variable.
• Gradient masking is the simplest approach when the tensor should remain a single variable.
• Separate variables are often cleaner when frozen and trainable parts have different roles.
• The right solution depends on whether you want a convenient patch or a model structure that reflects the constraint explicitly.