Introduction

In TensorFlow eager mode, computing a gradient between two layers means tracking the forward pass with tf.GradientTape and then asking for the gradient of a later tensor with respect to an earlier tensor. In TensorFlow 2, eager execution is already the default, so the main job is making sure both tensors are produced inside the tape context and that the earlier tensor is being watched if needed.

Build a Simple Sequential Model

Start with a small sequential network:

1import tensorflow as tf
2from tensorflow import keras
3
4model = keras.Sequential([
5    keras.layers.Dense(8, activation="relu"),
6    keras.layers.Dense(4, activation="relu"),
7    keras.layers.Dense(1)
8])

Suppose you want the gradient of the output of the second dense layer with respect to the output of the first dense layer.

Capture Intermediate Layer Outputs

The easiest way is to run the layers manually inside a GradientTape.

1import tensorflow as tf
2
3x = tf.random.normal((1, 5))
4
5layer1 = model.layers[0]
6layer2 = model.layers[1]
7
8with tf.GradientTape() as tape:
9    h1 = layer1(x)
10    tape.watch(h1)
11    h2 = layer2(h1)
12
13grad = tape.gradient(h2, h1)
14
15print(h1.shape)
16print(h2.shape)
17print(grad)

Here:

• 'h1 is the tensor between layer 1 and layer 2'
• 'h2 is the output of the next layer'
• 'tape.gradient(h2, h1) computes how h2 changes with respect to h1'

Why tape.watch Matters

TensorFlow automatically watches trainable variables, but it does not automatically watch every intermediate tensor for gradient targets in the way people sometimes expect.

Using:

tape.watch(h1)

ensures that h1 is tracked as a differentiable source for the gradient request.

If you omit it and later get None, that is one of the first things to check.

Compute the Gradient of a Scalar Loss

In practice, gradients are often taken from a scalar quantity such as a loss. If the later tensor is not scalar, TensorFlow effectively sums gradients over its components unless you reduce it explicitly.

1with tf.GradientTape() as tape:
2    h1 = layer1(x)
3    tape.watch(h1)
4    h2 = layer2(h1)
5    loss = tf.reduce_sum(h2)
6
7grad = tape.gradient(loss, h1)
8print(grad)

This is often clearer because the target of differentiation is explicitly a scalar value.

Use an Intermediate Model When Helpful

Another approach is to build a model that exposes intermediate outputs, then compute gradients from those tensors.

1intermediate_model = keras.Model(
2    inputs=model.input,
3    outputs=[model.layers[0].output, model.layers[1].output]
4)

That pattern is useful when you need repeated access to internal activations, but for one-off gradient calculations the manual layer-by-layer forward pass is often simpler and easier to reason about.

Common Pitfalls

The most common problem is asking for a gradient of a tensor that was created outside the tape context. If the tape did not see the forward pass, it cannot differentiate it.

Another issue is forgetting to watch the intermediate tensor when using it as the gradient source.

Developers also sometimes confuse gradients with respect to layer outputs and gradients with respect to trainable weights. Those are different requests:

• output-to-output uses tensors such as h2 and h1
• training uses variables such as layer1.kernel

Finally, if a non-differentiable operation sits between the two tensors, the gradient may be None even if the tape setup is correct.

Summary

• In TensorFlow 2, eager execution is already on by default.
• Use tf.GradientTape and compute both intermediate tensors inside the tape context.
• Watch the earlier tensor if you want gradients with respect to it.
• 'tape.gradient(later_tensor, earlier_tensor) gives the local sensitivity between layers.'
• If the gradient is None, check tape scope, watch, and whether the path is differentiable.