Introduction

Gradient clipping prevents exploding gradients during RNN training. TensorFlow provides two main approaches: tf.clip_by_value clips each gradient element independently to a fixed range, while tf.clip_by_global_norm scales all gradients proportionally so their combined norm stays below a threshold. Global norm clipping is preferred for RNNs because it preserves the relative direction of gradients across layers, whereas value clipping can distort the gradient direction and harm training.

tf.clip_by_value: Element-Wise Clipping

tf.clip_by_value clamps every individual gradient value to a [min, max] range:

1import tensorflow as tf
2
3gradients = [tf.constant([-10.0, 0.5, 100.0, -0.1])]
4
5clipped = [tf.clip_by_value(g, clip_value_min=-1.0, clip_value_max=1.0) for g in gradients]
6print(clipped[0].numpy())
7# [-1.0, 0.5, 1.0, -0.1]

Each element is clipped independently. The value 100.0 becomes 1.0 and -10.0 becomes -1.0, but 0.5 and -0.1 are untouched.

Usage in Training

1optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
2
3with tf.GradientTape() as tape:
4    loss = model(inputs, training=True)
5
6gradients = tape.gradient(loss, model.trainable_variables)
7clipped_gradients = [tf.clip_by_value(g, -1.0, 1.0) for g in gradients]
8optimizer.apply_gradients(zip(clipped_gradients, model.trainable_variables))

tf.clip_by_global_norm: Proportional Scaling

tf.clip_by_global_norm computes the global L2 norm across ALL gradients and scales them down proportionally if the norm exceeds the threshold:

1gradients = [tf.constant([3.0, 4.0]), tf.constant([0.0, 0.0, 5.0])]
2
3# Global norm = sqrt(3^2 + 4^2 + 0^2 + 0^2 + 5^2) = sqrt(50) ≈ 7.07
4clipped, global_norm = tf.clip_by_global_norm(gradients, clip_norm=5.0)
5
6print(f"Original norm: {global_norm.numpy():.2f}")  # 7.07
7print(clipped[0].numpy())  # [2.12, 2.83] — scaled by 5.0/7.07
8print(clipped[1].numpy())  # [0.0, 0.0, 3.54]

The scaling factor is min(1, clip_norm / global_norm). When the global norm is within the threshold, gradients are unchanged. When it exceeds the threshold, all gradients are multiplied by the same factor, preserving their relative magnitudes and directions.

Usage in Training

1optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
2
3with tf.GradientTape() as tape:
4    loss = model(inputs, training=True)
5
6gradients = tape.gradient(loss, model.trainable_variables)
7clipped_gradients, _ = tf.clip_by_global_norm(gradients, clip_norm=5.0)
8optimizer.apply_gradients(zip(clipped_gradients, model.trainable_variables))

Why Global Norm Is Better for RNNs

RNNs unroll across many timesteps, creating deep computation graphs where gradients flow through repeated matrix multiplications. This makes them prone to exploding gradients where some gradient components become enormous while others stay small.

Value clipping:     gradient = [0.001, 0.5, 1000.0] → [-1.0, 0.5, 1.0]
Global norm clip:   gradient = [0.001, 0.5, 1000.0] → [0.000001, 0.0005, 1.0]

Value clipping distorts the gradient direction — the tiny component (0.001) is left unchanged while the large component (1000.0) is crushed to 1.0. The gradient now points in a fundamentally different direction.

Global norm clipping preserves the direction by scaling everything proportionally. The relative relationship between components is maintained, leading to more stable training.

Choosing the Clip Value

For tf.clip_by_global_norm (recommended starting values)

1# Monitor gradient norms to choose a good threshold
2with tf.GradientTape() as tape:
3    loss = model(inputs, training=True)
4
5gradients = tape.gradient(loss, model.trainable_variables)
6global_norm = tf.linalg.global_norm(gradients)
7print(f"Gradient norm: {global_norm.numpy()}")
8
9# Run a few batches and observe the norm distribution
10# Set clip_norm to roughly the median or 90th percentile

For tf.clip_by_value

A common choice is [-1.0, 1.0] or [-5.0, 5.0]. These are less principled than global norm thresholds because the appropriate range depends on every layer's gradient scale.

Keras Built-In Clipnorm

Keras optimizers support gradient clipping directly:

1# Global norm clipping (recommended)
2optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, clipnorm=1.0)
3
4# Value clipping
5optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, clipvalue=0.5)
6
7# No manual gradient manipulation needed — compile and fit
8model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy')
9model.fit(train_data, epochs=10)

clipnorm uses global norm clipping internally. clipvalue uses value clipping.

Side-by-Side Comparison

Common Pitfalls

• Using clip_by_value for RNNs: Value clipping distorts gradient direction, causing training instability. Always prefer global norm clipping for recurrent architectures.
• Setting clip_norm too low: Aggressively clipping (e.g., clip_norm=0.1) effectively reduces the learning rate to near zero, causing extremely slow training. Monitor gradient norms first.
• Setting clip_norm too high: A threshold that never triggers (e.g., clip_norm=1000) provides no protection against exploding gradients. The threshold should be near the typical gradient norm.
• Clipping before vs after optimizer state updates: When using Adam or RMSProp, the optimizer uses raw gradients for its moment estimates. Clipping before apply_gradients means the optimizer sees clipped gradients, which is the standard approach. Clipping inside a custom training step at the wrong point can corrupt optimizer state.
• Forgetting None gradients: tape.gradient() returns None for variables not connected to the loss. Passing None to tf.clip_by_global_norm raises an error. Filter them out: [(g, v) for g, v in zip(grads, vars) if g is not None].

Summary

• tf.clip_by_value clips each gradient element to a fixed range — simple but distorts direction
• tf.clip_by_global_norm scales all gradients proportionally to keep the global norm below a threshold — preserves direction
• Use global norm clipping for RNNs, LSTMs, GRUs, and Transformers
• Start with clip_norm=1.0 to 5.0 and monitor gradient norms to tune
• Use clipnorm in Keras optimizers for the simplest integration