Introduction

In deep learning, the optimizer adjusts the weights of the neural network to minimize error. tf.train.AdamOptimizer, a variant of stochastic gradient descent, is widely appreciated for its ability to adaptively adjust the learning rate throughout training. However, there are instances when manually tweaking the learning rate is preferred to achieve smoother convergence or to respond to dynamic changes in the training environment.

Understanding the Learning Rate in AdamOptimizer

The learning rate is a hyperparameter that determines the size of the step taken during the optimization process. The tf.train.AdamOptimizer uses the following update rule for a weight parameter $w$:

$$m_t = \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t$$

$$v_t = \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2$$

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$$

$$\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

$$w_{t+1} = w_t - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$

Where:

• $m_t$ and $v_t$ are moving averages of the gradient and squared gradient, respectively.
• $g_t$ is the gradient of the loss with respect to $w_t$.
• $\eta$ is the learning rate.
• $\beta_1$ and $\beta_2$ are decay rates for the moment estimates (defaults: 0.9 and 0.999).
• $\epsilon$ is a small constant for numerical stability (default: $10^{-8}$).

Because Adam already adapts per-parameter learning rates through the moment estimates, the global $\eta$ acts as a scaling factor. Manually adjusting $\eta$ during training gives you coarse control over the overall step size while Adam handles fine-grained per-parameter adaptation.

Why Adjust Manually?

1. Dynamic Environments: In scenarios where data distribution changes, a static learning rate may not be ideal.
2. Learning Rate Schedules: Employing methods such as learning rate decay can help in converging faster. Common schedules include:
   • Step Decay: Reduce learning rate by a factor after a certain number of epochs.
   • Exponential Decay: Gradually reduce the learning rate exponentially.
   • Cosine Annealing: Adjust the learning rate following a cosine curve, often with warm restarts.
3. Fine-Tuning: When applying transfer learning, a smaller learning rate prevents large updates that could degrade pre-learned features.

How to Change Learning Rate Manually

Method 1: Using TensorFlow's Built-in Decay Functions

TensorFlow provides several utility functions for learning rate schedules:

1import tensorflow as tf
2
3# Exponential decay
4global_step = tf.Variable(0, trainable=False)
5initial_lr = 0.001
6decay_steps = 1000
7decay_rate = 0.96
8
9learning_rate = tf.train.exponential_decay(
10    initial_lr, global_step, decay_steps, decay_rate, staircase=True
11)
12
13optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
14train_op = optimizer.minimize(loss, global_step=global_step)

Method 2: Using a Placeholder (TensorFlow 1.x)

A placeholder allows you to feed a different learning rate at each training step:

1import tensorflow as tf
2
3lr_placeholder = tf.placeholder(tf.float32, shape=[])
4optimizer = tf.train.AdamOptimizer(learning_rate=lr_placeholder)
5train_op = optimizer.minimize(loss)
6
7with tf.Session() as sess:
8    sess.run(tf.global_variables_initializer())
9    for epoch in range(num_epochs):
10        current_lr = initial_lr * (0.95 ** epoch)
11        sess.run(train_op, feed_dict={lr_placeholder: current_lr})

Method 3: Using tf.Variable (TensorFlow 1.x and 2.x)

You can create a tf.Variable for the learning rate and update it during training:

1import tensorflow as tf
2
3lr_var = tf.Variable(0.001, trainable=False, dtype=tf.float32)
4optimizer = tf.train.AdamOptimizer(learning_rate=lr_var)
5train_op = optimizer.minimize(loss)
6
7# Later, update the learning rate
8new_lr_op = lr_var.assign(0.0001)
9
10with tf.Session() as sess:
11    sess.run(tf.global_variables_initializer())
12    # Train for a while at lr=0.001
13    for step in range(5000):
14        sess.run(train_op)
15    # Reduce learning rate
16    sess.run(new_lr_op)
17    # Continue training at lr=0.0001
18    for step in range(5000):
19        sess.run(train_op)

Method 4: Keras Callbacks (TensorFlow 2.x)

In TensorFlow 2.x with Keras, the LearningRateScheduler or ReduceLROnPlateau callbacks provide clean interfaces:

1import tensorflow as tf
2
3def lr_schedule(epoch, lr):
4    if epoch > 10:
5        return lr * 0.9
6    return lr
7
8model.compile(
9    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
10    loss='categorical_crossentropy'
11)
12
13model.fit(
14    x_train, y_train,
15    epochs=50,
16    callbacks=[tf.keras.callbacks.LearningRateScheduler(lr_schedule)]
17)

Important Caveat: Adam's Internal State

When you change the learning rate of an Adam optimizer mid-training, the internal momentum variables ($m_t$ and $v_t$) are not reset. These accumulated statistics were computed under the old learning rate. In most cases this is fine since Adam's adaptive nature absorbs the change gracefully. However, if you make a large jump in learning rate (for example, warm restarts), be aware that the optimizer's state may cause initially unexpected behavior until the moments adjust to the new regime.

Summary

Manually adjusting the learning rate in tf.train.AdamOptimizer is a powerful technique for controlling training dynamics. The best method depends on your TensorFlow version and how much flexibility you need. For most modern workflows, Keras callbacks with ReduceLROnPlateau or LearningRateScheduler provide the cleanest approach.