Introduction

In Keras, “adaptive learning rate” can refer to two related but different ideas. It can mean using an optimizer such as Adam that adapts updates internally, or it can mean changing the learning rate explicitly over time with a schedule or callback.

Adaptive Optimizers Are the Simplest Option

Many projects start with an optimizer that already adapts update magnitudes based on gradient history. Adam is the most common example.

1import tensorflow as tf
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Input(shape=(20,)),
5    tf.keras.layers.Dense(64, activation="relu"),
6    tf.keras.layers.Dense(1),
7])
8
9optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
10
11model.compile(
12    optimizer=optimizer,
13    loss="mse",
14    metrics=["mae"],
15)

This setup is adaptive, but the initial learning rate still matters. Adam is not a magic escape hatch for a badly chosen base learning rate.

Use a Learning Rate Schedule for Predictable Changes

If you want the learning rate to follow a known pattern over training steps, use a schedule object. That gives you repeatable behavior from run to run.

1import tensorflow as tf
2
3schedule = tf.keras.optimizers.schedules.ExponentialDecay(
4    initial_learning_rate=1e-3,
5    decay_steps=1000,
6    decay_rate=0.96,
7    staircase=True,
8)
9
10optimizer = tf.keras.optimizers.Adam(learning_rate=schedule)

This approach is useful when you already know the training recipe you want, such as a gradual decay every few hundred or thousand update steps.

You can use other schedules the same way, including piecewise or cosine decay patterns, as long as the training behavior is intentional and measurable.

Use a Callback When Validation Should Drive the Change

Sometimes you do not know the best schedule in advance. In that case, a callback such as ReduceLROnPlateau can lower the learning rate when a monitored metric stops improving.

1import tensorflow as tf
2
3reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(
4    monitor="val_loss",
5    factor=0.5,
6    patience=3,
7    min_lr=1e-6,
8    verbose=1,
9)
10
11history = model.fit(
12    x_train,
13    y_train,
14    validation_data=(x_val, y_val),
15    epochs=20,
16    callbacks=[reduce_lr],
17)

This is often a good choice when training curves are noisy and you want the model to back off automatically once progress stalls.

Piecewise and Custom Schedules

For training recipes with distinct phases, a piecewise schedule is easy to reason about:

1import tensorflow as tf
2
3schedule = tf.keras.optimizers.schedules.PiecewiseConstantDecay(
4    boundaries=[1000, 3000],
5    values=[1e-3, 5e-4, 1e-4],
6)
7
8optimizer = tf.keras.optimizers.SGD(
9    learning_rate=schedule,
10    momentum=0.9,
11)

If you want complete control, write a callback that updates the optimizer at chosen epochs:

1import tensorflow as tf
2
3class CustomLRScheduler(tf.keras.callbacks.Callback):
4    def on_epoch_begin(self, epoch, logs=None):
5        if epoch == 5:
6            self.model.optimizer.learning_rate.assign(5e-4)
7        elif epoch == 10:
8            self.model.optimizer.learning_rate.assign(1e-4)

This is useful when the learning-rate rule depends on milestones that are easier to express directly than through a built-in schedule object.

Log the Learning Rate During Training

If the learning rate can change during training, log it. Otherwise it is hard to tell whether the schedule or callback actually behaved as expected.

1import tensorflow as tf
2
3class LogLearningRate(tf.keras.callbacks.Callback):
4    def on_epoch_end(self, epoch, logs=None):
5        lr = self.model.optimizer.learning_rate
6
7        if callable(lr):
8            value = lr(self.model.optimizer.iterations).numpy()
9        else:
10            value = float(lr.numpy())
11
12        print(f"epoch={epoch} lr={value:.6f}")

Observability matters because learning-rate bugs often look like ordinary training instability.

Choosing the Right Strategy

A practical way to choose is:

• use Adam or RMSprop when you want a strong default quickly,
• use an explicit schedule when you need a reproducible training recipe,
• use ReduceLROnPlateau when validation behavior should trigger adaptation.

You can combine an adaptive optimizer with a schedule, but do it deliberately. Too many overlapping adaptation rules can make model behavior harder to reason about.

Common Pitfalls

• Treating adaptive optimizers and explicit learning-rate schedules as if they were the same thing.
• Starting with a base learning rate that is too large and expecting Adam or RMSprop to compensate.
• Using ReduceLROnPlateau without a meaningful validation metric.
• Combining several learning-rate mechanisms without tracking which one is actually driving the updates.
• Failing to log the effective learning rate and then debugging blind when training stalls or diverges.

Summary

• In Keras, adaptive learning rate can mean either adaptive optimizers or explicit rate changes over time.
• Adam is an easy adaptive default, but its base learning rate still matters.
• Schedules are useful when you want deterministic learning-rate changes during training.
• 'ReduceLROnPlateau is useful when validation metrics should decide when to reduce the rate.'
• Logging the effective learning rate makes training behavior much easier to debug.