Introduction

Adam is one of TensorFlow's most commonly used optimizers, but the confusion around it is predictable: it adapts per-parameter step sizes, yet it still has a global learning rate, momentum-style moving averages, and several stability knobs. That makes it easy to treat Adam as “automatic SGD that always works,” which is not what it actually is. The right mental model is that Adam is still gradient descent, just with extra state that changes how each parameter update is scaled.

What Adam Actually Tracks

Adam maintains two moving averages for each trainable parameter:

• the first moment, which tracks average gradient direction
• the second moment, which tracks average squared gradient magnitude

In simplified notation:

• 'm_t tracks the gradient mean'
• 'v_t tracks the gradient square mean'

These moving averages are bias-corrected in early steps, and the update uses them to scale the effective step size.

That is why Adam often converges faster than plain SGD in messy optimization landscapes: parameters with consistently large gradients get smaller effective updates, while parameters with smaller gradients can still move meaningfully.

The TensorFlow API Is Simpler Than the Theory Suggests

In TensorFlow or Keras, you normally instantiate Adam with a few hyperparameters and let the optimizer manage the internal moving averages.

1import tensorflow as tf
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Dense(32, activation="relu", input_shape=(10,)),
5    tf.keras.layers.Dense(1)
6])
7
8model.compile(
9    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
10    loss="mse"
11)

That is enough for many models. But “Adam worked with the default settings once” should not be confused with “Adam requires no tuning.”

Adam Still Has a Learning Rate

A very common misconception is that because Adam adapts updates automatically, the learning rate no longer matters. It still matters a lot.

The learning rate in Adam is still the top-level scale for parameter updates. If it is too large, training can oscillate or diverge. If it is too small, training can stall. Adam's adaptive scaling makes the optimizer more forgiving than plain SGD in many cases, but it does not eliminate the learning-rate problem.

A quick experiment with different learning rates makes this clear:

for lr in [1e-2, 1e-3, 1e-4]:
    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
    print(optimizer.learning_rate.numpy())

The difference between those values is often more important than small changes to other Adam settings.

beta_1, beta_2, and epsilon Are Not Random Magic Numbers

Adam's other important parameters are:

• 'beta_1: decay for the first moment estimate'
• 'beta_2: decay for the second moment estimate'
• 'epsilon: numerical stability term in the denominator'

Most of the time, the defaults are fine. But they are not sacred. If training becomes unstable, extremely noisy, or sensitive to small gradients, these settings can matter.

For example:

1optimizer = tf.keras.optimizers.Adam(
2    learning_rate=1e-3,
3    beta_1=0.9,
4    beta_2=0.999,
5    epsilon=1e-7
6)

The key point is that Adam's behavior comes from the combination of these values, not from the name “Adam” alone.

Adam Is Not Always the Best Final Optimizer

Another source of confusion is performance versus generalization. Adam often improves optimization speed, especially early in training, but that does not guarantee the best final validation or test performance.

In some tasks, SGD with momentum eventually generalizes better. This is why many strong training pipelines do not blindly stick with Adam forever. They tune it, schedule its learning rate, or compare it against SGD rather than assuming it is universally superior.

So if Adam trains fast but your final metrics disappoint, the optimizer may still be part of the explanation.

TensorFlow Training Example with Adam

A minimal custom training loop shows that Adam is still just applying gradients, not doing anything mystical.

1import tensorflow as tf
2
3x = tf.random.normal((32, 10))
4y = tf.random.normal((32, 1))
5model = tf.keras.Sequential([
6    tf.keras.layers.Dense(16, activation="relu"),
7    tf.keras.layers.Dense(1)
8])
9optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
10loss_fn = tf.keras.losses.MeanSquaredError()
11
12with tf.GradientTape() as tape:
13    predictions = model(x, training=True)
14    loss = loss_fn(y, predictions)
15
16grads = tape.gradient(loss, model.trainable_variables)
17optimizer.apply_gradients(zip(grads, model.trainable_variables))
18print(float(loss))

The optimizer stores moment information internally, but you still supply gradients in the same basic TensorFlow workflow.

Common Pitfalls

The most common mistake is assuming Adam removes the need to tune the learning rate. It does not.

Another mistake is treating faster initial convergence as proof of better final generalization. Those are not the same outcome.

Developers also blame Adam for instability when the underlying issue is elsewhere, such as exploding activations, bad normalization, poor batch sizing, or an inappropriate model architecture.

Summary

• Adam is gradient descent with per-parameter adaptive scaling based on first and second moment estimates.
• It still has a meaningful global learning rate and still requires tuning.
• 'beta_1, beta_2, and epsilon affect optimizer behavior but usually start well at their defaults.'
• Adam often converges quickly, but it does not automatically produce the best final model.
• Use Adam deliberately, compare it with alternatives, and debug the full training setup rather than treating the optimizer as magic.