Introduction

The learning rate is the training setting that most quickly decides whether optimization feels stable or hopeless. Too high, and loss explodes or oscillates. Too low, and training crawls for hours while barely improving. A good process is not to guess one number forever. It is to pick a sensible starting scale, watch the training signals, and use a schedule that reduces the rate as optimization progresses.

Start With the Optimizer and Scale

A "good" learning rate depends on the optimizer. For the same model, SGD, Adam, and AdamW often want different starting values.

Reasonable first guesses are:

• SGD without momentum: around 1e-2 or lower
• SGD with momentum: often 1e-2 to 1e-1
• Adam or AdamW: often around 1e-3

These are not laws. They are starting points. The important point is that the correct order of magnitude matters more than the third decimal place.

Read the Training Signals

You do not need a perfect theory to detect a bad learning rate. The first few hundred updates usually tell you enough.

Signs it is too high:

• loss jumps upward instead of trending down
• gradients or parameters become NaN
• validation metrics fluctuate wildly
• training becomes unstable after a few good steps

Signs it is too low:

• loss decreases extremely slowly
• training and validation curves are both flat
• doubling the epoch count changes little

A simple habit is to print the current learning rate and loss together. That gives you a direct view of how schedule changes affect optimization.

Use a Small Learning-Rate Range Test

A practical way to choose the initial rate is a range test: start very low, increase the learning rate gradually over one short run, and stop when the loss becomes unstable. Then pick a value below that instability region.

A simplified PyTorch example:

1import torch
2from torch import nn
3from torch.utils.data import DataLoader, TensorDataset
4
5x = torch.randn(512, 10)
6y = (x.sum(dim=1) > 0).long()
7loader = DataLoader(TensorDataset(x, y), batch_size=32, shuffle=True)
8
9model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))
10loss_fn = nn.CrossEntropyLoss()
11optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
12
13start_lr = 1e-5
14end_lr = 1e-1
15steps = len(loader)
16multiplier = (end_lr / start_lr) ** (1 / max(steps - 1, 1))
17
18for batch_x, batch_y in loader:
19    optimizer.zero_grad()
20    logits = model(batch_x)
21    loss = loss_fn(logits, batch_y)
22    loss.backward()
23    optimizer.step()
24
25    current_lr = optimizer.param_groups[0]["lr"]
26    print(f"lr={current_lr:.6f} loss={loss.item():.4f}")
27    optimizer.param_groups[0]["lr"] *= multiplier

When the loss starts rising sharply or turning unstable, you have gone too far. A usable initial rate is usually below that point.

Decay the Rate Instead of Keeping It Fixed

A fixed learning rate can work, but many models train better when the rate is reduced over time. Early in training you want larger steps. Later you want smaller, more precise ones.

A basic schedule in PyTorch using cosine decay:

1optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
2scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=20)
3
4for epoch in range(20):
5    for batch_x, batch_y in loader:
6        optimizer.zero_grad()
7        logits = model(batch_x)
8        loss = loss_fn(logits, batch_y)
9        loss.backward()
10        optimizer.step()
11    scheduler.step()
12    print("epoch", epoch, "lr", optimizer.param_groups[0]["lr"])

Other useful schedules are step decay, one-cycle, and reduce-on-plateau. The best one depends on how long training runs and how sensitive the model is to late-stage refinement.

Use Warmup for Fragile Training

Large models or aggressive optimizers often benefit from a short warmup phase. Warmup begins with a smaller learning rate and ramps up to the target value over the first few steps or epochs.

This is especially helpful when:

• batch sizes are large
• the model is deep
• mixed precision is used
• training becomes unstable at the very beginning

Warmup is not magic, but it reduces the chance that the optimizer takes destructive early steps before the model statistics settle down.

Tuning Rule of Thumb

A useful operational loop is:

1. start with the optimizer's usual default scale
2. run a short experiment and inspect loss stability
3. if unstable, lower the rate by 2x to 10x
4. if painfully slow, raise the rate moderately
5. add decay or warmup once the initial scale is reasonable

That process is much more reliable than blind grid search over dozens of random numbers.

Common Pitfalls

The biggest mistake is changing several training settings at once. If you change batch size, optimizer, augmentation, and learning rate together, you cannot tell which one caused the behavior.

Another mistake is trusting only the final validation score and ignoring the training curve. A rate that eventually works may still waste huge amounts of time if it is much too low.

Teams also often copy a learning rate from another project without checking the optimizer or batch size. Those details change the right scale substantially.

Finally, do not assume adaptive optimizers remove the need for learning-rate tuning. Adam still fails with bad rates. It just fails differently from SGD.

Summary

• The learning rate controls training speed and stability more than most other hyperparameters.
• Pick the initial scale based on the optimizer, then verify it from the early loss curve.
• A range test is a practical way to find a usable starting value.
• Decay schedules and warmup are often more effective than one fixed rate.
• Tune one variable at a time so the training signal stays interpretable.