Introduction

Learning-rate warm-up means starting training with a smaller learning rate and increasing it gradually until you reach the intended base learning rate. It is mainly used to stabilize the first part of training, when weights are still poorly calibrated and large optimizer steps can be unusually destructive.

Why Warm-up Exists

At the start of training, gradients can be noisy, activations can be poorly scaled, and optimizer state such as momentum estimates has not settled yet. If you immediately use an aggressive learning rate, the model may bounce around, diverge, or waste early iterations undoing unstable updates.

Warm-up reduces that risk by making the first steps intentionally conservative.

This is especially common in:

• transformer training
• very large batch training
• mixed-precision setups
• models that combine normalization, attention, and adaptive optimizers

Warm-up is not magic. It is a scheduling trick that makes early optimization gentler.

A Simple Mental Model

Suppose your intended learning rate is 1e-3. Without warm-up, step one already uses 1e-3.

With warm-up over the first 1000 steps, the schedule may look roughly like this:

• step 1 uses a tiny fraction of 1e-3
• step 500 uses about half of 1e-3
• step 1000 reaches the full 1e-3
• later steps follow the main schedule, such as constant, cosine decay, or linear decay

So warm-up is usually not the full schedule by itself. It is the beginning phase of a longer learning-rate policy.

Linear Warm-up Is the Usual Default

The most common version is linear warm-up. If warmup_steps is the number of steps reserved for warm-up and base_lr is the target learning rate, then during warm-up:

lr = base_lr * current_step / warmup_steps

After warm-up ends, you switch to the main schedule.

A simple PyTorch example makes the idea concrete.

1import torch
2from torch import nn
3from torch.optim import AdamW
4from torch.optim.lr_scheduler import LambdaLR
5
6model = nn.Linear(10, 1)
7optimizer = AdamW(model.parameters(), lr=1e-3)
8
9warmup_steps = 1000
10
11def lr_lambda(step):
12    if step < warmup_steps:
13        return float(step + 1) / float(warmup_steps)
14    return 1.0
15
16scheduler = LambdaLR(optimizer, lr_lambda=lr_lambda)
17
18for step in range(5):
19    optimizer.zero_grad()
20    x = torch.randn(32, 10)
21    y = torch.randn(32, 1)
22    loss = ((model(x) - y) ** 2).mean()
23    loss.backward()
24    optimizer.step()
25    scheduler.step()
26    print(step, optimizer.param_groups[0]["lr"])

This example warms the optimizer up to the configured 1e-3 learning rate.

Warm-up with a Decay Schedule

In practice, warm-up is often combined with decay rather than followed by a constant rate. For example:

• linear warm-up, then cosine decay
• linear warm-up, then linear decay
• warm-up, then inverse-square-root decay in some transformer setups

That combination is popular because it solves two different problems:

• warm-up protects the beginning of training
• decay helps convergence later in training

When Warm-up Helps Most

Warm-up tends to be most valuable when the optimizer or architecture is sensitive to early large updates. Large-batch training is a classic case because larger batches often motivate larger effective learning rates, which increases the risk of unstable first steps.

Warm-up can also help when a model trains fine eventually but shows erratic spikes or loss explosions right at the start.

Common Pitfalls

Using too many warm-up steps can make training unnecessarily slow. Warm-up should stabilize the beginning, not consume a huge share of the total budget without reason.

Assuming warm-up fixes every optimization problem is another mistake. If the base learning rate is fundamentally too high, warm-up may only delay failure.

Copying a warm-up schedule from another model family without checking batch size, optimizer, and total training length can also mislead you.

Finally, remember that warm-up is about the schedule, not only about choosing a smaller constant learning rate forever.

Summary

• learning-rate warm-up means increasing the learning rate gradually at the start of training
• it is used to make early optimization steps more stable
• linear warm-up is the most common variant
• warm-up is usually combined with a later decay schedule rather than used alone
• it helps most when early large updates would otherwise destabilize training, especially in large or sensitive deep-learning setups