Introduction

Teacher forcing is a training technique for sequence decoders where the model is fed the true previous token instead of always feeding back its own last prediction. In PyTorch, this usually appears in custom sequence-to-sequence training loops. It can make optimization much more stable, but it also creates a gap between training behavior and inference behavior.

Why Teacher Forcing Helps

In an autoregressive decoder, each prediction becomes part of the next step's input. If the model makes a bad prediction early, the error propagates forward and the rest of the sequence is now conditioned on a bad token.

Teacher forcing reduces that problem by replacing the model's previous prediction with the ground-truth token during training. This gives the decoder a cleaner context and lets it learn token transitions more easily in early training.

A Typical PyTorch Decoder Loop

In PyTorch, teacher forcing is usually not hidden inside a built-in layer. You implement it in the decoder loop yourself.

1import random
2import torch
3
4
5def decode(decoder, hidden, target_tokens, start_token, teacher_force_ratio=0.5):
6    batch_size, seq_len = target_tokens.shape
7    input_token = torch.full(
8        (batch_size,),
9        start_token,
10        dtype=torch.long,
11        device=target_tokens.device,
12    )
13    logits_per_step = []
14
15    for t in range(seq_len):
16        logits, hidden = decoder(input_token, hidden)
17        logits_per_step.append(logits.unsqueeze(1))
18
19        predicted = logits.argmax(dim=1)
20        use_teacher = random.random() < teacher_force_ratio
21        input_token = target_tokens[:, t] if use_teacher else predicted
22
23    return torch.cat(logits_per_step, dim=1), hidden

The key line is the branch between target_tokens[:, t] and predicted. That is teacher forcing.

Shift Inputs and Targets Correctly

One of the most common bugs is off-by-one alignment. If the decoder input at step t is the previous token, then the target for the prediction at step t is usually the current token.

For example, if the target sequence is:

• start token
• 'I'
• 'agree'
• end token

then the decoder might consume start token, predict I, then consume I, predict agree, and so on.

If you get the shifting wrong, the model can still train numerically while learning the wrong mapping.

Training and Inference Are Different

Teacher forcing exists only during training. At inference time, you do not have the ground-truth future tokens, so the decoder must feed back its own predictions.

That is why a model can show low training loss with strong teacher forcing and still behave badly in real generation. During inference, it has to recover from its own mistakes, and teacher forcing does not prepare it fully for that.

A common mitigation is to reduce the forcing ratio over time:

def teacher_force_ratio(epoch, max_epoch):
    return max(0.1, 1.0 - epoch / max_epoch)

This kind of schedule gradually pushes training behavior closer to inference behavior.

Loss Computation Still Looks Normal

Teacher forcing changes decoder inputs, not the loss formula. You still compute token-level loss against the target sequence:

1criterion = torch.nn.CrossEntropyLoss(ignore_index=pad_id)
2
3logits, _ = decode(decoder, hidden, targets[:, 1:], start_token)
4loss = criterion(
5    logits.reshape(-1, logits.size(-1)),
6    targets[:, 1:].reshape(-1),
7)

The important part is making sure the targets line up with the predicted time steps and that padding is masked properly.

When Too Much Teacher Forcing Hurts

Using a forcing ratio of 1.0 forever often produces a decoder that learns under unrealistically clean conditions. Training may look stable, but inference can degrade because the model rarely had to condition on its own outputs.

That does not mean teacher forcing is bad. It means it is a trade-off:

• high ratio: easier optimization, less realistic decoder context
• low ratio: harder optimization, more realistic context

The right value depends on model size, data difficulty, and how fragile autoregressive generation is in your task.

Common Pitfalls

• Using teacher forcing during inference by mistake.
• Misaligning decoder inputs and targets by one time step.
• Leaving the forcing ratio at 1.0 forever and never checking free-running generation quality.
• Forgetting to ignore padding tokens in the loss.
• Assuming teacher forcing solves all sequence-model instability when the real issue may be poor tokenization, weak data, or decoder design.

Summary

• Teacher forcing feeds the true previous token into the decoder during training.
• It helps optimization by reducing compounding errors early in the sequence.
• In PyTorch, it is usually implemented explicitly inside the decoder loop.
• Training and inference behave differently, so free-running evaluation still matters.
• Scheduled reduction of the forcing ratio is often more realistic than always forcing or never forcing.