Introduction

Label smoothing is a regularization technique that makes classification targets less extreme. Instead of training the model to assign full probability to the correct class and zero to every other class, you soften the target distribution slightly. In PyTorch, this is easy to use for standard classification because nn.CrossEntropyLoss supports a built-in label_smoothing argument.

Why Label Smoothing Helps

With ordinary one-hot targets, the model is encouraged to become very confident. That can hurt calibration and sometimes generalization, especially in problems where labels are noisy or classes overlap.

Label smoothing changes the target distribution so the correct class still gets most of the probability mass, but not all of it. For example, with smoothing 0.1 in a 5-class problem, the target is no longer “100 percent class 2, zero everywhere else.” It becomes a slightly softened distribution.

The effect is usually:

• less overconfidence
• smoother gradients
• sometimes better validation performance

It is not magic, but it is a practical regularization tool.

Use the Built-In PyTorch Support

For standard multiclass classification, the cleanest solution is the built-in loss argument.

1import torch
2import torch.nn as nn
3
4logits = torch.tensor([
5    [2.5, 0.3, -1.2],
6    [0.1, 1.7, 0.2],
7], dtype=torch.float32)
8
9targets = torch.tensor([0, 1], dtype=torch.long)
10
11criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
12loss = criterion(logits, targets)
13
14print(loss.item())

This is the preferred approach when your targets are class indices and the task fits standard cross-entropy training.

Understand the Smoothing Strength

The smoothing value is usually small, such as 0.05 or 0.1. If you make it too large, you can damage learning because the target stops carrying a strong enough signal.

In practice:

• '0.0 means no smoothing'
• small values are common starting points
• larger values need justification and validation

Treat it like any other hyperparameter. Do not assume more smoothing is better.

Manual Implementation for Custom Cases

If you need full control, you can create smoothed targets manually and compute the loss yourself.

1import torch
2import torch.nn.functional as F
3
4
5def smoothed_loss(logits, targets, num_classes, smoothing=0.1):
6    with torch.no_grad():
7        true_dist = torch.full_like(logits, smoothing / (num_classes - 1))
8        true_dist.scatter_(1, targets.unsqueeze(1), 1.0 - smoothing)
9
10    log_probs = F.log_softmax(logits, dim=1)
11    return -(true_dist * log_probs).sum(dim=1).mean()
12
13
14logits = torch.tensor([[2.5, 0.3, -1.2]], dtype=torch.float32)
15targets = torch.tensor([0], dtype=torch.long)
16
17print(smoothed_loss(logits, targets, num_classes=3, smoothing=0.1).item())

Manual control is useful when you need nonstandard target distributions or want to experiment beyond the built-in behavior.

When Label Smoothing Fits Best

Label smoothing is most natural in single-label multiclass classification. It is less obviously appropriate for every task. For example:

• multilabel classification often uses different losses and target semantics
• some knowledge-distillation setups already use soft targets
• highly imbalanced or specialized tasks may respond differently

So while the technique is common, it should still be validated on your actual problem.

Watch Model Calibration and Accuracy Together

One reason people use label smoothing is better calibration, not just higher accuracy. A model can have similar or slightly better accuracy while being less overly confident.

That means evaluation should not stop at top-1 accuracy. If confidence quality matters, also examine:

• validation loss
• confidence histograms
• calibration metrics
• downstream decision thresholds

Label smoothing is often most valuable when predictions will be consumed by another system that interprets confidence values.

Common Pitfalls

The first pitfall is applying label smoothing automatically without checking whether the task and loss function actually match the standard multiclass setup.

Another issue is using too much smoothing. If the target distribution becomes too soft, training can lose useful signal and accuracy may drop.

Developers also forget that the built-in CrossEntropyLoss(label_smoothing=...) expects class-index targets, not already one-hot encoded distributions.

Finally, do not assume label smoothing will always improve results. It is a regularization choice, not a guaranteed upgrade.

Summary

• Label smoothing softens one-hot classification targets to reduce overconfidence.
• In PyTorch, the easiest approach is nn.CrossEntropyLoss(label_smoothing=...).
• Small smoothing values are the normal starting point.
• Manual implementations are useful for custom target distributions.
• Validate the effect on both accuracy and confidence quality instead of assuming it helps automatically.