PyTorch Error: `multi-target not supported in CrossEntropyLoss()`

When working with PyTorch, one might encounter the error: `RuntimeError: multi-target not supported in CrossEntropyLoss()`. Understanding the cause of this error requires delving into the functionality of the `CrossEntropyLoss` function, its expected inputs, and contrasting it with a common confusion in its usage. This article aims to elucidate the problem, guiding you through potential solutions and the underlying theory in PyTorch.

Understanding `CrossEntropyLoss`

`CrossEntropyLoss` is a commonly used loss function in training classification models. It combines `LogSoftmax` and `NLLLoss` in one single class, making it suitable for multi-class classification tasks. Here's a brief technical explanation of its workings:

• Softmax Layer: Computes exponential scores and normalizes to probabilities. The output tensor is of shape `(N, C)` where `N` is the batch size and `C` is the number of classes.
• Logarithm Layer: Takes the logarithm of the probabilities.
• Negative Log-Likelihood (NLL): Computes the loss between the predicted log-probabilities and the true distribution.

Expected Inputs

1. Predictions (`input`): Should be of shape `(N, C)` where `N` is the batch size and `C` is the number of target classes. These are raw and unnormalized scores (logits).
2. Target (`target`): Should be of shape `(N,)`. Each value is a class index in the range `[0, C-1]`.

Cause of the Error

The error arises when the `target` input provided to `CrossEntropyLoss` is not in line with its expected format. Specifically, this error is triggered when:

• The `target` tensor shape is `(N, C)` (one-hot encoded).
• Instead of integers representing class indices, a multi-dimensional (e.g., one-hot encoded) target is provided.
• Softmax Layer in Model: If you apply a softmax layer in the last layer of your model, do not use `CrossEntropyLoss`. Instead, consider `NLLLoss` after applying `log` over softmax outputs.
• Focal `Loss` for Imbalanced Data: When dealing with class imbalance, consider using variations like Focal `Loss` which can handle difficulties in learning from imbalanced datasets by down-weighting easy examples.