Introduction

Categorical Cross-Entropy (CCE) is a widely used loss function in the field of machine learning and deep learning, particularly useful for classification tasks. When we deal with classes that are imbalanced, regular categorical cross-entropy can lead to biased results, favoring the majority class. To address this issue, we can use a variation called Weighted Sparse Categorical Cross-Entropy. This function allows us to intuitively assign different weights to each class, placing more emphasis on minority classes.

Sparse Categorical Cross-Entropy

Before diving into the weighted version, let's briefly go over Sparse Categorical Cross-Entropy. In situations where your target variables are integers representing class labels, sparse categorical cross-entropy is a suitable choice. Unlike categorical cross-entropy that requires one-hot encoded vectors as targets, sparse categorical cross-entropy uses integers directly.

Given an input vector $x$ and a target class $y$, sparse categorical cross-entropy is expressed as:

$$L(y, \hat{y}) = -\log(\hat{y}[y])$$

Where $\hat{y}$ is the predicted probability distribution produced by the model, usually obtained after passing raw logits through a softmax function, and $\hat{y}[y]$ is the predicted probability of the true target class $y$.

Weighted Sparse Categorical Cross-Entropy

Motivation

In many real-world applications, data can be highly imbalanced. For example, in medical diagnosis, the occurrence of rare diseases is significantly lower than common conditions. If a standard loss function is used, the model might tend to ignore minority classes due to their low impact on the loss term. By introducing class weights, we can penalize wrong predictions from less frequent classes more than those from more frequent ones.

Definition

Given class weights vector $w$ where $w = [w_0, w_1, \ldots, w_{n-1}]$, where $n$ is the number of classes, the weighted sparse categorical cross-entropy can be defined as:

$$L\_w(y, \hat{y}) = -w[y] \cdot \log(\hat{y}[y])$$

Where $w[y]$ is the weight for the true class $y$.

Implementation Example

Suppose you are classifying images of cats, dogs, and elephants, and your dataset is heavily biased towards cats and dogs. You can set the weights inversely proportional to their occurrences:

• Advantages: • Helps in dealing with imbalanced datasets. • Allows for better recall of minority classes. • Considerations: • The choice of weights is crucial; they can be set either manually based on expert knowledge or calculated automatically, for example, inversely proportional to class frequencies. • Overweighting minority classes too much may lead to high sensitivity and false positives.