Introduction

In machine learning, especially in tasks like image classification or language processing, training datasets often suffer from class imbalance, where some classes are underrepresented compared to others. This imbalance can lead to biased models that perform poorly on the minority classes. To address this issue, one effective approach is median frequency balancing used in conjunction with TensorFlow's `sparse_softmax_cross_entropy_with_logits` loss function.

Understanding Cross-Entropy `Loss`

Before delving into median frequency balancing, it’s important to understand the basics of cross-entropy loss, particularly `sparse_softmax_cross_entropy_with_logits`. This loss function is widely used for classification problems where classes are mutually exclusive. It combines softmax and cross-entropy in a single operation, ensuring numerical stability. When applied, it takes logits and sparse labels as inputs, where logits are the unnormalized predictions outputted by the model, and sparse labels represent the index of the true class.

Softmax Function

The softmax function converts logits into probabilities by normalizing the exponentials of the input numbers:

$$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$$

Cross-Entropy `Loss`

The cross-entropy loss measures the dissimilarity between the true label distribution and the predicted distribution obtained from softmax:

$$\text{CrossEntropyLoss} = -\sum_{i} y_i \log(\text{softmax}(z_i))$$

where $y_i$ is the true label in one-hot encoded form.

Median Frequency Balancing

Median frequency balancing is a technique used to counteract class imbalances by adjusting the loss function. The main idea is to assign different weights to each class in such a way that underrepresented classes have a higher influence during training.

Calculating Class Weights

Class weights can be determined using the median frequency of each class. This is computed by:

1. Calculating the frequency of each class in the dataset.
2. Computing the median of these frequencies.
3. Determining the class weight for each class based on the ratio of the median frequency to the frequency of the class.

Mathematically, the weight for class $i$ is given as:

$$\text{Weight}_i = \frac{\text{Median Frequency}}{\text{Frequency of class } i}$$

Applying Class Weights in `Loss` Function

When using `sparse_softmax_cross_entropy_with_logits`, you can incorporate these class weights to compute a weighted loss. In TensorFlow, this can be implemented by multiplying the computed loss by the class-specific weight:

• Class 0: $5 / 10 = 0.5$
• Class 1: $5 / 5 = 1.0$
• Class 2: $5 / 1 = 5.0$