Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow. It enables fast experimentation through a high-level, user-friendly interface while providing the option to tune deeply for power users. A common task in training deep learning models is managing imbalanced datasets. Keras offers two key arguments in the fit() and fit_generator() methods that help handle class imbalance: class_weight and sample_weight . Although they both provide ways to weight inputs differently during training, they serve distinct purposes and have separate mechanisms. This article delves into their differences, use cases, and advantages.

Understanding class_weight

Definition

The class_weight parameter is a dictionary that maps class indices (integers) to a weight (float). This approach alters the contribution of classes during the loss computation. By providing higher weights to the minority classes, the model can focus more on classes that are less represented in the training data.

Use Case

• Imbalanced Datasets: In cases where one class significantly outnumbers others, such as fraud detection or rare disease prediction, directly addressing this imbalance is crucial. class_weight can effectively make the model learn equally from each class even if data is skewed.

Technical Explanation

The weights determine how the classes contribute to the overall loss:

• For each sample $(x_i, y_i)$, the loss is multiplied by the class weight corresponding to $y_i$.
• Thus, the loss contribution becomes: $$L_{\text{weighted}} = w_{y_i} \cdot L(x_i, y_i)$$

Example

• Hard Example Mining: If specific instances are known to be particularly challenging or critical, they can be assigned a higher weight.
• Data Quality: Samples with uncertain labels or lower confidence can be down-weighted to reduce their impact.
• For each sample $(x_i, y_i)$, the loss is scaled by $s_i$, the sample's specific weight:
• Performance Impact: Both approaches may lead to increased variance. When using class_weight , ensure that excessively high weights do not lead the model to overfit the minority class. Similarly, with sample_weight , very high or low weights could destabilize training.
• Compatibility: Ensure your data pipeline and compute resources can handle any increased complexity due to weighted loss computations. Verify that your generator's output is compatible with sample weight usage when using fit_generator() .