Introduction

In machine learning, especially when training deep learning models using frameworks like TensorFlow, one can encounter numeric stability issues that result in outputs like Inf (Infinity). A common situation in which this might occur is while using the softmax cross entropy with logits function. Let's dive into why this happens and how it can be addressed.

Understanding Softmax Cross Entropy with Logits

Before we explore the issue, it's crucial to comprehend what softmax cross entropy with logits is and how it functions in the context of neural networks.

The softmax function converts raw scores (logits) into probabilities by exponentiating them and then normalizing by dividing with the sum of all exponentiated logits: $$\sigma(\boldsymbol{z}*i) = \frac{e^{z\_i}}{\sum*{j=1}^{K} e^{z\_j}}$$ where $K$ is the total number of classes.

The cross-entropy loss then measures the difference between two probability distributions — the predicted probability (output of the softmax) and the true distribution (usually represented as one-hot encoded vectors): $$H(p, q) = -\sum\_{i} p(x\_i) \log(q(x\_i))$$

In TensorFlow, the function combining these two operations is tf.nn.softmax_cross_entropy_with_logits(logits, labels) .

Why Does It Become Inf?

Several factors can cause the function result to become Inf , primarily due to numeric instability:

1. Very Large or Small Logits: Exponentiation of large logits can lead to overflow, resulting in Inf.
2. Incorrect Input Shapes: Mismatched dimensions between logits and labels may cause computational errors.
3. Dividing by Zero: A scenario in which, after exponentiation, the numerator becomes infinite, and since a finite number divided by zero approaches infinity.
4. Floating Point Precision: The inherent precision limits of floating point representations can lead to inaccuracies in computation, especially when numbers of vastly differing scales are involved.

Example Scenario

Below is a simple TensorFlow implementation that might produce Inf errors:

• Regularization: Implementing regularization techniques can help limit the magnitude of weights and subsequently logits. • Training Strategies: Adopt techniques such as gradient clipping during training to handle anomalies resulting from large updates.