In the context of neural networks, the softmax function is widely used in the output layer of classification models. It converts raw scores from the network into probabilities, allowing us to interpret them meaningfully as class probabilities. When implementing backpropagation, understanding how to derive the gradient of the softmax function with respect to its inputs is crucial. This article delves into the mathematical underpinnings of taking the derivative of the softmax output and provides practical context for its use.

The Softmax Function

The softmax function is defined as follows for a vector of scores $\mathbf{z} = [z_1, z_2, ..., z_n]$:

$$\sigma(\mathbf{z})*i = \frac{e^{z\_i}}{\sum*{j=1}^{n} e^{z\_j}}$$

where $i$ denotes the index of a particular output.

The softmax function maps the input vector to a probability distribution, where the sum of all probabilities is 1.

Derivative of the Softmax

To compute the derivative of the softmax output with respect to its inputs, we examine how a small change in one input $z_k$ affects the output $σ(z)_i$.

Derivative for the Same Index

For the derivative with respect to the same index $i$, we compute:

$$\frac{\partial \sigma(\mathbf{z})\_i}{\partial z\_i} = \sigma(\mathbf{z})\_i (1 - \sigma(\mathbf{z})\_i)$$

Derivative for Different Indices

For the derivative with respect to a different index $j$, where $i \neq j$, we have:

$$\frac{\partial \sigma(\mathbf{z})\_i}{\partial z\_j} = -\sigma(\mathbf{z})\_i \sigma(\mathbf{z})\_j$$

Combined Result

These derivatives are consolidated into a Jacobian matrix for the softmax function, which is crucial for computing the gradient:

$$\frac{\partial \sigma(\mathbf{z})}{\partial \mathbf{z}} = \begin{bmatrix} \sigma(\mathbf{z})\_1 (1 - \sigma(\mathbf{z})\_1) & -\sigma(\mathbf{z})\_1 \sigma(\mathbf{z})\_2 & \ldots & -\sigma(\mathbf{z})\_1 \sigma(\mathbf{z})\_n \\ -\sigma(\mathbf{z})\_2 \sigma(\mathbf{z})\_1 & \sigma(\mathbf{z})\_2 (1 - \sigma(\mathbf{z})\_2) & \ldots & -\sigma(\mathbf{z})\_2 \sigma(\mathbf{z})\_n \\ \vdots & \vdots & \ddots & \vdots \\ -\sigma(\mathbf{z})\_n \sigma(\mathbf{z})\_1 & -\sigma(\mathbf{z})\_n \sigma(\mathbf{z})\_2 & \ldots & \sigma(\mathbf{z})\_n (1 - \sigma(\mathbf{z})\_n) \end{bmatrix}$$

Backpropagation Context

In backpropagation, we often use the derivative of the loss function with respect to the network's outputs. Suppose we have a loss function $\mathcal{L}$, such as cross-entropy. To propagate the gradients back through the network, we need the derivative:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}} = \frac{\partial \mathcal{L}}{\partial \sigma(\mathbf{z})} \cdot \frac{\partial \sigma(\mathbf{z})}{\partial \mathbf{z}}$$

For classification tasks using cross-entropy loss, this reduces computation as follows: if $y_i$ is the true label, the gradient through softmax simplifies to the difference between probability and true label:

$$\frac{\partial \mathcal{L}}{\partial z\_i} = \sigma(\mathbf{z})\_i - y\_i$$

This simplification is due to the chain rule and the nature of the cross-entropy loss applied to the softmax output.

Key Points Summary

Conclusion

Understanding the derivatives of the softmax function is critical in the optimization of neural networks, particularly during the backpropagation stage. Armed with this mathematical foundation, one can implement efficient and accurate models capable of meaningful classification performance. By leveraging simplifications such as those seen with cross-entropy loss, neural networks can be optimized more effectively, making gradient descent-based approaches more powerful and efficient.