Keras is a high-level neural networks API written in Python, and its primary purpose is to support rapid experimentation. It has become a popular choice for defining and training complex deep learning models thanks to its simplicity and ease of use. Two fundamental metrics that Keras uses to evaluate models are "accuracy" and "loss". Understanding how these metrics are defined and calculated is important for interpreting model performance and fine-tuning.

Accuracy in Keras

Accuracy is a common metric used for classification problems to assess how well the model's predictions match the true labels. In Keras, accuracy is calculated based on how many predictions match the actual labels, expressed as a percentage.

Categorical Accuracy

1. Categorical Accuracy is used when dealing with multiclass classification problems, where each output prediction is an exclusive class. The ground truth (true label) and the predictions should be one-hot encoded:
   $$\text{Categorical Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$
   For example, if the ground truth for a sample is `[0, 1, 0]` and the model predicts `[0.1, 0.7, 0.2]`, the model's prediction is considered correct since the index with the highest prediction value (second index) matches with the one-hot encoded true label.
2. Sparse Categorical Accuracy is utilized when the true labels are provided as integers rather than one-hot encoded vectors. In this scenario, Keras automatically computes the accuracy by comparing the index of the maximum predicted value against the integer labels.

Binary Accuracy

Binary accuracy is suited for binary classification problems. When dealing with binary classification, predictions are often logistic, ranging between 0 and 1. The threshold is typically set at 0.5 to decide the class:

$$\text{Binary Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$

If the predicted probability is greater than 0.5, the sample is assumed to belong to the positive class (1), and otherwise to the negative class (0).

`Loss` in Keras

`Loss` functions indicate how well a model is performing during training and are crucial during the optimization process. Keras offers a range of pre-defined loss functions that are suitable for different tasks.

Mean Squared Error (MSE)

MSE is commonly used in regression problems and is defined as the average of the squared differences between the predicted and actual values:

$$MSE = \frac{1}{n} \sum_{i=1}^{n} (y_{\text{true},i} - y_{\text{pred},i})^2$$

Cross-Entropy `Loss`

Cross-entropy loss is particularly useful for classification tasks:

1. Categorical Cross-Entropy is applicable when the targets are one-hot encoded. It measures the dissimilarity between the predicted probability distribution and the true distribution:
   $$\text{Loss} = -\sum_{i=1}^{C} y_{\text{true},i} \cdot \log(y_{\text{pred},i})$$
2. Sparse Categorical Cross-Entropy is used when the targets are integers. This version is more memory-efficient for cases where the target is a single integer per class.
3. Binary Cross-Entropy evaluates the performance of a classification model whose output is a probability value between 0 and 1:
   $$\text{Loss} = -\frac{1}{n} \sum_{i=1}^{n} \left(y_{\text{true},i} \cdot \log(y_{\text{pred},i}) + (1 - y_{\text{true},i}) \cdot \log(1 - y_{\text{pred},i})\right)$$

Summary Table

Conclusion

Understanding how Keras defines "accuracy" and "loss" is vital for effectively evaluating model performance and tuning models for better results. Equipped with this knowledge, practitioners can choose the most appropriate metrics and loss functions according to their specific problem domain, ensuring they effectively measure what they intend to optimize.