Introduction

Logistic regression is a foundational machine learning algorithm used for binary classification tasks. It estimates the probability that a given input belongs to a category, usually employing the logistic (sigmoid) function. However, during the training phase, particularly when calculating the cost function, practitioners may encounter an unusual problem where the cost function returns NaN (Not a Number). This can result from numerical instability or data preprocessing issues. This article aims to explore the reasons why this happens and how to resolve it.

The Logistic Regression Model

Logistic Function

Logistic regression models the probability that the dependent variable belongs to a particular category. The logistic, or sigmoid function, is defined as:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

where $z$ is the weighted sum of the input features.

Cost Function

The cost function for logistic regression is given by the log loss function:

$$J(\theta) = -\frac{1}{m}\sum\_{i=1}^{m}\left[y^{(i)}\log(h\_\theta(x^{(i)})) + (1 - y^{(i)})\log(1 - h\_\theta(x^{(i)}))\right]$$

where: • $m$ is the number of training samples. • $y^{(i)}$ is the true label of sample $i$. • $h_\theta(x^{(i)})$ is the predicted probability for sample $i$.

Why Does the Cost Function Return NaN?

Several issues can lead to NaN values in the cost function:

1. Division by Zero: When $h_\theta(x^{(i)})$ is exactly 0 or 1, $\log(h_\theta(x^{(i)}))$ or $\log(1 - h_\theta(x^{(i)}))$ can result in NaN.
2. Numerical Overflow/Underflow: Large positive or negative inputs to the sigmoid function can cause overflow in exponential calculations, leading to NaN.
3. Data Precision: Very small feature values or a large range of values in datasets might affect calculations due to floating-point precision limits.
4. Improper Data Scaling: Data that has not been normalized or standardized can lead to poor performance and numerical issues.
5. Extreme Learning Rate: A learning rate that is too large can result in drastic updates to parameters leading to undefined operations.

Handling NaN in the Cost Function

Techniques to Prevent NaN

1. Clipping Predictions: Constrain the values of $h_\theta(x^{(i)})$ to a range slightly away from 0 and 1: