Logistic regression is a statistical method used for binary classification, and its cost function is central to how it works. Unlike linear regression, where the cost function typically is the Mean Squared Error, the cost function for logistic regression incorporates a logarithmic expression. Here's why this formulation is both appropriate and effective.

Binary Classification and Logistic Regression

In logistic regression, the primary aim is to classify input data into one of two categories. The model achieves this by outputting probabilities through the logistic (sigmoid) function, constrained between 0 and 1:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Where $z = w^T x + b$ is a linear combination of input features $x$, the weights $w$, and bias $b$.

Cost Function in Logistic Regression

For a binary classification problem, it is crucial that the model not only predicts the correct class but also provides a measure of confidence in its predictions. This necessitates a different approach from mean squared error (used in linear regression), leading us to the logarithmic cost function:

Log `Loss` (Cross-Entropy Loss)

The logarithmic cost, often referred to as Log `Loss` or Cross-Entropy Loss, penalizes wrong predictions more when these are made with high confidence:

$$L(y, \hat{y}) = -[y \log(\hat{y}) + (1-y) \log(1 - \hat{y})]$$

Where $y$ is the actual label (0 or 1), and $\hat{y}$ is the predicted probability of the positive class.

Why Use a Logarithmic Function?

1. Asymmetry and Penalty Magnitude: • Incorrect predictions are penalized logarithmically, which means if a wrong prediction is made with high confidence, the penalty is much larger. • This logarithmic nature helps in dealing more effectively with misclassifications compared to quadratic penalties (e.g., MSE).
2. Convexity: • The cost function forms a convex surface, essential for optimization. This ensures the model converges to a global minimum rather than getting stuck in local minima, which would otherwise be a risk with non-convex functions.
3. Interpretability: • Logarithmic cost functions translate multiplicative differences in probabilities into additive differences in cost, which is more interpretable and simplifies gradient calculations.

Working Example

Consider a binary classification problem where the true labels of a dataset sample are `[1, 0, 1, 1, 0]`, and the predicted probabilities are `[0.9, 0.1, 0.8, 0.6, 0.4]`. The Log `Loss` for each sample would be computed as follows:

• For the first instance: $-[(1) \log(0.9) + (0) \log(1-0.9)] = 0.105$ • For the second instance: $-[(0) \log(0.1) + (1) \log(1-0.1)] = 0.105$

Continuing this calculation for each pair, you obtain individual loss values, which are averaged across the dataset for the total cost.

Table: `Loss` Function Examples

\begin{array}{|c|c|c|c|} \hline \text{True Label (y)} & \text{Predicted Probability (\hat{y})} & \text{Individual Loss} & \text{Penalty Type} \\ \hline 1 & 0.9 & 0.105 & \text{Low Cost for Accurate Prediction} \\ \hline 0 & 0.1 & 0.105 & \text{Low Cost for Accurate Prediction} \\ \hline 1 & 0.8 & 0.223 & \text{Moderate Cost} \\ \hline 1 & 0.6 & 0.511 & \text{Higher Cost due to Uncertainty} \\ \hline 0 & 0.4 & 0.511 & \text{Higher Cost due to Inaccuracy} \\ \hline \end{array}

Gradient Descent and Optimization

The logarithmic cost function assists in computing a gradient efficiently, due to its continuous and differentiable nature. The gradient of the cost function with respect to weights is straightforward, as it builds on matrix derivatives:

$$\frac{\partial L}{\partial w} = (y - \hat{y}) \cdot x$$

During optimization, algorithms like Gradient Descent iteratively update the weights to minimize this cost, steadily improving the model's predictive ability.

Conclusion

The logarithmic expression in logistic regression's cost function elegantly addresses the key challenges of binary classification. It imposes a structure that not only ensures reliable predictions but also simplifies the optimization process due to its convex nature. Key qualities like offering an interpretable penalty for misclassified but high-confidence predictions remain foundational for its preference over other possible loss functions, such as quadratic functions, in binary classification contexts.