Introduction to Multi-Class Logistic Regression

Logistic regression is a popular statistical method used for binary classification problems, which predicts the probability of one binary outcome. However, many real-world problems involve multi-class classification, that is, more than two classes or outcomes. Multi-class logistic regression is an extension that enables such predictions.

In this article, we will delve into the workings of multi-class logistic regression using the Python library SciKit Learn, a leading machine learning toolkit.

Logistic Regression

Before diving into multi-class logistic regression, it's important to first understand the basic logistic regression model. Logistic regression is used to model the probability of a certain class or event. It is a linear classifier that uses a logistic function to output probabilities between 0 and 1:

$$\text{Probability} = \frac{1}{1 + e^{-(\beta\_0 + \beta\_1X\_1 + \ldots + \beta\_nX\_n)}}$$

Where: • $\beta_0, \beta_1, \ldots, \beta_n$ are the parameters to be estimated. • $X_1, \ldots, X_n$ are the independent variables.

Multi-Class Logistic Regression

One-vs-Rest (OvR) Strategy

The most common approach for extending logistic regression to multi-class problems is the One-vs-Rest (OvR) strategy. Here, a binary classifier is built for each class, distinguishing that class from all other classes. The class with the highest probability is chosen as the predicted class.

Softmax Function

Another approach is the Softmax function, commonly used in conjunction with neural networks but applicable to logistic regression too. It generalizes the logistic function for multiple classes:

$$P(y = C\_i | X) = \frac{e^{\beta\_{i0} + \beta\_{i1}X\_1 + \ldots + \beta\_{in}X\_n}}{\sum\_{j=1}^{k} e^{\beta\_{j0} + \beta\_{j1}X\_1 + \ldots + \beta\_{jn}X\_n}}$$

Where: • $C_i$ is class $i$. • $k$ is the total number of classes.

Implementing Multi-Class Logistic Regression in SciKit Learn

SciKit Learn provides a straightforward API to implement multi-class logistic regression using the `LogisticRegression` class.

Example