Introduction

Logistic regression is a widely used statistical technique for binary classification problems. Unlike ordinary least squares regression, logistic regression predicts the probability that a given input point belongs to a particular class. This is achieved through a logistic function, which maps predicted output probabilities between 0 and 1.

Locally weighted logistic regression (LWLR), also known as locally weighted scatterplot smoothing (LOWESS) for classification, is a type of non-parametric approach that enhances the prediction capability of standard logistic regression by applying weights to the data points based on their distance from the target point. This ensures that points closer to the prediction point have more influence than those further away, allowing better handling of non-linear decision boundaries.

Technical Explanation

Logistic Regression Formulation

In logistic regression, the relationship between the dependent binary variable and the independent variables is modeled using the logistic function:

$$P(y=1|X) = \frac{1}{1 + e^{-(\theta\_0 + \theta\_1 X\_1 + \theta\_2 X\_2 + \cdots + \theta\_n X\_n)}}$$

Where: • $P(y=1|X)$ is the probability that the output variable is one (given the vector of inputs $X$). • $\theta_0, \theta_1, \ldots, \theta_n$ are the parameters.

The goal is to determine the optimal $\theta$ values that maximize the likelihood of the observed data.

Locally Weighted Logistic Regression (LWLR)

LWLR adapts the logistic regression by incorporating a weight function. Unlike global logistic regression that assumes uniform influence of all data points, LWLR assigns weights to each data point, which diminish with distance from the target data point:

1. Weight Function: The weight $W(i)$ for a training sample $x^{(i)}$ is given by: $$W(i) = \exp\left(-\frac{(x^{(i)} - x)^2}{2\tau^2}\right)$$ Where $\tau$ is the bandwidth parameter determining how quickly the influence of data points decreases with distance.
2. Cost Function: The locally weighted cost function is: $$J(\theta) = \sum\_{i=1}^{m} W(i) \left[y^{(i)} \log(h\_\theta(x^{(i)})) + (1-y^{(i)}) \log(1-h\_\theta(x^{(i)}))\right]$$
3. Optimization: LWLR optimizes the above objective by considering weights, typically using iterative techniques like gradient descent or sophisticated methods like Newton-Raphson.

Example

Consider a binary classification problem where the dataset contains non-linear relationships. Standard logistic regression might struggle here, as it imposes a linear decision boundary. LWLR can effectively model complex decision boundaries because it effectively "focuses" on a local neighborhood of data points while making the prediction.

• Medical Diagnosis: LWLR can enhance medical imaging through precise cancer cell classification, especially when dealing with non-linear features. • Finance: Predicting stock market trends where non-linear relationships are common between various financial indicators. • Anomaly Detection: Identifying rare events in manufacturing by adjusting the sensitivity to local conditions.