Introduction

In regression analysis, one of the primary goals is to determine the relationship between a dependent variable and one or more independent variables. Two popular methods employed in linear regression analysis are the Normal Equation and the numpy.linalg functions, such as least_squares and solve . These methods are essential for computing the optimal parameters of the model, minimizing the error between predicted and actual outcomes. This article provides a detailed exploration of these methods, their technical underpinnings, differences, and examples of their application.

The Normal Equation

The Normal Equation is a mathematical formulation used to find the least-squares solution to a linear regression problem. If we denote $X$ as the matrix of input features, $y$ as the vector of outputs, and $\theta$ as the vector of parameters we wish to find, the Normal Equation is given by:

$$\theta = (X^TX)^{-1}X^Ty$$

In essence, it calculates the parameters that minimize the squared differences between the observed and predicted values. Here is a step-by-step breakdown of what each component does:

1. Matrix Multiplication ($X^T X$): Multiplies the transpose of the feature matrix $X$ with $X$ itself. This step ensures that we're accounting for all feature interactions.
2. Inverse ($(...)^{-1}$): The inverse of the resulting matrix from the previous multiplication is calculated. This step solves for the matrix to be used in calculating the parameter vector $\theta$.
3. Dot Product ($X^T y$): Multiplies the transpose of the feature matrix $X$ with the output vector $y$. This product is crucial for aligning our model's outputs with the actual observations.
4. Final Multiplication: Finally, we multiply the inverse matrix from the second step with the result from the third step to compute the optimal parameters $\theta$.

Pros and Cons

• Pros: The Normal Equation provides an exact solution without iteration, which is beneficial for smaller datasets. • Cons: Computing the inverse is computationally expensive ($O(n^3)$ complexity where $n$ is the number of features), and it becomes impractical for very large datasets as it can lead to numerical instability.

Numpy's least_squares

and solve

numpy.linalg.solve

This function is designed to solve linear equations of the form $Ax = b$ for the unknown $x$, where $A$ and $b$ are known. It's akin to a direct approach for small to moderate-sized systems. For the linear regression problem, we use it to solve for $\theta$ by rewriting the Normal Equation:

$$X^TX\theta = X^Ty$$

Here is how you might use numpy.linalg.solve in practice:

• Pros of solve : Fast and efficient for systems where matrix $X$ is square and doesn't suffer from ill-conditioning issues. • Cons of solve : Prone to numerical instability if the matrix is poorly conditioned. • Pros of lstsq : Robust against ill-conditioning of $X$, can handle cases where $X$ is not square. • Cons of lstsq : Generally slower than solve as it computes more information than just the solution vector.