Introduction

Linear regression is simple to implement with NumPy, but plain least squares can overfit or become unstable when features are correlated. Regularization fixes that by penalizing large coefficients, with ridge regression as the easiest NumPy implementation and lasso requiring an iterative solver.

Ridge Regression in Closed Form

For ordinary least squares, the coefficient vector is based on the normal equation. Ridge regression adds an L2 penalty, which leads to this modified system:

(X^T X + lambda I)^{-1} X^T y

That makes ridge regression a natural fit for NumPy because it can be written directly with matrix operations.

1import numpy as np
2
3X = np.array([
4    [1.0, 1.0],
5    [1.0, 2.0],
6    [2.0, 2.0],
7    [2.0, 3.0],
8])
9y = np.array([6.0, 8.0, 9.0, 11.0])
10
11lambda_ = 1.0
12
13X_bias = np.c_[np.ones(len(X)), X]
14I = np.eye(X_bias.shape[1])
15I[0, 0] = 0.0   # do not regularize the bias term
16
17weights = np.linalg.inv(X_bias.T @ X_bias + lambda_ * I) @ X_bias.T @ y
18print(weights)

The I[0, 0] = 0.0 line is important if you do not want the intercept penalized.

Why Regularization Helps

Regularization discourages the model from using very large weights to explain training noise. That helps when:

• features are highly correlated
• the dataset is small
• the number of features is large relative to the number of rows

Ridge regression does not usually force weights all the way to zero, but it shrinks them toward smaller values and often improves generalization.

A Reusable NumPy Ridge Function

It is straightforward to wrap ridge regression in a function:

1import numpy as np
2
3
4def ridge_fit(X, y, lambda_):
5    X = np.asarray(X, dtype=float)
6    y = np.asarray(y, dtype=float)
7
8    X_bias = np.c_[np.ones(len(X)), X]
9    penalty = np.eye(X_bias.shape[1])
10    penalty[0, 0] = 0.0
11
12    return np.linalg.solve(X_bias.T @ X_bias + lambda_ * penalty, X_bias.T @ y)
13
14
15def ridge_predict(X, weights):
16    X = np.asarray(X, dtype=float)
17    X_bias = np.c_[np.ones(len(X)), X]
18    return X_bias @ weights
19
20
21weights = ridge_fit(X, y, lambda_=1.0)
22predictions = ridge_predict(X, weights)
23print(predictions)

Using np.linalg.solve is usually better than explicitly computing a matrix inverse.

What About Lasso

Lasso uses an L1 penalty rather than an L2 penalty. That makes it useful for feature selection because some coefficients can be driven to exactly zero. The tradeoff is that there is no equally neat closed-form solution like ridge.

You normally solve lasso with iterative methods such as coordinate descent or subgradient methods. That can still be done in NumPy, but it is more code and more care than ridge.

So if your goal is to understand regularized linear regression in pure NumPy, ridge is the cleanest first implementation.

Feature Scaling Matters

Regularization strength interacts with feature scale. If one feature has values around 0.01 and another around 10_000, the penalty does not treat them fairly.

That is why standardizing features before regularized regression is often a good idea:

X_mean = X.mean(axis=0)
X_std = X.std(axis=0)
X_scaled = (X - X_mean) / X_std

Without scaling, your chosen lambda_ may behave very differently from what you intended.

Common Pitfalls

• Penalizing the intercept term when you intended to regularize only feature weights.
• Using np.linalg.inv everywhere instead of np.linalg.solve.
• Forgetting to scale features before applying regularization.
• Expecting lasso to have the same simple closed-form implementation as ridge.
• Choosing lambda_ blindly without validation on held-out data.

Summary

• Ridge regression is the easiest regularized linear regression to implement in pure NumPy.
• Its closed-form solution adds an L2 penalty to the normal equation.
• Do not regularize the intercept unless that is an intentional choice.
• Standardize features so the penalty behaves consistently across columns.
• Use ridge for a clean NumPy implementation and treat lasso as a more iterative optimization problem.