Introduction

Regularized logistic regression in MATLAB is a strong baseline for binary classification when feature count is high or features are correlated. Regularization controls model complexity and helps prevent overfitting on training data. The practical challenge is implementing cost and gradient correctly while excluding bias from penalty.

Core Sections

Build the regularized objective correctly

For binary labels, logistic regression predicts probabilities using sigmoid. With L2 regularization, the cost includes a penalty term on parameters except theta(1).

1function [J, grad] = lrCostFunctionReg(theta, X, y, lambda)
2    m = length(y);
3
4    h = sigmoid(X * theta);
5
6    thetaReg = theta;
7    thetaReg(1) = 0;
8
9    J = (-1 / m) * (y' * log(h) + (1 - y)' * log(1 - h)) ...
10        + (lambda / (2 * m)) * sum(thetaReg .^ 2);
11
12    grad = (1 / m) * (X' * (h - y)) + (lambda / m) * thetaReg;
13end
14
15function g = sigmoid(z)
16    g = 1 ./ (1 + exp(-z));
17end

This function is the core of training. If it is wrong, no optimizer setting will produce reliable models.

Train the model with fminunc

Use fminunc or fmincg to optimize parameters. Feature scaling usually improves convergence speed and stability.

1Xraw = [34 78; 30 43; 35 72; 60 86; 79 75; 75 68];
2y = [0; 0; 0; 1; 1; 1];
3
4mu = mean(Xraw);
5sigma = std(Xraw);
6Xn = (Xraw - mu) ./ sigma;
7
8X = [ones(size(Xn, 1), 1) Xn];
9initialTheta = zeros(size(X, 2), 1);
10lambda = 1.0;
11
12options = optimset('GradObj', 'on', 'MaxIter', 400, 'Display', 'off');
13[thetaOpt, cost] = fminunc(@(t) lrCostFunctionReg(t, X, y, lambda), initialTheta, options);
14
15disp(thetaOpt);
16disp(cost);

Convergence quality depends on scaling, learning landscape, and lambda choice.

Tune lambda with validation data

A single lambda value is rarely optimal. Use a validation split and compare performance across a lambda grid.

1lambdas = [0 0.01 0.1 1 10];
2valAcc = zeros(length(lambdas), 1);
3
4Xtrain = X(1:4, :); ytrain = y(1:4);
5Xval = X(5:end, :);  yval = y(5:end);
6
7for i = 1:length(lambdas)
8    lam = lambdas(i);
9    [t, ~] = fminunc(@(th) lrCostFunctionReg(th, Xtrain, ytrain, lam), ...
10                     zeros(size(Xtrain, 2), 1), options);
11    p = sigmoid(Xval * t) >= 0.5;
12    valAcc(i) = mean(double(p == yval));
13end
14
15disp([lambdas' valAcc]);

Use validation metrics to select lambda instead of choosing by intuition.

Handle numerical stability

When predicted probabilities are near zero or one, log(h) and log(1-h) can underflow. Clip probabilities slightly for robust training in difficult datasets.

For example, replace h with min(max(h, 1e-12), 1 - 1e-12) before computing cost. This keeps objective finite and avoids optimization breakdown.

Add pipeline checks around training

Model code should include checks for NaN values, class imbalance warnings, and feature scaling consistency between train and inference. Many production model failures come from inconsistent preprocessing, not from the classifier itself.

Keep preprocessing statistics such as mean and standard deviation with the model artifact so inference uses identical transformations.

Extend to polynomial features cautiously

For non-linear boundaries, polynomial feature mapping can help. However, higher-dimensional mapping increases overfitting risk, so regularization and validation become even more important.

Prefer incremental complexity: start linear, validate, then add feature mapping only when needed by error analysis.

Monitor model behavior after deployment

Training accuracy alone is not enough. Track prediction drift, class distribution changes, and probability calibration in production scoring logs. A model that performed well during training can degrade if incoming feature distributions shift.

Store model version, lambda value, and preprocessing statistics with each deployment artifact. This metadata is essential when you need to compare behavior across releases or perform rollback after regression findings.

Common Pitfalls

• Regularizing theta(1) and unintentionally biasing intercept behavior.
• Skipping feature normalization before optimization.
• Choosing lambda without validation-based comparison.
• Ignoring numerical instability in log computations.
• Training and inference pipelines using different preprocessing parameters.

Summary

• Implement L2 regularized cost and gradient with intercept excluded from penalty.
• Train using a gradient-aware optimizer such as fminunc.
• Select lambda through validation, not guesswork.
• Add numerical stability guards for extreme probability values.
• Keep preprocessing consistent across training and deployment.