Introduction

Linear regression with gradient descent is one of the simplest machine-learning algorithms to implement from scratch. In Java, the core idea is straightforward: predict with a line, measure error, compute gradients for the slope and intercept, and update both parameters repeatedly until the loss stabilizes.

The Model and Update Rule

For one-feature linear regression, the model is:

prediction = w * x + b

where:

• 'w is the weight or slope'
• 'b is the bias or intercept'

Gradient descent updates these values by moving them a small step opposite the gradient of the loss.

A Minimal Java Implementation

1public class LinearRegressionGD {
2    public static void main(String[] args) {
3        double[] x = {1, 2, 3, 4, 5};
4        double[] y = {3, 5, 7, 9, 11};
5
6        double w = 0.0;
7        double b = 0.0;
8        double learningRate = 0.01;
9        int epochs = 5000;
10
11        int n = x.length;
12
13        for (int epoch = 0; epoch < epochs; epoch++) {
14            double dw = 0.0;
15            double db = 0.0;
16            double loss = 0.0;
17
18            for (int i = 0; i < n; i++) {
19                double prediction = w * x[i] + b;
20                double error = prediction - y[i];
21                loss += error * error;
22                dw += error * x[i];
23                db += error;
24            }
25
26            dw = (2.0 / n) * dw;
27            db = (2.0 / n) * db;
28            loss = loss / n;
29
30            w -= learningRate * dw;
31            b -= learningRate * db;
32
33            if (epoch % 1000 == 0) {
34                System.out.printf("epoch=%d loss=%.6f w=%.4f b=%.4f%n", epoch, loss, w, b);
35            }
36        }
37
38        System.out.printf("final model: y = %.4fx + %.4f%n", w, b);
39    }
40}

This example trains a line close to y = 2x + 1.

Why the Code Works

For mean squared error, the gradient tells you how much changing w and b will affect the loss. If predictions are too high, gradient descent pushes the parameters downward. If they are too low, it pushes them upward.

The learning rate controls the size of each update step. Too small and training is slow. Too large and the algorithm may overshoot or diverge.

Feature Scaling Still Matters

Even in simple linear regression, feature scaling can help gradient descent converge more smoothly when input values are large.

1public static double[] normalize(double[] values) {
2    double max = values[0];
3    for (double v : values) {
4        if (v > max) max = v;
5    }
6
7    double[] result = new double[values.length];
8    for (int i = 0; i < values.length; i++) {
9        result[i] = values[i] / max;
10    }
11    return result;
12}

For one tiny example it is not essential, but for realistic data it often improves stability.

How to Predict After Training

Once training finishes, use the learned parameters directly.

double xNew = 6.0;
double prediction = w * xNew + b;
System.out.println("prediction = " + prediction);

That is the entire purpose of training: find parameters that generalize to unseen input values.

Common Pitfalls

The most common mistake is choosing a learning rate that is too large, which makes the loss explode instead of decreasing.

Another mistake is forgetting to average gradients over the dataset, which changes the scale of updates and can make tuning inconsistent.

Developers also often test on perfectly linear toy data and assume the same settings will behave well on noisy or differently scaled real data.

Summary

• Linear regression predicts with w * x + b.
• Gradient descent updates w and b using the loss gradient.
• Learning rate strongly affects convergence behavior.
• Feature scaling can make training more stable.
• A from-scratch Java implementation is small enough to understand fully, which makes it a useful learning exercise.