Introduction

The idea that a weight vector is orthogonal to a decision plane is a geometric property of linear decision functions. You see it in perceptrons, logistic regression, and the last layer of many neural networks. Once this geometry is clear, margin distance, confidence sign, and parameter updates all become easier to reason about.

Linear Decision Function and Boundary

A linear classifier computes a score of the form w · x + b. Classification often uses the sign of that score, while the decision boundary is the set of points where the score is zero.

Boundary equation:

w · x + b = 0

Here:

• 'w controls orientation of the boundary'
• 'b controls translation of the boundary'
• 'x is an input point in feature space'

In two dimensions, the boundary is a line. In three dimensions, it is a plane. In higher dimensions, it is a hyperplane.

Why w Must Be Orthogonal

Take any two points x1 and x2 that lie on the boundary. Both satisfy:

w · x1 + b = 0

w · x2 + b = 0

Subtract the equations:

w · (x1 - x2) = 0

The vector x1 - x2 is a direction that lies along the boundary. Dot product zero means this in boundary direction is perpendicular to w. Since this is true for any boundary direction, w is normal to the full boundary.

That is the core proof. It does not depend on a specific training algorithm.

Geometric Intuition

Think of moving a point in space:

• move parallel to boundary and score stays unchanged
• move in direction of w and score increases fastest
• move in opposite direction and score decreases fastest

So the weight vector indicates the steepest ascent direction for the linear score. That is why it is drawn as an arrow perpendicular to the decision surface.

Distance to the Decision Plane

Signed distance from a point x to the boundary is:

(w · x + b) / ||w||

Interpretation:

• magnitude is geometric distance
• sign indicates which side of boundary the point is on

Python example:

1import numpy as np
2
3w = np.array([2.0, 1.0])
4b = -3.0
5x = np.array([1.5, 0.2])
6
7signed_distance = (np.dot(w, x) + b) / np.linalg.norm(w)
8print(signed_distance)

This quantity appears directly in margin based methods such as SVM.

Effect of Parameter Updates

Training changes decision geometry in two ways:

• updates to w rotate the boundary
• updates to b shift the boundary without changing orientation

Perceptron style update example:

1import numpy as np
2
3w = np.zeros(2)
4b = 0.0
5lr = 0.1
6
7# one misclassified sample
8x = np.array([1.0, 2.0])
9y = 1  # class in {-1, +1}
10
11if y * (np.dot(w, x) + b) <= 0:
12    w = w + lr * y * x
13    b = b + lr * y
14
15print("w:", w, "b:", b)

After updates, the normal vector changes and so does the boundary angle.

Connection to Neural Networks

People often ask this in neural network context. The statement is exactly true for any affine decision layer. In deep models, earlier layers transform inputs into learned features, and the final linear layer then applies the same geometry in that transformed space.

So orthogonality still holds for the final layer boundary in feature space. What changes is the representation, not the linear geometry itself.

Role of Feature Scaling

Feature scaling changes coordinate units, which changes how orientation appears in raw input space. A model trained with unscaled features can produce a boundary dominated by large scale dimensions.

Practical implication:

• standardize features before linear classification
• store scaler parameters and reuse them at inference

This keeps decision geometry consistent between training and prediction.

Common Pitfalls

A common misunderstanding is to think b affects direction. It does not. Bias shifts boundary position only.

Another pitfall is interpreting raw score as distance without dividing by ||w||. Magnitude of w changes score scale, so normalization is required for geometric distance.

Teams also conflate nonlinear model boundaries with this linear proof. The orthogonality statement applies directly to affine boundaries. Nonlinear models need local gradient analysis for similar geometric intuition.

Summary

• Linear decision boundaries satisfy w · x + b = 0.
• The weight vector w is orthogonal to every direction inside that boundary.
• Signed distance uses normalized score (w · x + b) / ||w||.
• Updating w rotates boundaries, while updating b translates them.
• In neural networks, the same geometry holds for linear decision layers in learned feature space.