Introduction

Support Vector Machines (SVM) are powerful supervised learning models used for classification and regression tasks. One of the core concepts within SVM is the margin, which quantifies the distance between the decision boundary (hyperplane) and the nearest data points. Understanding the functional margin is crucial because it drives the optimization that determines the SVM model.

What Is the Functional Margin?

The functional margin measures the confidence of the classification made by the separating hyperplane. A larger functional margin means the data point is farther from the decision boundary and classified more confidently.

Mathematical Formulation

In a binary classification problem with labels $y_i \in {-1, 1}$, let:

• $\mathbf{x}_i$ be the feature vector of data point $i$
• $\mathbf{w}$ be the normal vector to the hyperplane
• $b$ be the bias term

The hyperplane is defined by all points $\mathbf{x}$ satisfying $\mathbf{w} \cdot \mathbf{x} + b = 0$.

For a data point $\mathbf{x}_i$ with label $y_i$, the functional margin is:

$$\gamma^{(i)} = y_i (\mathbf{w} \cdot \mathbf{x}_i + b)$$

Here, $\mathbf{w} \cdot \mathbf{x}_i$ is the dot product between the normal vector and the feature vector. The term $\mathbf{w} \cdot \mathbf{x}_i + b$ gives a signed distance from the point to the hyperplane (scaled by $\|\mathbf{w}\|$). Multiplying by the true label $y_i$ ensures that correct classifications yield a positive margin.

Interpreting the Functional Margin

• Positive margin ($\gamma^{(i)} > 0$): The point is classified correctly. The larger the value, the more confidently the point sits on the correct side of the boundary.
• Negative margin ($\gamma^{(i)} < 0$): The point is misclassified. It lies on the wrong side of the hyperplane.
• Zero margin ($\gamma^{(i)} = 0$): The point lies exactly on the decision boundary.

The Problem with Functional Margin: Scale Sensitivity

The functional margin has a critical flaw: it is not scale-invariant. If you multiply $\mathbf{w}$ and $b$ by a constant $k$, the functional margin scales by $k$ as well:

$$\gamma'^{(i)} = y_i (k\mathbf{w} \cdot \mathbf{x}_i + kb) = k \cdot \gamma^{(i)}$$

This means you can make the functional margin arbitrarily large without actually changing the hyperplane or improving classification. To fix this, SVM uses the geometric margin.

Geometric Margin

The geometric margin normalizes the functional margin by the norm of $\mathbf{w}$:

$$\hat{\gamma}^{(i)} = \frac{\gamma^{(i)}}{\|\mathbf{w}\|}$$

This gives the true Euclidean distance from the data point to the hyperplane, which is invariant to scaling of $\mathbf{w}$ and $b$. The SVM optimization objective is to maximize the minimum geometric margin across all training points:

$$\max_{\mathbf{w}, b} \frac{1}{\|\mathbf{w}\|} \quad \text{subject to} \quad y_i(\mathbf{w} \cdot \mathbf{x}_i + b) \geq 1 \quad \forall i$$

The constraint $\gamma^{(i)} \geq 1$ is a normalization convention. By fixing the functional margin of the closest points to 1, maximizing $\frac{1}{\|\mathbf{w}\|}$ is equivalent to maximizing the geometric margin.

Worked Example

Consider three data points:

• $\mathbf{x}_1 = (1, 2)$ with label $y_1 = +1$
• $\mathbf{x}_2 = (2, 3)$ with label $y_2 = -1$
• $\mathbf{x}_3 = (3, 4)$ with label $y_3 = +1$

With hyperplane parameters $\mathbf{w} = (0.5, 1)$ and $b = -3$:

Functional margins:

• $\gamma^{(1)} = (+1)(0.5 \cdot 1 + 1 \cdot 2 - 3) = (+1)(2.5 - 3) = -0.5$
• $\gamma^{(2)} = (-1)(0.5 \cdot 2 + 1 \cdot 3 - 3) = (-1)(4 - 3) = -1$
• $\gamma^{(3)} = (+1)(0.5 \cdot 3 + 1 \cdot 4 - 3) = (+1)(5.5 - 3) = 2.5$

Interpretation:

• $\mathbf{x}_1$ has a negative margin, so it is misclassified.
• $\mathbf{x}_2$ also has a negative margin, so it is misclassified.
• $\mathbf{x}_3$ has a positive margin of 2.5, so it is correctly classified with good confidence.

Geometric margins (with $\|\mathbf{w}\| = \sqrt{0.25 + 1} \approx 1.118$):

• $\hat{\gamma}^{(1)} = -0.5 / 1.118 \approx -0.447$
• $\hat{\gamma}^{(2)} = -1 / 1.118 \approx -0.894$
• $\hat{\gamma}^{(3)} = 2.5 / 1.118 \approx 2.236$

Summary

The functional margin captures classification confidence but is sensitive to the scale of the weight vector. The geometric margin corrects for this by normalizing. SVM's optimization maximizes the geometric margin by convention fixing the minimum functional margin to 1, which leads to the elegant formulation of finding the weight vector with minimum norm that correctly separates the data.