Introduction

Finding the minimum point of a function is a core optimization task in machine learning, simulation, and numerical engineering. The right algorithm depends on what you know about the function, such as smoothness, derivatives, and whether the function is unimodal on a bounded interval. Choosing a method without those assumptions often produces unstable or misleading results.

Core Sections

1. Match algorithm choice to function properties

Before implementation, classify the problem:

• One-dimensional or multi-dimensional.
• Derivative available or unavailable.
• Unimodal interval guaranteed or not guaranteed.
• Local minimum acceptable or global minimum required.

For one-dimensional unimodal functions on a closed interval, ternary search or golden-section search is simple and reliable. For multi-dimensional smooth functions, gradient-based methods are usually faster. If the landscape has many local minima, multi-start or global search methods become important.

2. Ternary search for unimodal one-dimensional functions

Ternary search repeatedly narrows the interval by comparing values at two internal points. It is robust and does not require derivatives.

1from typing import Callable, Tuple
2
3
4def ternary_search(f: Callable[[float], float], left: float, right: float, eps: float = 1e-7) -> Tuple[float, float]:
5    while right - left > eps:
6        m1 = left + (right - left) / 3.0
7        m2 = right - (right - left) / 3.0
8        if f(m1) < f(m2):
9            right = m2
10        else:
11            left = m1
12    x = (left + right) / 2.0
13    return x, f(x)
14
15
16f = lambda x: (x - 2.5) ** 2 + 3.0
17x_star, y_star = ternary_search(f, -10.0, 10.0)
18print(round(x_star, 6), round(y_star, 6))

This method is easy to test and works well when unimodality holds.

3. Gradient descent when derivatives are available

For smooth multi-dimensional objectives, gradient descent can converge quickly if step size is chosen carefully. Start with a conservative learning rate and stop using gradient norm or function-change thresholds.

1import numpy as np
2
3
4def grad_f(x: np.ndarray) -> np.ndarray:
5    # f(x, y) = (x - 1)^2 + 2(y + 2)^2
6    return np.array([2 * (x[0] - 1), 4 * (x[1] + 2)], dtype=float)
7
8
9def gradient_descent(x0: np.ndarray, lr: float = 0.1, steps: int = 200) -> np.ndarray:
10    x = x0.astype(float)
11    for _ in range(steps):
12        x -= lr * grad_f(x)
13    return x
14
15
16minimum = gradient_descent(np.array([8.0, -10.0]))
17print(minimum)

This example is intentionally simple. Real systems should include line search or adaptive step methods to reduce sensitivity.

4. Numerical stability and stopping criteria

Optimization code fails most often because stopping rules are weak. Good stopping rules combine multiple checks:

• maximum iterations
• small gradient norm
• small change in function value
• small change in parameter vector

Use all of them together, then log the reason for termination. Logging termination reason helps diagnose whether the optimizer truly converged or just hit iteration limits.

5. Validation strategy

Validate on functions with known analytic minima first. Then test noisy functions and piecewise functions that resemble production objectives. Keep regression tests for initialization sensitivity by running multiple starting points and checking spread in final values.

If reproducibility matters, fix random seeds and store optimizer settings with experiment outputs. Without that metadata, comparing model versions becomes difficult.

Common Pitfalls

• Applying local optimization to multimodal functions and assuming global optimality.
• Using a fixed learning rate that is too large, causing oscillation or divergence.
• Relying on a single stopping condition and misclassifying incomplete runs as converged.
• Ignoring initialization sensitivity in non-convex problems.
• Benchmarking only on smooth toy functions and skipping real noisy objectives.

Summary

• Algorithm selection should follow function assumptions, not habit.
• Ternary search is reliable for one-dimensional unimodal intervals.
• Gradient methods are effective for smooth problems with good step control.
• Convergence logic must include robust stopping criteria and termination logging.
• Validation on known and realistic objectives builds trust before deployment.