Introduction

People often expect gradient descent to be a large mysterious subsystem hidden somewhere deep in a machine learning library. In reality, the core idea is tiny: compute a gradient, multiply it by a learning rate, and subtract that update from the parameter.

The Core Update Rule

At its simplest, gradient descent is just:

parameter = parameter minus learning rate times gradient

If you write it by hand, the code is short enough to fit on one screen:

1w = 5.0
2learning_rate = 0.1
3
4for _ in range(5):
5    grad = 2 * (w - 3)   # derivative of (w - 3)^2
6    w = w - learning_rate * grad
7    print(round(w, 4))

The actual "gradient descent code" is the line:

w = w - learning_rate * grad

Everything else is setup around computing the gradient and repeating the update.

Why It Feels Hidden in Frameworks

Modern libraries separate training into several steps:

• forward pass
• loss computation
• automatic differentiation
• optimizer update

That separation makes the update rule less visible, even though it is still there. For example, in TensorFlow:

1import tensorflow as tf
2
3x = tf.constant([1.0, 2.0, 3.0])
4y = tf.constant([2.0, 4.0, 6.0])
5w = tf.Variable(0.0)
6optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)
7
8for _ in range(20):
9    with tf.GradientTape() as tape:
10        predictions = w * x
11        loss = tf.reduce_mean(tf.square(predictions - y))
12    grads = tape.gradient(loss, [w])
13    optimizer.apply_gradients(zip(grads, [w]))
14
15print(float(w.numpy()))

Here, the gradient descent step is inside optimizer.apply_gradients(...). The optimizer object hides the arithmetic, but conceptually it is still applying the same parameter update rule.

What Changes Across Variants

Batch gradient descent, stochastic gradient descent, and mini-batch gradient descent do not change the central rule. They mainly change how much data is used to compute each gradient.

For example, a stochastic-style update might look like this:

1import numpy as np
2
3X = np.array([1.0, 2.0, 3.0, 4.0])
4y = np.array([2.0, 4.0, 6.0, 8.0])
5w = 0.0
6lr = 0.05
7
8for x_i, y_i in zip(X, y):
9    grad = 2 * x_i * (w * x_i - y_i)
10    w = w - lr * grad
11
12print(round(w, 4))

The update is still one subtraction step. The only change is how often it happens and how the gradient is estimated.

Where to Look in a Real Codebase

If you are trying to find "the gradient descent code" in a project, search for these signs:

• optimizer creation such as SGD, Adam, or RMSprop
• gradient computation through autograd, tape, or backward passes
• update calls such as step() or apply_gradients()
• training loops that repeatedly change model parameters

That is where the algorithm lives in practice. Beginners often search for the word "gradient" and miss the real update logic hidden behind the optimizer API.

Gradient Descent Versus Other Optimizers

Another source of confusion is that many projects do not use plain gradient descent at all. They use variants such as momentum, RMSprop, or Adam. Those optimizers still rely on gradients, but they add extra state or scaling rules on top of the basic update step.

So if you cannot find a plain "subtract learning rate times gradient" line, it may be because the library is using a richer optimizer that wraps the same core idea.

Common Pitfalls

• Expecting gradient descent to be a huge standalone block of code instead of a small repeated update rule.
• Confusing gradient calculation with the optimizer step that actually changes parameters.
• Searching only for the word "gradient" and overlooking step() or apply_gradients().
• Assuming every optimizer in a framework is plain gradient descent.
• Studying the math without tracing the actual training loop where parameters are updated.

Summary

• Gradient descent is fundamentally "parameter minus learning rate times gradient."
• In handwritten code, the update step is usually only one line.
• Frameworks hide that step behind optimizer APIs for convenience.
• Different variants mostly change how gradients are estimated and applied, not the basic idea.
• To find the code in a real project, look for optimizer updates inside the training loop.