Introduction

Gradient descent is a fundamental optimization technique widely used in machine learning to find the minimum of a function. It is particularly useful in linear regression for adjusting the parameters to minimize the cost function. This article focuses on the process of simultaneously updating the parameters $\theta_0$ and $\theta_1$ in gradient descent. We will explore how to implement this in Python and provide an in-depth understanding of the method's mechanics.

Gradient Descent Basics

Gradient descent works by iteratively updating parameters in the opposite direction of the gradient of the cost function. The cost function, in the case of linear regression, is typically the Mean Squared Error (MSE). The parameters, $\theta_0$ and $\theta_1$, represent the intercept and the slope of the regression line, respectively.

For a hypothesis function $h(x) = \theta_0 + \theta_1 x$, the cost function $J(\theta_0, \theta_1)$ is expressed as:

$$J(\theta_0, \theta_1) = \frac{1}{2m} \sum_{i=1}^{m} (h(x^{(i)}) - y^{(i)})^2$$

where $m$ is the number of training examples, $x^{(i)}$ is the input, and $y^{(i)}$ is the actual output.

Partial Derivatives

To update the parameters, we need the partial derivatives of $J(\theta_0, \theta_1)$ with respect to $\theta_0$ and $\theta_1$:

1. $\frac{\partial}{\partial \theta_0} J(\theta_0, \theta_1) = \frac{1}{m} \sum_{i=1}^{m} (h(x^{(i)}) - y^{(i)})$
2. $\frac{\partial}{\partial \theta_1} J(\theta_0, \theta_1) = \frac{1}{m} \sum_{i=1}^{m} (h(x^{(i)}) - y^{(i)}) x^{(i)}$

The gradient descent update rules are as follows:

$$\theta_0 := \theta_0 - \alpha \frac{\partial}{\partial \theta_0} J(\theta_0, \theta_1)$$

$$\theta_1 := \theta_1 - \alpha \frac{\partial}{\partial \theta_1} J(\theta_0, \theta_1)$$

where $\alpha$ is the learning rate.

Why Simultaneous Update Matters

To ensure the correctness of the gradient descent algorithm, the updates to $\theta_0$ and $\theta_1$ must be simultaneous. This means you compute both partial derivatives using the current values of $\theta_0$ and $\theta_1$ before changing either one. If you update $\theta_0$ first and then use the new $\theta_0$ to compute the gradient for $\theta_1$, the result will be incorrect because the gradient for $\theta_1$ should have been evaluated at the original parameter values.

Python Implementation

Below is a Python implementation showcasing how to simultaneously update $\theta_0$ and $\theta_1$ in gradient descent.

1import numpy as np
2
3def gradient_descent(X, y, alpha=0.01, iterations=1000):
4    m = len(y)
5    theta0 = 0.0
6    theta1 = 0.0
7    cost_history = []
8
9    for _ in range(iterations):
10        # Predictions with current parameters
11        predictions = theta0 + theta1 * X
12
13        # Compute errors
14        errors = predictions - y
15
16        # Compute gradients (using current theta0 and theta1)
17        grad0 = (1 / m) * np.sum(errors)
18        grad1 = (1 / m) * np.sum(errors * X)
19
20        # Simultaneous update: both use the OLD values
21        theta0 = theta0 - alpha * grad0
22        theta1 = theta1 - alpha * grad1
23
24        # Track cost
25        cost = (1 / (2 * m)) * np.sum(errors ** 2)
26        cost_history.append(cost)
27
28    return theta0, theta1, cost_history

The key detail is on the lines computing grad0 and grad1. Both gradients are calculated from the same (unmodified) theta0 and theta1. Only after both gradients are ready do we apply the updates. This is the simultaneous update pattern.

Incorrect (Non-Simultaneous) Update

For contrast, here is the wrong approach where $\theta_0$ is updated before computing the gradient for $\theta_1$:

1# WRONG: non-simultaneous update
2theta0 = theta0 - alpha * grad0   # theta0 changes here
3grad1 = (1/m) * np.sum((theta0 + theta1 * X - y) * X)  # uses new theta0!
4theta1 = theta1 - alpha * grad1

This bug is subtle because gradient descent may still converge, but it will follow a different (suboptimal) path and may converge to the wrong minimum or take many more iterations.

Practical Considerations

• Initialization: $\theta_0$ and $\theta_1$ are initialized to zeros. Random initialization also works but zero is standard for linear regression.
• Cost History: Tracking the cost over iterations helps determine whether the algorithm is converging. A decreasing cost curve is a good sign.
• Learning Rate: A too-small $\alpha$ leads to slow convergence. A too-large $\alpha$ causes the cost to oscillate or diverge. A common debugging technique is to plot cost vs. iteration and adjust $\alpha$ until you see smooth, steady decrease.
• Stopping Criteria: You can stop when the change in cost between iterations drops below a threshold (for example, $10^{-6}$), or after a fixed number of iterations.
• Vectorized Form: In practice, both parameters are stored in a single vector $\theta$ and updated in one step: $\theta := \theta - \alpha \nabla J(\theta)$. NumPy makes this efficient and naturally simultaneous since the gradient vector is computed before any assignment.

Summary

Simultaneous update of $\theta_0$ and $\theta_1$ is not just a stylistic choice. It is a correctness requirement of the gradient descent algorithm. The gradients must be evaluated at the same point in parameter space before any parameter is modified. In Python, this is straightforward: compute all gradients first, then apply all updates. Using NumPy's vectorized operations makes this pattern both natural and efficient.