Introduction

Keras does not provide a separate optimizer called “full gradient descent.” Instead, full gradient descent is a training regime where each parameter update uses the entire training dataset. In Keras, the usual way to get that behavior is to make the batch size equal to the full dataset size so one optimizer step happens per epoch.

What Full Gradient Descent Means

There are three common training regimes:

• stochastic gradient descent: one sample per update
• mini-batch gradient descent: a subset per update
• full or batch gradient descent: the entire dataset per update

Full gradient descent computes one gradient from all training examples, then updates the weights once.

This makes the update stable and low-noise, but it can be slow and memory-heavy on large datasets. That is why modern deep learning more often uses mini-batches.

The Simple Keras Way: Set batch_size to the Dataset Size

If the entire training set fits comfortably in memory, you can approximate full gradient descent by setting the batch size to the number of training samples.

1import numpy as np
2import tensorflow as tf
3
4X = np.random.randn(1000, 20).astype("float32")
5y = np.random.randint(0, 2, size=(1000, 1)).astype("float32")
6
7model = tf.keras.Sequential([
8    tf.keras.layers.Dense(16, activation="relu", input_shape=(20,)),
9    tf.keras.layers.Dense(1, activation="sigmoid")
10])
11
12model.compile(optimizer="sgd", loss="binary_crossentropy")
13model.fit(X, y, epochs=10, batch_size=len(X), shuffle=False)

With batch_size=len(X), each epoch performs exactly one optimizer update using the full dataset.

Why shuffle=False Is Often a Good Idea Here

For full gradient descent, shuffling matters less than in mini-batch training because the entire dataset is used in each update. Setting shuffle=False can make the training behavior slightly easier to reason about and reproduce.

That said, if the full dataset is used every time, shuffling does not change the computed batch composition, only the internal order presented to the pipeline.

Full Gradient Descent Is a Training Schedule, Not a Different Optimizer

People sometimes think they need a special optimizer implementation for full gradient descent. Usually they do not. The optimizer can still be SGD, Adam, RMSprop, or something else. What changes is the batch size and therefore how often the optimizer applies updates.

For example, this is still full-batch training with Adam:

model.compile(optimizer="adam", loss="binary_crossentropy")
model.fit(X, y, epochs=10, batch_size=len(X), shuffle=False)

Whether that is a good idea depends on the problem, but it shows that “full gradient descent” is about update frequency and data usage, not about a dedicated optimizer class.

A Custom Training Loop Makes the Behavior Explicit

If you want complete control, you can write one training step per epoch yourself.

1import tensorflow as tf
2
3optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
4loss_fn = tf.keras.losses.BinaryCrossentropy()
5
6for epoch in range(5):
7    with tf.GradientTape() as tape:
8        predictions = model(X, training=True)
9        loss = loss_fn(y, predictions)
10
11    grads = tape.gradient(loss, model.trainable_variables)
12    optimizer.apply_gradients(zip(grads, model.trainable_variables))
13    print(f"epoch={epoch} loss={float(loss):.4f}")

This makes the one-update-per-full-dataset behavior completely explicit.

Why Full Gradient Descent Is Rare in Deep Learning Practice

For large neural networks, full gradient descent is often inefficient because:

• each update is expensive
• memory requirements can be high
• mini-batch noise can actually help optimization
• GPUs and training pipelines are usually optimized around batches, not one giant step

So while full-batch training is easy to implement, it is usually better for small datasets, demonstrations, or experiments where exact gradient behavior matters more than raw training efficiency.

Use It Deliberately, Not by Accident

Setting batch_size=len(X) on a very large dataset can slow training significantly or even fail due to memory pressure. Full gradient descent should therefore be a conscious choice, not just something copied from an example without considering dataset size.

If the dataset is large, mini-batches are usually the practical solution.

Common Pitfalls

The most common mistake is assuming Keras has a separate built-in “full gradient descent” optimizer. It does not.

Another mistake is setting the batch size to the full dataset without considering whether the data even fits comfortably in memory.

Developers also confuse the optimizer choice with the batching regime. Full gradient descent is about using the full dataset per update, not about switching to a special optimizer class.

Summary

• Full gradient descent in Keras usually means one optimizer update per epoch using the entire dataset.
• The easiest way to get that behavior is batch_size=len(X).
• The optimizer can still be SGD, Adam, or another optimizer; the key change is the batch regime.
• A custom training loop can make the full-batch update pattern explicit.
• Full gradient descent is usually practical only for smaller datasets or controlled experiments.