Introduction

torch.optim.SGD does not have a batch-size setting and therefore has no default batch size of its own. The optimizer simply updates parameters using whatever gradients your training loop has accumulated. In practice, batch size comes from the data pipeline, most often from DataLoader.

Where Batch Size Actually Comes From

In a typical PyTorch training loop, batching happens before the optimizer step. The loader yields a batch, the model computes a loss on that batch, and loss.backward() accumulates gradients for that batch size.

1import torch
2from torch import nn
3from torch.utils.data import DataLoader, TensorDataset
4
5features = torch.randn(100, 10)
6targets = torch.randn(100, 1)
7
8dataset = TensorDataset(features, targets)
9loader = DataLoader(dataset, batch_size=32, shuffle=True)
10
11model = nn.Linear(10, 1)
12optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
13loss_fn = nn.MSELoss()
14
15for batch_x, batch_y in loader:
16    optimizer.zero_grad()
17    predictions = model(batch_x)
18    loss = loss_fn(predictions, batch_y)
19    loss.backward()
20    optimizer.step()

Here, the effective batch size is 32 because that is what the DataLoader emits. The optimizer does not override or redefine that number.

What If You Do Not Set batch_size

If you use DataLoader and omit the batch_size argument, PyTorch defaults the loader batch size to 1. That can create confusion, because people sometimes see single-sample updates and assume the optimizer imposed that behavior. It did not. The loader did.

There is also no requirement to use a DataLoader at all. You can build batches manually, use full-batch training, or accumulate gradients across several mini-batches before stepping the optimizer. SGD is agnostic to all of that.

Why This Distinction Matters

Batch size affects memory use, gradient noise, and update frequency. Learning rate, momentum, and weight decay affect how the optimizer transforms those gradients into parameter updates. Mixing those responsibilities conceptually makes tuning harder.

Once you separate them, debugging becomes clearer:

• change batch_size in the input pipeline
• change lr, momentum, or weight_decay in the optimizer
• change gradient accumulation logic in the training loop

Those are related decisions, but they live in different parts of the code.

Gradient Accumulation Is Not a Hidden Batch Size

Some training loops simulate a larger effective batch size by accumulating gradients over several mini-batches before calling optimizer.step().

1accumulation_steps = 4
2
3for step, (batch_x, batch_y) in enumerate(loader, start=1):
4    predictions = model(batch_x)
5    loss = loss_fn(predictions, batch_y) / accumulation_steps
6    loss.backward()
7
8    if step % accumulation_steps == 0:
9        optimizer.step()
10        optimizer.zero_grad()

In this pattern, the optimizer still has no built-in batch size. The training loop is controlling the effective update size by deciding when gradients are applied.

Common Pitfalls

The most common mistake is looking for a batch_size argument on torch.optim.SGD. It is not there because batching is not the optimizer's job.

Another mistake is forgetting that DataLoader defaults to 1 when batch_size is omitted. That can make training unexpectedly slow and noisy if you assumed mini-batches were happening automatically.

Be careful when comparing experiments. If you change batch size, you may also need to revisit the learning rate and training schedule. Those settings interact, but they are still different knobs.

Finally, do not confuse batch size with dataset size or epoch length. Batch size is only the number of samples used for one gradient computation step.

Summary

• 'torch.optim.SGD has no default batch size because it does not own batching.'
• Batch size normally comes from DataLoader, where the default is 1 if you omit it.
• The optimizer only uses the gradients produced by your training loop.
• Gradient accumulation changes the effective update size without changing the optimizer API.
• Tune batch size and optimizer hyperparameters separately so your training logic stays clear.