Introduction

PyTorch autograd is often described as tape-based because it records the operations executed during the forward pass and uses that recorded history to compute gradients during the backward pass. The key idea is dynamic graph construction: the graph is built as your Python code runs, not frozen ahead of time.

What the "Tape" Metaphor Means

Imagine every differentiable tensor operation being written onto a tape:

1. create tensors that require gradients
2. apply operations to them
3. store enough information to compute local derivatives later
4. walk backward through that recorded chain when backward() is called

That recording is what makes automatic differentiation work without you manually deriving every gradient formula.

A Small Example

1import torch
2
3x = torch.tensor([2.0], requires_grad=True)
4y = x * x + 3 * x
5
6print(y)
7print(y.grad_fn)
8
9y.backward()
10print(x.grad)

Output looks like this conceptually:

tensor([10.], grad_fn=...)
tensor([7.])

The derivative of x^2 + 3x at x = 2 is 2x + 3 = 7, and autograd computes it automatically.

Dynamic Graphs Are the Point

PyTorch builds the autograd graph on the fly as operations execute. That is why control flow in regular Python works naturally:

1import torch
2
3def f(x):
4    if x.item() > 0:
5        return x * x
6    return x * 3
7
8x = torch.tensor([2.0], requires_grad=True)
9y = f(x)
10y.backward()
11print(x.grad)

Only the operations that actually ran are part of the graph. This dynamic behavior is one of the main reasons PyTorch is easy to experiment with.

What Gets Stored

Tensors that require gradients track:

• whether gradient computation is enabled
• the operation that created them
• references needed for backward propagation

That is why you can inspect grad_fn on non-leaf tensors. It points to the autograd function associated with the recorded operation.

Leaf tensors such as model parameters accumulate gradients in their .grad field after backward().

Why the Graph Usually Disappears After Backward

By default, PyTorch frees the graph after backward to save memory. If you need to backpropagate through the same graph more than once, you must request retention explicitly:

y.backward(retain_graph=True)

That behavior is important because autograd graphs can become large. The default design favors memory efficiency.

Tape-Based Does Not Mean "Always Stored Forever"

The tape metaphor is helpful, but it is not a literal permanent recording. PyTorch stores the information needed for differentiation during the forward pass and then consumes that information during backward unless you tell it to retain the graph.

So the real mental model is:

• build graph dynamically
• use it for gradient computation
• release it when no longer needed

Common Pitfalls

The biggest mistake is using in-place operations on tensors that autograd still needs. That can destroy values required for gradient computation and trigger runtime errors.

Another mistake is expecting .grad to be populated on every intermediate tensor. By default, gradients are accumulated on leaf tensors, such as parameters, not every temporary value.

A third issue is forgetting to clear gradients between optimizer steps. In PyTorch, gradients accumulate unless you call optimizer.zero_grad() or an equivalent reset.

Summary

• PyTorch autograd is tape-based in the sense that it records executed operations during the forward pass.
• The graph is dynamic and built as Python code runs.
• 'backward() traverses that graph in reverse and applies the chain rule.'
• Graphs are usually freed after backward unless retained explicitly.
• Understanding leaf tensors, grad_fn, and in-place-operation risks is essential for correct autograd use.