Introduction

By default, PyTorch frees the computation graph after calling .backward() once. Calling .backward() a second time raises a RuntimeError because the intermediate values needed for gradient computation are gone. To compute gradients twice — whether for double backpropagation, second-order optimization, or gradient penalty — you must pass retain_graph=True on the first backward call or use torch.autograd.grad with create_graph=True.

The Error

1import torch
2
3x = torch.tensor([2.0], requires_grad=True)
4y = x ** 3
5y.backward()
6y.backward()  # RuntimeError: Trying to backward through the graph a second time

After the first .backward(), PyTorch releases the graph to save memory. The second call fails because there is no graph left to traverse.

Fix 1: retain_graph=True

Pass retain_graph=True to keep the computation graph alive after the first backward pass:

1x = torch.tensor([2.0], requires_grad=True)
2y = x ** 3
3
4y.backward(retain_graph=True)  # First backward — graph is kept
5print(x.grad)  # tensor([12.]) — dy/dx = 3x^2 = 12
6
7x.grad.zero_()  # Clear accumulated gradients
8y.backward()     # Second backward — now the graph is freed
9print(x.grad)   # tensor([12.])

The graph persists until a backward call without retain_graph=True, which frees it. Always zero out gradients between calls to avoid accumulation.

Fix 2: torch.autograd.grad with create_graph

For second-order derivatives (Hessian, gradient penalty), use torch.autograd.grad with create_graph=True. This builds a new graph of the gradient computation itself, enabling differentiation through it:

1x = torch.tensor([2.0], requires_grad=True)
2y = x ** 3  # y = x^3
3
4# First derivative: dy/dx = 3x^2
5grad_y = torch.autograd.grad(y, x, create_graph=True)[0]
6print(grad_y)  # tensor([12.])
7
8# Second derivative: d^2y/dx^2 = 6x
9grad2_y = torch.autograd.grad(grad_y, x)[0]
10print(grad2_y)  # tensor([12.]) — 6 * 2 = 12

create_graph=True means the first gradient is itself part of a computation graph, so you can differentiate through it again.

Practical Example: Gradient Penalty (WGAN-GP)

Gradient penalties require computing the gradient of the discriminator output with respect to the input, then computing a loss based on the gradient norm:

1def gradient_penalty(discriminator, real, fake, device):
2    batch_size = real.size(0)
3    alpha = torch.rand(batch_size, 1, 1, 1, device=device)
4    interpolated = (alpha * real + (1 - alpha) * fake).requires_grad_(True)
5
6    d_interpolated = discriminator(interpolated)
7
8    gradients = torch.autograd.grad(
9        outputs=d_interpolated,
10        inputs=interpolated,
11        grad_outputs=torch.ones_like(d_interpolated),
12        create_graph=True,   # Needed to backprop through the penalty
13        retain_graph=True,
14    )[0]
15
16    gradients = gradients.view(batch_size, -1)
17    penalty = ((gradients.norm(2, dim=1) - 1) ** 2).mean()
18    return penalty
19
20# In the training loop:
21gp = gradient_penalty(D, real_images, fake_images, device)
22d_loss = d_fake.mean() - d_real.mean() + 10 * gp
23d_loss.backward()  # Backprops through the gradient penalty

Without create_graph=True, the gradient penalty would be treated as a constant and the discriminator would not receive the penalty signal during optimization.

Practical Example: Computing the Hessian

For small models, you can compute the full Hessian matrix:

1x = torch.tensor([1.0, 2.0], requires_grad=True)
2y = x[0]**2 * x[1] + x[1]**3  # y = x0^2 * x1 + x1^3
3
4# First derivatives
5grads = torch.autograd.grad(y, x, create_graph=True)[0]
6# grads[0] = 2*x0*x1 = 4, grads[1] = x0^2 + 3*x1^2 = 13
7
8# Hessian: differentiate each gradient component w.r.t. x
9hessian = []
10for g in grads:
11    row = torch.autograd.grad(g, x, retain_graph=True)[0]
12    hessian.append(row)
13
14H = torch.stack(hessian)
15print(H)
16# tensor([[ 2*x1,  2*x0],    = [[4, 2],
17#          [ 2*x0, 6*x1]])      [2, 12]]

retain_graph vs create_graph

create_graph=True implies retain_graph=True — you do not need to set both.

Memory Considerations

Retaining or creating graphs consumes significantly more memory:

1# Memory-efficient: single backward, graph freed
2loss.backward()
3
4# Memory-heavy: graph retained
5loss.backward(retain_graph=True)
6
7# Most memory: graph of gradients also retained
8grads = torch.autograd.grad(loss, params, create_graph=True)

For large models, prefer computing second-order information with Hessian-vector products instead of the full Hessian to limit memory usage.

Common Pitfalls

• Forgetting to zero gradients between backward calls: PyTorch accumulates gradients by default. Call optimizer.zero_grad() or param.grad.zero_() before each backward pass to avoid adding gradients from the previous call.
• Using retain_graph=True when create_graph=True is needed: retain_graph keeps the original forward graph but does not make the gradient computation differentiable. For second derivatives, you must use create_graph=True.
• Memory leaks from always retaining the graph: If you pass retain_graph=True in a training loop and never do a final backward without it, the graph accumulates indefinitely. Ensure the last backward call frees the graph.
• Detaching when you should not: Calling .detach() on a tensor removes it from the computation graph. If you detach the gradient before computing the second derivative, the second backward pass returns zero.
• In-place operations breaking the graph: Operations like x.add_(1) or x[0] = 5 modify tensors in place and can invalidate the computation graph. Use out-of-place operations (x = x + 1) when you need to backpropagate through the result.

Summary

• PyTorch frees the computation graph after .backward() by default
• Use retain_graph=True to call .backward() multiple times on the same graph
• Use create_graph=True with torch.autograd.grad to compute second-order derivatives
• create_graph=True is required for gradient penalties (WGAN-GP), meta-learning (MAML), and Hessian computation
• Always zero gradients between backward calls to prevent accumulation
• Retaining and creating graphs increases memory usage — free the graph when no longer needed