Introduction

Usually, yes: the hidden state is reset between mini-batches unless you explicitly carry it forward. That default is important because most training pipelines treat each batch as an independent collection of sequences, not as one continuous stream.

The Default Mental Model

Inside a single sequence, an RNN updates hidden state from one time step to the next. Across different mini-batches, frameworks generally do not assume continuity unless you ask for it.

So there are two separate questions:

• Does the state persist across time steps within one forward pass? Yes.
• Does the state automatically persist across later mini-batches? Usually no.

That distinction explains a lot of confusion.

Keras: Stateless Versus Stateful

In Keras, the default behavior is stateless. Each batch starts with a fresh initial state unless you pass one manually.

1import numpy as np
2from tensorflow import keras
3
4x = np.random.randn(4, 5, 3).astype("float32")
5
6model = keras.Sequential([
7    keras.layers.SimpleRNN(8, input_shape=(5, 3)),
8    keras.layers.Dense(1),
9])
10
11y = model(x)
12print(y.shape)

Here the RNN processes each sample across 5 time steps, but it does not carry hidden state from one batch of 4 samples into the next batch.

If you set stateful=True, Keras keeps state across batches for the same sample index position:

1import numpy as np
2from tensorflow import keras
3
4layer = keras.layers.SimpleRNN(4, stateful=True, batch_input_shape=(2, 3, 1))
5model = keras.Sequential([layer])
6
7x1 = np.ones((2, 3, 1), dtype="float32")
8x2 = np.ones((2, 3, 1), dtype="float32")
9
10out1 = model(x1)
11out2 = model(x2)
12print(out1.numpy())
13print(out2.numpy())
14
15model.reset_states()

With stateful=True, batch ordering matters. Sample position 0 in batch 2 receives the carried state from sample position 0 in batch 1.

PyTorch: Explicit State Passing

PyTorch is explicit about this. The RNN returns the final hidden state, and if you want continuity you pass that state into the next call.

1import torch
2
3rnn = torch.nn.RNN(input_size=3, hidden_size=5, batch_first=True)
4x1 = torch.randn(2, 4, 3)
5x2 = torch.randn(2, 4, 3)
6
7out1, h1 = rnn(x1)
8out2, h2 = rnn(x2, h1)
9
10print(out1.shape, out2.shape)

If you omit h1 in the second call, PyTorch starts from zeros by default.

That makes the rule very clear: persistence between mini-batches happens only when your code chooses it.

When Carrying State Is Useful

State carryover is useful when a long sequence is chopped into chunks for efficiency. Language modeling and streaming time-series tasks often work that way.

Example: if one logical sequence is 10,000 time steps long, you may train on chunks of 100 time steps. In that setup, passing the final state of chunk 1 into chunk 2 preserves continuity without forcing one giant forward pass.

But once you do that, you usually also need truncated backpropagation through time. In PyTorch, that often means detaching the hidden state:

out, hidden = rnn(x1)
hidden = hidden.detach()
out2, hidden = rnn(x2, hidden)

Without detaching, the computation graph grows across chunks and memory use becomes a problem.

When Resetting Is The Right Choice

If each training example is independent, resetting state between batches is correct. Carrying state across unrelated samples leaks information and makes the training signal invalid.

That is why the default reset behavior is sensible for classification, many forecasting tasks with separate windows, and datasets where batches are shuffled.

Common Pitfalls

The biggest mistake is enabling stateful behavior while still shuffling data randomly. If the next batch is unrelated to the previous one, carried state is harmful, not helpful.

Another common error is forgetting that stateful Keras models assume a fixed batch size and consistent sample ordering. If either changes, the mapping of states to samples breaks.

In PyTorch, developers sometimes pass hidden state across chunks but forget to detach it. That causes unnecessary graph growth and can lead to memory issues.

Finally, do not confuse resetting between batches with resetting between epochs. Those are separate choices in your training loop.

Summary

• By default, RNN state is usually reset between mini-batches.
• Keras keeps state across batches only if you use stateful=True or pass state manually.
• PyTorch keeps state only when you pass the returned hidden state into the next call.
• Carry state forward only when batches represent consecutive chunks of the same logical sequence.
• If batches are independent, resetting state is the correct behavior.