A feedforward neural network processes one input and produces one output. It has no memory. You cannot ask it "what did you see three inputs ago?" because by design, every input is treated independently. This works for classifying images or tabular data, but it fails for text. The meaning of "bank" depends on what came before it. The grammatical form of a verb depends on the subject earlier in the sentence. Language is inherently sequential.

The Recurrent Neural Network (RNN) solves this by maintaining a hidden state , a vector that summarizes everything the network has seen so far. At each time step $t$, the RNN takes two inputs: the current token $x_t$ and the previous hidden state $h_{t-1}$. It produces one output: the new hidden state $h_t$. The hidden state is then passed forward to the next time step, carrying information about the history of the sequence.

In NumPy, the entire forward step is one line of linear algebra:

1import numpy as np
2
3def rnn_step(h_prev, x, W_h, W_x, b):
4    """One time step of a vanilla RNN."""
5    h = np.tanh(W_h @ h_prev + W_x @ x + b)
6    return h

The same weight matrices $W_h$ and $W_x$ are used at every time step. This weight sharing is what makes an RNN a recurrence, the same transformation is applied to each input in the context of the accumulated history. A sentence of length 100 passes through 100 applications of the same set of weights.

To visualize an RNN, "unroll" it through time: draw one copy of the cell for each time step, connected by arrows representing the hidden state flow. The unrolled view makes it clear that an RNN over a sequence of length $T$ is really a very deep feedforward network, $T$ layers deep, where all layers share the same weights. This weight sharing is what gives RNNs parameter efficiency: a sentence of length 100 uses the same number of parameters as a sentence of length 10.

Bidirectional RNNs

Standard RNNs process sequences left to right, the hidden state at position $t$ only sees positions $1$ through $t$. For tasks like named entity recognition, where knowing what comes after a word is as important as what came before it, a bidirectional RNN runs two separate RNNs: one forward and one backward over the same sequence. The two hidden states at each position are concatenated to form the final representation.

The bidirectional design doubles the parameter count but dramatically improves performance on understanding tasks. It is impractical for generation tasks (you cannot process future tokens before generating the current one), but for classification or sequence labeling, it is the standard extension of vanilla RNNs.

Teacher Forcing

Training a sequence-to-sequence RNN requires deciding what to feed as input at each time step during training. The model's own generated output from the previous step is often wrong early in training, feeding wrong tokens into subsequent steps would compound errors and make learning very slow. Teacher forcing instead feeds the ground-truth previous token at each step during training, regardless of what the model predicted. This makes training stable but creates an exposure bias: the model never sees its own mistakes during training, then sees them at inference time. Scheduled sampling addresses this by gradually replacing ground-truth tokens with model-predicted tokens during training as training progresses.