Introduction

Using an LSTM decoder without teacher forcing means the decoder feeds its own previous prediction back as the next input instead of consuming the ground-truth token at every step. That is exactly what happens at inference time in many sequence models, but it requires a slightly different implementation than the standard training shortcut. In TensorFlow, the practical solution is to write the decoding loop yourself so you control what gets fed back at each step.

What Teacher Forcing Changes

With teacher forcing, the decoder receives the correct previous token during training. That makes optimization easier because the model never has to recover from its own bad predictions inside the training loop.

Without teacher forcing, the decoder runs in open-loop mode:

• predict the next token
• choose a token from that prediction
• feed that chosen token back into the decoder

This is closer to real inference, but training is usually less stable because one early mistake can affect the rest of the sequence.

Build a Decoder That Can Feed Back Its Own Output

The core idea is to use an embedding layer, an LSTMCell, and a final projection layer. At each step, the model predicts logits, chooses the next token with argmax, embeds that token, and continues.

1import tensorflow as tf
2
3
4class GreedyDecoder(tf.keras.Model):
5    def __init__(self, vocab_size, embedding_dim, hidden_dim):
6        super().__init__()
7        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
8        self.cell = tf.keras.layers.LSTMCell(hidden_dim)
9        self.output_layer = tf.keras.layers.Dense(vocab_size)
10
11    def call(self, start_tokens, initial_state, max_length):
12        token_ids = start_tokens
13        state = initial_state
14        outputs = []
15
16        for _ in range(max_length):
17            embedded = self.embedding(token_ids)
18            cell_output, state = self.cell(embedded, state)
19            logits = self.output_layer(cell_output)
20            next_tokens = tf.argmax(logits, axis=-1, output_type=tf.int32)
21            outputs.append(next_tokens)
22            token_ids = next_tokens
23
24        return tf.stack(outputs, axis=1)
25
26
27batch_size = 2
28hidden_dim = 16
29vocab_size = 20
30
31decoder = GreedyDecoder(vocab_size=vocab_size, embedding_dim=8, hidden_dim=hidden_dim)
32start_tokens = tf.constant([1, 1], dtype=tf.int32)
33initial_state = [
34    tf.zeros((batch_size, hidden_dim)),
35    tf.zeros((batch_size, hidden_dim)),
36]
37
38predicted_sequences = decoder(start_tokens, initial_state, max_length=5)
39print(predicted_sequences)

This decoder runs without teacher forcing because each step uses the previously predicted token, not a token from the target sequence.

Training Without Teacher Forcing

Inference-only decoding is easy, but training without teacher forcing is where the design choice matters. You usually need a custom training loop so the decoder can generate the next input token from its own output.

1import tensorflow as tf
2
3loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
4optimizer = tf.keras.optimizers.Adam()
5
6
7@tf.function
8def train_step(decoder, start_tokens, target_tokens, initial_state):
9    with tf.GradientTape() as tape:
10        token_ids = start_tokens
11        state = initial_state
12        total_loss = 0.0
13
14        for t in tf.range(tf.shape(target_tokens)[1]):
15            embedded = decoder.embedding(token_ids)
16            cell_output, state = decoder.cell(embedded, state)
17            logits = decoder.output_layer(cell_output)
18            total_loss += loss_fn(target_tokens[:, t], logits)
19            token_ids = tf.argmax(logits, axis=-1, output_type=tf.int32)
20
21        mean_loss = total_loss / tf.cast(tf.shape(target_tokens)[1], tf.float32)
22
23    gradients = tape.gradient(mean_loss, decoder.trainable_variables)
24    optimizer.apply_gradients(zip(gradients, decoder.trainable_variables))
25    return mean_loss

This is true open-loop training. The decoder must live with its own predictions at each step.

Why People Often Mix Training Modes

Pure no-teacher-forcing training can work, but it is often harder to optimize than teacher forcing. Many production systems use a compromise:

• full teacher forcing early in training
• partial teacher forcing later
• inference with no teacher forcing

This family of ideas is often called scheduled sampling or curriculum-style training. The reason is simple: teacher forcing helps the model learn local next-token predictions, while open-loop decoding teaches it to recover from its own errors.

If you jump directly to pure open-loop training, convergence may be slow or unstable.

A Simpler Inference-Only Pattern

Sometimes the real question is not "how do I train without teacher forcing," but "how do I run the decoder without teacher forcing at prediction time." In that case, you can train normally with teacher forcing and write a separate autoregressive decoding method for inference. That is often the most practical design because it gives easier training and realistic decoding at serving time.

The TensorFlow text generation tutorials follow this general idea: training may use teacher forcing, while custom generation code feeds back model outputs step by step.

Common Pitfalls

One common mistake is assuming that setting return_sequences=True on an LSTM automatically creates an autoregressive decoder. It does not. You still need explicit logic that decides what token is fed into the next step.

Another mistake is training with teacher forcing and expecting the same computation graph to behave like open-loop decoding during inference. The decoder input path is different, so you usually need a dedicated decoding routine.

Developers also sometimes feed raw logits back into the embedding layer. That is incorrect. The decoder should feed token IDs, or in more advanced setups, sampled token choices, back into the next embedding lookup.

Finally, if open-loop training becomes unstable, do not assume the architecture is wrong immediately. The training regime itself is harder, so a mixed strategy may be the better engineering choice.

Summary

• A decoder without teacher forcing feeds back its own predicted token at each step.
• In TensorFlow, this usually means writing the decoding loop manually.
• True open-loop training often needs a custom training step.
• Many systems train with teacher forcing and decode without it during inference.
• If pure no-teacher-forcing training is unstable, a mixed schedule is often more practical.