Introduction

LSTM models are a practical choice when you want a network to learn patterns from ordered values such as sensor readings, traffic counts, or sampled waveforms. In PyTorch, sequence generation usually means training on short windows of past values and then asking the model to predict the next value repeatedly.

Preparing Sequential Training Data

An LSTM does not read a whole time series as one flat vector. It expects steps over time, so the first job is to convert raw data into fixed-length windows. Each training example contains a short input sequence and the target value that follows it.

The example below builds a simple sine wave dataset. The helper returns tensors shaped for batch_first=True, so each input has the form batch, time, features.

1import math
2import torch
3
4
5def make_series(length: int = 400):
6    values = [math.sin(i * 0.05) for i in range(length)]
7    return torch.tensor(values, dtype=torch.float32)
8
9
10def make_windows(series: torch.Tensor, window_size: int = 20):
11    xs = []
12    ys = []
13    for i in range(len(series) - window_size):
14        xs.append(series[i:i + window_size].unsqueeze(-1))
15        ys.append(series[i + window_size])
16    x = torch.stack(xs)
17    y = torch.stack(ys).unsqueeze(-1)
18    return x, y
19
20
21series = make_series()
22x_train, y_train = make_windows(series, window_size=20)
23print(x_train.shape)  # torch.Size([380, 20, 1])
24print(y_train.shape)  # torch.Size([380, 1])

Two details matter here. First, the model predicts one step ahead, not an entire future sequence at once. Second, scaling is often useful for real data. A sine wave is already small and smooth, but measured business or industrial data often needs normalization before training.

Building the LSTM Model

In PyTorch, nn.LSTM handles the recurrent state updates for you. A small forecasting model usually consists of an LSTM layer followed by a linear layer that maps the last hidden state to the next numeric value.

1import torch.nn as nn
2
3
4class LSTMForecaster(nn.Module):
5    def __init__(self, input_size=1, hidden_size=32, num_layers=1):
6        super().__init__()
7        self.lstm = nn.LSTM(
8            input_size=input_size,
9            hidden_size=hidden_size,
10            num_layers=num_layers,
11            batch_first=True,
12        )
13        self.output = nn.Linear(hidden_size, 1)
14
15    def forward(self, x):
16        lstm_out, _ = self.lstm(x)
17        last_step = lstm_out[:, -1, :]
18        return self.output(last_step)

This model is intentionally small. For many sequence problems, a larger hidden state, more layers, or dropout may help, but simple models are easier to debug. If a tiny model cannot learn a clean synthetic series, the issue is usually in the data pipeline or tensor shapes rather than model capacity.

Training and Generating New Values

Training is standard supervised learning. Feed a window in, compare the predicted next value against the true next value, and minimize mean squared error. After training, generation becomes autoregressive: take the latest window, predict one value, append it, and repeat.

1model = LSTMForecaster()
2loss_fn = nn.MSELoss()
3optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
4
5for epoch in range(200):
6    prediction = model(x_train)
7    loss = loss_fn(prediction, y_train)
8
9    optimizer.zero_grad()
10    loss.backward()
11    optimizer.step()
12
13seed = x_train[0:1].clone()
14generated = []
15
16with torch.no_grad():
17    for _ in range(10):
18        next_value = model(seed)
19        generated.append(next_value.item())
20        next_step = next_value.unsqueeze(1)
21        seed = torch.cat([seed[:, 1:, :], next_step], dim=1)
22
23print(generated)

The generation loop is where many beginners get lost. The model only predicts one next point at a time, so you must feed its output back into the input window. That feedback is what turns a one-step forecaster into a sequence generator.

For real projects, split the data into train and validation sets. If validation error rises while training error falls, the model is starting to memorize the training windows instead of learning a pattern that generalizes.

Common Pitfalls

One frequent mistake is passing tensors with the wrong shape. With batch_first=True, the LSTM expects batch, time, features. If you pass time, batch, features, training may fail or silently learn the wrong mapping.

Another common issue is forgetting that long generated sequences accumulate error. Even if one-step predictions look good, repeated feedback can drift. Reducing that drift may require better scaling, more representative training windows, or training on noisier data.

It is also easy to confuse sequence generation with sequence classification. A classifier uses the final hidden state to predict a label. A generator uses that state to predict the next numeric step and then rolls forward.

Summary

• Use sliding windows to convert a raw time series into supervised training samples.
• In PyTorch, nn.LSTM plus a linear output layer is enough for basic one-step forecasting.
• Sequence generation is autoregressive: predict one value, append it to the window, and repeat.
• Check tensor shapes early, because many LSTM bugs come from incorrect batch, time, features ordering.
• Evaluate both one-step accuracy and multi-step drift before trusting generated sequences.