Introduction

For a many-to-many time-series classifier, the model should emit one prediction per timestep rather than one prediction for the entire sequence. In TensorFlow, that usually means an RNN layer with return_sequences=True, followed by a sigmoid output applied across the sequence so each timestep gets its own binary score.

What Many-to-Many Means Here

Suppose each input sequence has shape (timesteps, features) and each timestep also has a binary label. Then the target shape is not one label per sequence. It is one label per timestep.

Typical shapes:

• input: (batch, timesteps, features)
• target: (batch, timesteps, 1)

That is different from many-to-one classification, where the target might be only (batch, 1).

Build an RNN That Returns a Sequence

The most important configuration is return_sequences=True. Without it, the RNN outputs only the final hidden state, which is wrong for timestep-level labels.

1import tensorflow as tf
2import numpy as np
3
4timesteps = 20
5features = 3
6
7model = tf.keras.Sequential([
8    tf.keras.layers.Input(shape=(timesteps, features)),
9    tf.keras.layers.LSTM(16, return_sequences=True),
10    tf.keras.layers.Dense(1, activation="sigmoid")
11])
12
13model.compile(
14    optimizer="adam",
15    loss="binary_crossentropy",
16    metrics=["accuracy"]
17)
18
19model.summary()

The final Dense(1, activation="sigmoid") is applied at every timestep because the input to the dense layer is a 3D tensor. Keras handles that across the last axis automatically.

Train on Sequence Labels

Now create data where every timestep has its own label. The example below marks a timestep as 1 when the first feature is positive.

1import numpy as np
2
3num_samples = 200
4x = np.random.randn(num_samples, timesteps, features).astype("float32")
5y = (x[:, :, 0] > 0).astype("float32")[..., np.newaxis]
6
7history = model.fit(
8    x,
9    y,
10    epochs=3,
11    batch_size=16,
12    validation_split=0.2,
13    verbose=0
14)
15
16print(history.history["loss"][-1])

This is a simple synthetic dataset, but it demonstrates the correct shape relationship between inputs and labels.

Inspect Per-Timestep Predictions

Inference returns a probability for each timestep.

1pred = model.predict(x[:1], verbose=0)
2
3print(pred.shape)
4print(np.round(pred[0, :5, 0], 3))

The output shape is (1, timesteps, 1), which is exactly what you want for many-to-many binary classification.

To turn probabilities into binary labels:

binary_pred = (pred >= 0.5).astype("int32")
print(binary_pred[0, :5, 0])

Padding and Masking

Real time-series datasets often contain sequences of different lengths. In that case, pad them to a common length and use masking so the model does not treat padding as real timesteps.

1model = tf.keras.Sequential([
2    tf.keras.layers.Input(shape=(None, features)),
3    tf.keras.layers.Masking(mask_value=0.0),
4    tf.keras.layers.GRU(16, return_sequences=True),
5    tf.keras.layers.Dense(1, activation="sigmoid")
6])

If you pad the labels too, be careful during loss computation. The padded timesteps should not influence the metric the same way real timesteps do.

When Bidirectional Layers Help

If labels depend on both past and future context, a bidirectional RNN can improve performance.

1model = tf.keras.Sequential([
2    tf.keras.layers.Input(shape=(timesteps, features)),
3    tf.keras.layers.Bidirectional(
4        tf.keras.layers.LSTM(16, return_sequences=True)
5    ),
6    tf.keras.layers.Dense(1, activation="sigmoid")
7])

This is useful for offline labeling tasks, but less suitable when predictions must be causal and available in real time.

Common Pitfalls

• Forgetting return_sequences=True and accidentally producing one output per sequence instead of one per timestep.
• Shaping labels as (batch, 1) when the task actually needs (batch, timesteps, 1).
• Using the wrong loss for binary timestep labels.
• Ignoring padding and letting padded timesteps distort training.
• Expecting the model to be causal when a bidirectional layer uses future context by design.

Summary

• For timestep-level binary labels, use a many-to-many RNN with return_sequences=True.
• Keep target labels aligned to the sequence shape, usually (batch, timesteps, 1).
• A sigmoid dense layer on top of the RNN gives one binary probability per timestep.
• Use masking when sequences are padded.
• Choose bidirectional layers only when future context is allowed by the task.