Introduction

When the input is a sequence of sentences rather than a single flat sequence of tokens, you need to model two levels of structure: words inside each sentence and sentences inside the larger document or conversation. In Keras, that usually means building a hierarchical model instead of flattening everything into one long token list.

Why Sentence Sequences Need Hierarchical Structure

If you collapse an entire document into one token stream, the model loses the explicit boundary between sentences. That can be acceptable for some tasks, but for document classification, conversation analysis, or next-sentence style tasks, sentence structure often matters.

A common representation is:

• 'batch'
• 'num_sentences'
• 'num_words_per_sentence'

So each training example is a 2D block of token ids, and the dataset as a whole is 3D.

Preparing the Data

Here is a toy dataset where each sample contains three sentences and each sentence is padded to four tokens:

1import numpy as np
2
3x = np.array([
4    [[1, 2, 3, 0], [4, 5, 0, 0], [6, 7, 8, 9]],
5    [[2, 3, 4, 0], [5, 6, 0, 0], [7, 8, 9, 10]],
6], dtype="int32")
7
8y = np.array([0, 1], dtype="int32")
9
10print(x.shape)  # (batch, sentences, words)

In a real project, you would:

• split documents into sentences
• tokenize each sentence
• map tokens to integer ids
• pad words within each sentence
• pad or truncate the number of sentences per example

A Hierarchical Keras Model

One common design is a sentence encoder wrapped in TimeDistributed, followed by a second recurrent layer over the sentence sequence.

1import tensorflow as tf
2
3vocab_size = 5000
4embedding_dim = 32
5num_sentences = 3
6num_words = 4
7
8sentence_encoder = tf.keras.Sequential([
9    tf.keras.layers.Input(shape=(num_words,)),
10    tf.keras.layers.Embedding(vocab_size, embedding_dim, mask_zero=True),
11    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(16)),
12])
13
14model = tf.keras.Sequential([
15    tf.keras.layers.Input(shape=(num_sentences, num_words)),
16    tf.keras.layers.TimeDistributed(sentence_encoder),
17    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(16)),
18    tf.keras.layers.Dense(2, activation="softmax"),
19])
20
21model.compile(
22    optimizer="adam",
23    loss="sparse_categorical_crossentropy",
24    metrics=["accuracy"],
25)
26
27model.fit(x, y, epochs=3, verbose=0)

This architecture works in two stages:

• each sentence becomes a fixed-size vector
• the sequence of sentence vectors is modeled at the document level

That is why it is often called a hierarchical sequence model.

When to Use This Pattern

This approach is useful for tasks such as:

• document classification
• email or ticket routing
• conversation-turn modeling
• predicting labels from multi-sentence contexts

It is most helpful when sentence boundaries carry meaning. If your task only depends on a short flat sequence, a standard token-level model may be simpler.

Alternatives

You do not have to use stacked LSTMs. Other valid options include:

• a word-level encoder plus sentence-level transformer
• 'Conv1D for sentence encoding and an RNN over sentences'
• a transformer over the full token stream if the sequence length is manageable

The core idea stays the same: preserve the sentence grouping if the grouping matters.

Common Pitfalls

The most common mistake is feeding a 2D token matrix into a model that expects three dimensions. For hierarchical training, the model input must reflect both sentence count and token count.

Another issue is inconsistent padding. If some examples have varying numbers of sentences or words without consistent padding and truncation, batching becomes messy and mask handling can break.

A third pitfall is flattening the entire document too early and then wondering why the sentence-level model provides no benefit. If you remove sentence boundaries in preprocessing, the hierarchy is already gone.

Finally, make sure the labels match the output layer. A document-level classifier needs one label per document, not one label per sentence unless you explicitly design a sentence-labeling model.

Summary

• Sequences of sentences are naturally hierarchical, not just long flat token lists.
• Represent inputs with dimensions for both sentence count and word count.
• A common Keras solution is sentence encoding plus a second model over sentence vectors.
• Keep sentence boundaries during preprocessing if they matter to the task.
• Check input shape, padding strategy, and label granularity carefully.