Introduction

In TensorFlow, a sliding window over image sequences usually has two dimensions of movement: time and space. The common building blocks are tf.signal.frame for sequence windows and tf.image.extract_patches for 2D spatial patches. Combining them gives you a clean way to generate training or inference windows from image sequences.

Start with a Clear Tensor Shape

For a sequence of images, a practical shape is:

• 'T for time or frame count'
• 'H for height'
• 'W for width'
• 'C for channels'

So the tensor shape is:

[T, H, W, C]

Example:

1import tensorflow as tf
2
3sequence = tf.reshape(
4    tf.range(5 * 4 * 4, dtype=tf.float32),
5    (5, 4, 4, 1)
6)
7
8print(sequence.shape)

This creates 5 grayscale frames of size 4 x 4.

Slide Across the Sequence Dimension First

If you want temporal windows such as 3 consecutive frames at a time, use tf.signal.frame along the time axis.

1import tensorflow as tf
2
3sequence = tf.reshape(
4    tf.range(5 * 4 * 4, dtype=tf.float32),
5    (5, 4, 4, 1)
6)
7
8time_windows = tf.signal.frame(
9    sequence,
10    frame_length=3,
11    frame_step=1,
12    axis=0
13)
14
15print(time_windows.shape)

The result shape is:

(3, 3, 4, 4, 1)

That means:

• '3 temporal windows'
• each window contains 3 frames
• each frame is 4 x 4 x 1

This is the temporal equivalent of a sliding window over a 1D sequence.

Extract 2D Patches from Each Frame

Once you have sequence windows, you can extract spatial patches with tf.image.extract_patches. Because that API expects a batch of images, flatten the first two dimensions temporarily.

1import tensorflow as tf
2
3time_windows = tf.signal.frame(sequence, frame_length=3, frame_step=1, axis=0)
4
5num_windows = tf.shape(time_windows)[0]
6window_size = tf.shape(time_windows)[1]
7
8batched_frames = tf.reshape(time_windows, (-1, 4, 4, 1))
9
10patches = tf.image.extract_patches(
11    images=batched_frames,
12    sizes=[1, 2, 2, 1],
13    strides=[1, 1, 1, 1],
14    rates=[1, 1, 1, 1],
15    padding="VALID"
16)
17
18print(patches.shape)

Each output location now holds a flattened 2 x 2 patch from one frame.

Restore the Sequence Structure

After extracting patches, reshape the batch back into sequence-window form.

1patches = tf.reshape(
2    patches,
3    (num_windows, window_size, 3, 3, 4)
4)
5
6print(patches.shape)

In this example:

• '3 temporal windows'
• '3 frames per temporal window'
• '3 x 3 spatial patch positions'
• '4 values per flattened 2 x 2 patch'

This gives you a combined time-and-space sliding-window tensor that can be fed into further preprocessing or a model.

Use VALID Versus SAME Deliberately

padding="VALID" means only fully contained patches are extracted. padding="SAME" adds implicit padding so output positions align more closely with the original image size.

For sliding-window training data, VALID is often easier to reason about because every patch contains only real image pixels. SAME can be useful when model input geometry or downstream alignment matters more than strict boundary purity.

Dataset Pipelines for Larger Inputs

For larger sequences, generating every patch eagerly can consume a lot of memory. A tf.data.Dataset pipeline is often better.

1import tensorflow as tf
2
3frames = tf.data.Dataset.from_tensor_slices(sequence)
4windows = frames.window(size=3, shift=1, drop_remainder=True)
5windows = windows.flat_map(lambda w: w.batch(3))
6
7for window in windows.take(2):
8    print(window.shape)

This gives you temporal windows lazily, which is useful when the sequence comes from disk, video decoding, or a large training corpus.

Common Pitfalls

The most common mistake is mixing up the sequence axis and the image batch axis, which produces windows of the wrong shape. Another is forgetting that tf.image.extract_patches flattens each spatial patch into the last dimension, so the output shape can look confusing until you reshape it deliberately. Developers also generate all windows eagerly for large datasets and then hit memory pressure that a tf.data pipeline could have avoided.

Summary

• Represent image sequences clearly, usually as [T, H, W, C].
• Use tf.signal.frame for temporal sliding windows.
• Use tf.image.extract_patches for spatial sliding windows.
• Reshape carefully because extracted patches are flattened in the last dimension.
• Prefer tf.data pipelines when the full window tensor would be too large to materialize eagerly.