Introduction

The original Transformer ("Attention is All You Need") adds positional encoding only at the input layer — not at every layer. This works because self-attention preserves positional information through residual connections. Each layer's output includes a residual connection from its input, which carries the positional signal forward. Adding positional encoding at every layer is unnecessary (the information persists) and potentially harmful (it can overweight position relative to content). Some later architectures like DeBERTa and ALiBi do inject position at every layer, but with relative position representations rather than repeating absolute sinusoidal encodings.

How Positional Encoding Works in the Original Transformer

1import torch
2import math
3
4def sinusoidal_encoding(seq_len, d_model):
5    pe = torch.zeros(seq_len, d_model)
6    position = torch.arange(0, seq_len).unsqueeze(1).float()
7    div_term = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model))
8
9    pe[:, 0::2] = torch.sin(position * div_term)  # Even indices
10    pe[:, 1::2] = torch.cos(position * div_term)  # Odd indices
11    return pe
12
13# Added once at the input
14# x = token_embeddings + positional_encoding
15pe = sinusoidal_encoding(512, 768)
16print(pe.shape)  # (512, 768)

The positional encoding is added to token embeddings before the first encoder/decoder layer. After this point, positional information propagates through the network via residual connections.

Why Residual Connections Preserve Position

1Layer N output = LayerNorm(Attention(x) + x)
2                                         ^
3                                         |
4                             Residual connection carries
5                             positional signal from input

Each Transformer layer has the form:

1# Simplified Transformer layer
2def transformer_layer(x):
3    # Self-attention + residual
4    attn_out = self_attention(x)
5    x = layer_norm(x + attn_out)  # x still contains positional info
6
7    # FFN + residual
8    ffn_out = feed_forward(x)
9    x = layer_norm(x + ffn_out)   # Positional info still present
10    return x

Because each layer adds its transformation to the input (rather than replacing it), the original positional encoding is never lost. It gets blended with learned representations but remains part of the signal.

Why Adding Position at Every Layer Would Be Redundant

1# Hypothetical: adding PE at every layer
2def transformer_with_pe_every_layer(x, pe):
3    x = x + pe  # Layer 0
4    for layer in self.layers:
5        x = layer(x)
6        x = x + pe  # Re-adding PE — overwrites content evolution
7    return x

Problems with this approach:

1. Double-counting: Position is already in the signal from residual connections. Adding it again amplifies positional bias relative to semantic content.
2. Interfering with learned representations: By layer 6+, the network has learned high-level abstractions. Re-injecting raw positional encoding disrupts these representations.
3. No empirical benefit: Experiments show no improvement from repeating absolute sinusoidal encodings at every layer.

Architectures That Do Inject Position at Every Layer

Some newer architectures inject positional information per layer, but with relative (not absolute) position:

1# ALiBi: Attention with Linear Biases (Press et al., 2021)
2# Adds a linear position bias to attention scores at every layer
3def alibi_attention(Q, K, V, slopes):
4    scores = Q @ K.T / math.sqrt(d_k)
5    # Add position-dependent bias at every layer
6    positions = torch.arange(seq_len)
7    bias = slopes * (positions.unsqueeze(0) - positions.unsqueeze(1))
8    scores = scores + bias  # Relative position injected per layer
9    return softmax(scores) @ V
10
11# DeBERTa: Disentangled attention
12# Separates content and position into two vectors
13# Position-to-content and content-to-position attention computed per layer
14
15# RoPE (Rotary Position Embedding) — used in LLaMA, GPT-NeoX
16# Applies rotation matrix to Q and K at every layer
17def apply_rope(x, positions, dim):
18    freqs = 1.0 / (10000 ** (torch.arange(0, dim, 2).float() / dim))
19    angles = positions.unsqueeze(-1) * freqs.unsqueeze(0)
20    cos_angles = torch.cos(angles)
21    sin_angles = torch.sin(angles)
22    x_even = x[..., 0::2] * cos_angles - x[..., 1::2] * sin_angles
23    x_odd = x[..., 0::2] * sin_angles + x[..., 1::2] * cos_angles
24    return torch.stack([x_even, x_odd], dim=-1).flatten(-2)

These approaches work because they inject relative position information (how far apart tokens are) directly into the attention computation, rather than adding absolute position to the token representations.

Absolute vs Relative Position Encoding

Experimental Evidence

Research has shown:

• Repeating absolute sinusoidal encodings at every layer provides no measurable improvement on standard benchmarks (WMT translation, GLUE).
• Relative position methods (RoPE, ALiBi) applied per layer improve length generalization — the model handles sequences longer than those seen during training.
• Learned absolute positional embeddings at the input only (BERT-style) perform comparably to sinusoidal encodings.

Common Pitfalls

• Assuming positional information is lost in deep layers: Residual connections preserve the positional signal throughout the network. Probing experiments confirm that position can be decoded from any layer's representations, even the final layer. The information does not need to be re-injected.
• Confusing absolute and relative position injection: Adding absolute sinusoidal encodings at every layer is redundant and harmful. Injecting relative position (RoPE, ALiBi) at every layer is different — it modifies the attention pattern, not the token representations, and provides fresh positional context for each layer's attention computation.
• Thinking sinusoidal encoding is the only option: Modern LLMs (LLaMA, GPT-NeoX, BLOOM) use RoPE or ALiBi, which are injected per-layer and handle long sequences better. Sinusoidal encoding was a starting point, not the final answer.
• Ignoring the interaction between position and content: In the original Transformer, positional encoding is added to content embeddings, entangling position and meaning in the same vector. DeBERTa explicitly separates content and position into disentangled representations, which improves performance on tasks where position matters differently from content.
• Applying position encoding after layer normalization: Layer normalization rescales the representation, which can dilute the positional signal if encoding is added afterward. The original Transformer adds positional encoding before any layer processing, preserving the full signal magnitude.

Summary

• The original Transformer adds positional encoding only at the input, not at every layer
• Residual connections propagate positional information through all layers without re-injection
• Adding absolute positional encoding at every layer overweights position and degrades learned representations
• Modern architectures (RoPE, ALiBi, DeBERTa) inject relative position at every layer through attention mechanisms, which is fundamentally different from repeating absolute encodings
• Relative position methods improve length generalization and are now standard in large language models