Introduction

Transformers process tokens in parallel, which means the model needs an explicit way to represent token order. Sinusoidal embeddings solve that problem by adding a deterministic position signal to each token embedding before attention is computed.

Why Positional Information Is Needed

Self-attention can compare every token with every other token, but the attention mechanism itself does not know whether one token came before or after another. If you feed the same words in a different order, the raw attention layers would otherwise see the same set of token vectors.

The Transformer paper solved this by adding a positional encoding to the token embedding at each sequence index. A token at position 5 therefore gets a different combined vector than the same token at position 50.

What Makes the Encoding Sinusoidal

Instead of learning a separate vector for each position, the original paper uses sine and cosine waves with different frequencies. Even dimensions use sine and odd dimensions use cosine.

One common form is:

PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

This design gives each position a unique pattern across dimensions. Low-frequency components change slowly, while high-frequency components change quickly. The model can combine those signals to reason about both absolute and relative distance.

An important advantage is extrapolation. Because the encoding is generated from a formula rather than a lookup table, you can compute positions that were not explicitly seen as parameters during training.

NumPy Example

The following code builds sinusoidal embeddings for a sequence length and model width. It is small enough to inspect directly and mirrors the common implementation used in deep-learning libraries.

1import numpy as np
2
3
4def sinusoidal_embedding(seq_len, d_model):
5    positions = np.arange(seq_len)[:, np.newaxis]
6    even_dims = np.arange(0, d_model, 2)
7    angle_rates = np.exp(-np.log(10000.0) * even_dims / d_model)
8    angles = positions * angle_rates
9
10    pe = np.zeros((seq_len, d_model))
11    pe[:, 0::2] = np.sin(angles)
12    pe[:, 1::2] = np.cos(angles)
13    return pe
14
15
16embedding = sinusoidal_embedding(seq_len=4, d_model=8)
17print(np.round(embedding, 4))

In a Transformer, this matrix is added elementwise to the token embedding matrix. The token vectors still carry semantic meaning, while the sinusoidal term injects order.

Why Different Frequencies Help

If every dimension used the same wave, nearby positions would look too similar. By spreading frequencies across the embedding width, each position gets a richer signature. The model can then learn linear operations that capture distance relationships.

For example, positions 10 and 11 produce related but not identical values across the embedding dimensions. That lets attention layers infer notions such as "next token" or "roughly ten steps away" without hard-coded grammar rules.

This is also why sinusoidal embeddings remain a useful baseline even though many modern systems use learned positional embeddings, rotary encodings, or other variants. The method is simple, parameter-free, and mathematically stable.

Practical Notes

In production code, positional encodings are usually created once up to a maximum sequence length and stored as a tensor. During a forward pass, the model slices the needed prefix and adds it to the token embeddings.

When the model dimension is odd, implementations must handle the final unmatched dimension carefully. Many libraries avoid that situation by choosing an even d_model, such as 512 or 768.

Common Pitfalls

• Treating sinusoidal embeddings as a replacement for token embeddings is incorrect. They are added to token embeddings, not used alone for language meaning.
• Generating the encoding with the wrong shape causes silent broadcasting bugs. Check that the position matrix and embedding matrix align on both sequence length and width.
• Forgetting that even and odd dimensions use different functions breaks the intended pattern. Use sine on even indices and cosine on odd indices.
• Assuming sinusoidal embeddings are always superior is too simplistic. Learned or rotary methods can work better depending on the model and task.
• Recomputing the full matrix on every batch wastes time. Cache the positional encoding up to the maximum needed sequence length.

Summary

• Sinusoidal embeddings give Transformers explicit token-order information.
• They use sine and cosine waves at multiple frequencies to encode position.
• The encoding is deterministic, so it does not add trainable parameters.
• A practical implementation builds a position-by-dimension matrix and adds it to token embeddings.
• The method remains a strong baseline because it is simple, interpretable, and efficient.