Sinusoidal Embedding as Introduced in "Attention is All You Need"

The concept of sinusoidal embedding was introduced in the seminal paper titled "Attention is All You Need" by Vaswani et al. in 2017. The paper outlined the Transformer model, which revolutionized the field of natural language processing. This article provides a deep dive into the concept of sinusoidal embedding, its purpose, and how it operates within the context of the Transformer model.

Understanding Sinusoidal Embedding

1. The Need for Positional Encoding

Transformers, unlike recurrent neural networks (RNNs), do not inherently consider the sequential order of input data. They process data simultaneously across multiple parallel layers. Therefore, to retain the information about the order of words in sequences, positional encodings are added to the input embeddings. Sinusoidal embedding is one such method of positional encoding.

2. Mathematical Representation

Sinusoidal embedding uses two functions based on sine and cosine waves. For a straightforward representation, given a position $pos$ and dimension $i$, the encoding is computed as follows:

• Even indices:
\\text{PE}(pos, 2i) = \sin(pos / 10000^{2i/d})\

• Odd indices:
\\text{PE}(pos, 2i+1) = \cos(pos / 10000^{2i/d})\

In these equations: • $pos$ is the position in the sequence. • $i$ is the dimension. • $d$ is the embedding dimension, a hyperparameter of the model.

3. Properties of Sinusoidal Embeddings

Sinusoidal embeddings have a number of important properties which contribute to the effective modeling of sequential data:

• Scale-Invariance: By adjusting the frequency using $10000^{2i/d}$ and varying it across dimensions, the embedding can capture both local and global position relationships within the sequence. • Periodicity: Since sine and cosine are periodic functions, they inherently encapsulate the concept of repeating patterns which could correlate with cycles in textual or sequential data. • Derivation of Relative Positions: The dot products can capture relative position differences, an advantageous property in tasks such as translation where the relative position provides context.

Technical Advantages

Why sinusoidal over learned embeddings?

• Generalization: Sinusoidal functions do not count on the model to learn positional information from scratch and thereby offer a continuous and smooth representation of positions. • Reduction in Parameters: Since sinusoidal embeddings are derived from fixed functions, they do not contribute additional parameters to the model, unlike learned embeddings which need updates and are more prone to overfitting.

4. Implementation Example

Here is a Python snippet showcasing a simple sinusoidal position encoding implementation: