BERT (Bidirectional Encoder Representations from Transformers) is a model designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. This revolutionary approach paved the way for significant improvements in NLP tasks, and understanding how TokenEmbeddings are created in BERT is crucial to appreciating its efficacy. Token embeddings in BERT capture syntactic, positional, and segment-related information essential for deep learning models. Below, we delve into the process of creating these embeddings, breaking down the technical elements and exploring their integration in BERT.

Breakdown of TokenEmbeddings in BERT

Token embeddings in BERT combine three distinct types of embeddings:

1. Token Embeddings: Each word or token is mapped to a unique vector.
2. Segment Embeddings: These denote the sentence segments to differentiate different input sentences during the pre-training phase.
3. Position Embeddings: Positional information is encoded to track the order of tokens within the sentences.

Let's explore each type in detail.

1. Token Embeddings

Token embeddings in BERT are created using the WordPiece tokenization process. This process divides the text into tokens, including subword, word, or character units. These units are then matched against a vocabulary list, wherein each token is assigned a unique embedding vector. The goal of WordPiece tokenization is to reduce the size of the vocabulary to achieve better generalization and handle out-of-vocabulary words effectively.

WordPiece Tokenization Example: Suppose the vocabulary includes tokens like [reading, ##ing, ##read]. When the word "unreading" is tokenized, it might be split into [un, ##read, ##ing]. Each of these fragments then obtains an embedding vector from the pre-trained vocabulary matrix.

2. Segment Embeddings

BERT is trained on two sentence pairs where each sentence in a pair belongs to a distinct segment. During the training phase, segment embeddings are used to identify these different segments, allowing BERT to understand which sentence a token belongs to, thereby enhancing its context. Segment embeddings are created as follows:

• A sentence labeled "A" is assigned one set of embeddings. • A sentence labeled "B" is assigned another distinct set of embeddings.

If only a single sentence is used during inference, a single segment embedding is employed.

3. Position Embeddings

Unlike RNNs that inherently capture sequence order, Transformers and therefore BERT, do not. To compensate for this, position embeddings provide explicit positional information. Each position in the input sequence has a unique embedding to signify its position in the sequence. Importantly, position embeddings in BERT are learned rather than sinusoidal (as used in some Transformers).

Integration of TokenEmbeddings

The final token embedding for a token in BERT is the sum of its token embedding, segment embedding, and position embedding. These combined embeddings form a comprehensive representation utilized by BERT's self-attention mechanism to grasp contextual relationships.

Technical Consideration: If $E_{token}$, $E_{segment}$, and $E_{position}$ represent the token, segment, and position embeddings respectively, the final embedding $E_{final}$ for a token can be simplified as:

$$E_{final} = E_{token} + E_{segment} + E_{position}$$

Example Table

For clarity, here's a simplified table to summarize these key components:

Additional Considerations

• Variable Length & Masking: BERT uses padding and masking to handle input sequences of variable lengths, ensuring consistent matrix dimensions.

• Pre-training Tasks: • Masked Language Modeling (MLM): A percentage of inputs are masked, and the model predicts the masked words. • Next Sentence Prediction (NSP): Determines relationships between sentences to improve understanding of context.

These tasks require a robust embedding mechanism, which is competently provided through the combined embeddings in BERT.

In conclusion, the creation of TokenEmbeddings in BERT is a sophisticated amalgamation of subword tokenization, segment demarcation, and positional nuances. This layered embedding approach underpins the powerful contextual understanding that BERT provides, influencing modern NLP methodologies significantly.