Multi-head self-attention is a critical component of the Transformer architecture, introduced in the seminal paper "Attention is All You Need" by Vaswani et al. (2017). The implementation in TensorFlow known as tf.keras.layers.MultiHeadAttention is a popular choice for developers and researchers working with such models. However, one key argument, key_dim , can sometimes lead to confusion as it does not always seem to match the examples from the original paper. In this article, we'll explore the key_dim parameter in detail, why this discrepancy may occur, and how to properly utilize MultiHeadAttention in practice.

Understanding key_dim

The key_dim parameter in tf.keras.layers.MultiHeadAttention represents the dimensionality of the query and key vectors. This is crucial as it directly impacts the size of the linear transformations applied to the input data, and consequently, the calculation of attention scores.

Theoretical Background

In the Transformer model as described by Vaswani et al., the attention mechanism is implemented through the following steps:

1. Linear Projections: Inputs are linearly transformed into three separate matrices: queries ($Q$), keys ($K$), and values ($V$). Each of these matrices is projected using learned weight matrices.
2. Scaled Dot-Product Attention: The attention scores are calculated via the dot product of queries and keys, scaled by the square root of the dimensionality of the keys, which is denoted as d_k .

$$\text{Attention}(Q, K, V) = \text{softmax} \left( \frac{QK^T}{\sqrt{d\_k}} \right) V$$

1. Multi-Head Extension: Instead of computing a single attention output, multiple sets of projections (called heads) are used, each with its attention calculation. This allows the model to jointly attend to information from different representation subspaces at different positions.

Discrepancy and Clarification

In the paper, the dimensionality of the queries and keys ($d_k$) for each head is typically set such that the total dimensionality after concatenating the heads is equal to the input dimensionality. For example, if the input dimension is 512 and there are 8 heads, then each head will have a key dimension of 64 ($512 \div 8$).

However, in the use of the MultiHeadAttention layer, key_dim is specified directly and does not necessarily have to equal the dimension derived from the paper's implicit division. This allows for greater flexibility in certain applications but can be a source of confusion if not aligned with the model's expected architecture.

Example Discrepancy

Suppose you're working with the following setup from the paper:

• Input dimension: 512 • Number of heads: 8 • Expected key dimension per head: 64

However, your implementation specifies key_dim=60 . This may be done for various reasons, such as tailoring the model for specific data characteristics or computational constraints. The key here is to ensure that your choice of key_dim still respects the architecture's requirements for proper functioning, including:

• Ensuring that the total dimension of outputs aligns with subsequent model layers. • Monitoring the computational efficiency, as smaller dimensions may reduce the computational burden.

Practical Usage

Common Setup

• Ensure alignment with other parts of the architecture. • Use model summaries and visualizations to confirm dimensions. • Validate model performance empirically and compare it with setups using canonical key_dim values.