Positional Embeddings

Typically, the input sequence $x$ B x T x emb_dim is transformed before self-attention by adding a positional embedding i.e. $\hat{x}=x+p$ where $p$ T x emb_dim can be learnable or can be fixed functions e..g sinuisodal.

$q_m^T{k}_n$ is the inner product of the mth query vector and the nth key vector. It is the $Q^T{K}_{mn}$ value of the self-attention matrix. This is the operation that typically enables knowledge conveyance between tokens at different positions.

In theory, one could hope that this relation is satisfied $q_m^T{k}_n=⟨f_q(x_m,m),f_k(x_n,n)⟩=g(x_m,x_n,m-n)$ i.e. inner product encodes position information only in the relative form.

RoPE

Incorporating the relative position embedding is straightforward: simply rotate the word embedding vector by amount of angle multiples of its position index.

rotation matrix = $R_{{\theta }_i,m}^d=\left(\begin{array}{cc}\cos (m{\theta }_i)& -\sin (m{\theta }_i)\\ \sin (m{\theta }_i)& \cos (m{\theta }_i)\end{array}\right)$ where ${\theta }_i=10000^{-2i/d}$

For an embedding of dimension $d$, RopE divides the embedding into $d/2$ blocks and rotates each block $b$ separately by the rotation matrix $R_{{\theta }_b,m}^d$