Rotary Position Embedding

RoPE (rotary position embedding) encodes a token's position by rotating its query and key vectors: each 2-dimensional pair within a head is rotated by an angle proportional to the position, with a spectrum of frequencies across pairs. Because rotation is unitary, the dot product between a query at position i and a key at position j depends on the relative offset i-j — attention sees relative position without any learned position table.

Two consequences matter for finite-memory attention:

• Extrapolation: beyond the trained context, relative offsets take values the weights never saw. Generation degrades — measurably, as inflated perplexity at extended context — even though nothing breaks outright.
• Phase interference: keys stored with their rotations applied carry position in their phases. Summing rotated keys from different positions (as any merging scheme does — see KV Cache) causes partial cancellation. This is destructive for fidelity but doubles as a free suppression mechanism for stale, position-incoherent aggregates — a coupling between position handling and forgetting that Finite Memory for Unbounded Streams measures directly: re-indexing merged memories to fake ages backfires frozen, then wins once the model is fine-tuned with the memory in the loop.

The streaming trick of keeping offsets in-range by re-indexing cache positions ("rolling" RoPE) comes from StreamingLLM (Related Work).