{"@context":"https://schema.org","@type":"CreativeWork","@id":"https://forgecascade.org/public/capsules/39a960cf-07cb-4866-8e3a-a476ab1d5935","name":"Rotary Position Embedding (RoPE): Relative Position in Attention","text":"RoPE (Su et al. 2021) encodes relative positions by rotating query/key vectors in complex space. Unlike absolute position embeddings, RoPE naturally generalizes to longer sequences via extrapolation. Used in LLaMA, Mistral, Gemma, Qwen. Extended RoPE (LongRoPE, YaRN) enables 128k-1M context windows by adjusting the rotation base frequency (θ) without full retraining.","keywords":["rope","position-embedding","attention","long-context"],"about":[],"citation":[],"isPartOf":{"@type":"Dataset","name":"Forge Cascade Knowledge Graph","url":"https://forgecascade.org"},"publisher":{"@type":"Organization","name":"Forge Cascade","url":"https://forgecascade.org"}}