Positional Encoding and RoPE: How LLMs Know Word Order

Understand why attention cannot see word order, how older methods added position, and how Rotary Position Embedding (RoPE) lets modern LLMs focus on the distance between tokens, plus how context windows get stretched.

28 minIntermediateCode Examples

The Core Idea: Attention treats text like a bag of words, so a model needs a way to know where each token sits. Modern LLMs use RoPE: they rotate each query and key by an amount that depends on position, so the attention score only depends on the distance between two tokens.

  1. Beginner: Attention Cannot Tell Word Order

Self-attention compares every token with every other token, but it does it like a bag of words. If you shuffle the words, the raw attention math gives the same pairs of scores. To a plain attention layer, dog bites man and man bites dog look almost identical, yet they mean very different things.

So we must inject position information into the model. This is called positional encoding. The question is how, and the answer changed a lot between the original Transformer (2017) and today's LLMs.

Simple analogy: Imagine a class photo where nobody is standing in order. You know who is in the photo, but not who stands next to whom. Positional encoding is like putting a numbered sticker on every person.

  1. Beginner: Three Ways to Add Position

Researchers have tried several families of ideas. Here is the whole story in one table:

MethodHow it worksUsed byMain weakness
Sinusoidal (absolute)Adds a fixed wave pattern for each position to the token embeddingOriginal Transformer (2017)Struggles on text longer than seen in training
Learned (absolute)Learns one vector per position, like a lookup tableGPT-2, BERTHard limit: no vector exists for unseen positions
Relative (RoPE, ALiBi)Encodes the distance between two tokens inside the attention scoreMost modern LLMs (RoPE is the most popular)Needs tricks to stretch to much longer text

The key insight: What usually matters in language is how far apart two words are, not their exact seat numbers. Relative methods build that directly into attention, and that is why they won.

  1. Medium: RoPE (Rotary Position Embedding) in Plain English

RoPE gives each token's query and key vector a rotation that depends on its position. Token at position 0 is not rotated, position 1 is rotated a little, position 2 a bit more, and so on, like turning a dial one notch per word.

The magic happens in the dot product. When query and key are both rotated, the angle between them depends only on the difference of their positions. The score between word 5 and word 2 is therefore the same as between word 105 and word 102, because both pairs are exactly 3 apart.

1
Split the vector into pairs: A 128-number vector becomes 64 little 2D arrows.
2
Pick a speed for each pair: The first pairs spin fast, the last pairs spin slowly. This lets the model feel both nearby and faraway distances.
3
Rotate by position × speed: The arrow of a token at position mm is turned by the angle m⋅θim \cdot \theta_i.
4
Take the dot product as usual: The result now depends on the relative distance m−nm - n.

θi=10000−2i/dangle for pair i=m⋅θi\theta_i = 10000^{-2i/d} \qquad \text{angle for pair } i = m \cdot \theta_i

Why it is elegant: RoPE adds no extra parameters and does not touch the values. It only rotates queries and keys, and rotation never changes a vector's length, so the signal stays stable.

  1. Medium: A Tiny Worked Example With One Rotating Arrow

To see the idea with numbers, use a single pair of dimensions, a 2D arrow starting at (1,0)(1, 0), and a rotation of 30° per position. Each position turns the arrow a further 30°:

PositionTotal rotationArrow after rotation
00°(1.000, 0.000)
130°(0.866, 0.500)
260°(0.500, 0.866)
390°(0.000, 1.000)

Now take the dot product of an arrow at position mm with an identical arrow at position nn. The result is cos⁡((m−n)⋅30°)\cos((m - n) \cdot 30°). Only the gap matters:

Pair of positionsGapDot product
(3, 3)01.000
(3, 2)10.866
(3, 1)20.500
(3, 0)30.000
(13, 10)30.000

See it? The pairs (3, 0) and (13, 10) are far apart in absolute position, yet they get the same score because both gaps are 3. That is the relative-position property, and nobody had to teach it to the model.

  1. Advanced: Stretching the Context Window

A model trained with positions up to 4,096 has never seen a rotation angle for position 20,000. If you simply feed it longer text, the angles go outside the range it learned, and quality drops sharply. Several techniques stretch the usable context:

TechniqueIdeaTrade-off
Position interpolationSqueeze new positions into the old range, so position 8,000 is treated as 4,000Needs a short fine-tune; nearby words become slightly harder to tell apart
NTK-aware scalingChange the rotation speeds (the base number) instead of all positions equally, keeping fast pairs sharpWorks with little or no fine-tuning
YaRNMixes the two ideas and adjusts attention strengthStrong results with a small amount of extra training
ALiBiSkips rotations and subtracts a penalty that grows with distanceSimple, but a different family from RoPE
A practical note
When a model advertises a context window much longer than its original training length, a method like these is usually involved. Longer context also means a bigger KV cache, as you saw in the previous lesson, so cost grows too.

  1. Visual Explanation: The Rotating Arrows and the Gap Rule

Each token gets turned a little more than the one before it. Pick a position to see where its arrow points (using 30° per position) and how similar it is to the arrow at position 0.

🧭 ROTATION EXPLORER · CLICK A POSITION30° per position, one pair

  1. Python Implementation: RoPE From Scratch and the Relative-Position Test

This function rotates every pair of dimensions by the angle position × speed. Then we test the key promise of RoPE: the score depends only on the distance between tokens, not on where they sit in the text.

rope_from_scratch.pyPython 3.10+ · NumPy only
import numpy as np def rope(x, position, base=10000.0):    # Rotate each pair of dimensions (x[2i], x[2i+1]) by an angle that grows with position    d = len(x)    i = np.arange(d // 2)    theta = base ** (-2 * i / d)          # one frequency per pair: fast to slow    angle = position * theta    cos, sin = np.cos(angle), np.sin(angle)    out = np.empty_like(x)    out[0::2] = x[0::2] * cos - x[1::2] * sin    out[1::2] = x[0::2] * sin + x[1::2] * cos    return out rng = np.random.default_rng(0)q = rng.normal(size=8)k = rng.normal(size=8) # 1. Same distance (3 apart) at very different absolute positionsa = np.dot(rope(q, 5), rope(k, 2))b = np.dot(rope(q, 105), rope(k, 102))print("score at positions (5, 2):     ", round(float(a), 4))print("score at positions (105, 102): ", round(float(b), 4))print("Same score for the same distance:", np.isclose(a, b)) # 2. A different distance gives a different scorec = np.dot(rope(q, 5), rope(k, 4))print("score at distance 1:", round(float(c), 4), "| at distance 3:", round(float(a), 4)) # 3. Rotation never changes the length of a vectorprint("Length before and after rotation:", round(float(np.linalg.norm(q)), 4), round(float(np.linalg.norm(rope(q, 50))), 4)) # 4. Position interpolation: squeeze 8192 positions into the 4096 range seen in trainingscale = 4096 / 8192print("Position 8000 is treated as position", 8000 * scale)
EXPECTED OUTPUT (your numbers may differ slightly)
score at positions (5, 2): -1.5865
score at positions (105, 102): -1.5865
Same score for the same distance: True
score at distance 1: -1.6954 | at distance 3: -1.5865
Length before and after rotation: 1.8627 1.8627
Position 8000 is treated as position 4000.0

Look at the first three lines of output: positions (5, 2) and (105, 102) are 100 steps apart in absolute terms but both have a gap of 3, and the scores are identical. Distance 1 gives a different score, so the model can tell near from far.

Interactive Knowledge Check

Question 1: What is the main advantage of RoPE?
Question 2: A model was trained on 4,096 positions. What does position interpolation do to reach 8,192?

Key Points

✓Plain attention is blind to word order, so position information must be added.
✓Older methods add a fixed or learned vector per position (absolute). Modern methods encode the distance (relative).
✓RoPE rotates pairs of dimensions of Q and K by an angle equal to position × speed.
✓Because of the rotation, the dot product depends only on the gap between positions, and vector length is never changed.
✓RoPE adds no parameters, which is one reason it became the standard in open LLMs.
✓Context can be stretched past the training length with position interpolation, NTK-aware scaling or YaRN, usually with a little extra training.

The Big Picture

Positional Encoding in One Line

Token meaning + rotation by position → attention sees distance, not seat numbers

Order is what turns a pile of words into a sentence. RoPE gives the model that sense of order in a way that is cheap, stable and works at many text lengths.

Up Next: You now know the two most important parts inside an LLM block: attention and positions. In Modern LLM Architectures you will see how they are combined with RMSNorm, SwiGLU, GQA and Mixture of Experts into today's models.