Sliding-window attention
Letting each token attend only to a few thousand neighbours instead of the whole sequence, so cost grows with length rather than with its square.
what it is made of · the vocabulary
local attentionSWA
The standard trade for long context: cheap and fast, at the price of the model genuinely forgetting the far past unless a few layers are left global.