Attention
The mechanism by which a model decides, for each word it is producing, which earlier words to weigh.
what it is made of · the vocabulary
self-attentioncross-attention
It is the reason a model can carry a pronoun back to the right noun forty sentences earlier. Every token attends to every other, so doubling the input roughly quadruples the work; the many named variants below all exist to cheat that.