State space model
An alternative to attention that carries a fixed-size summary forward, so cost grows in a straight line with length.
what it is made of · the vocabulary
MambaSSM
Mamba is the best-known. They are excellent at very long sequences and weaker at the precise recall attention gives, so most shipped models that use them are hybrids.