Mixture of experts
A model split into many specialised parts, of which only a few run for any given token.
what it is made of · the vocabulary
MoEsparse modelactive parameters
A mixture of experts with 400 billion parameters might compute with 30 billion at a time. That makes it cheap to run and expensive to hold: you need memory for all of it and compute for a fraction. It is why model names carry two numbers — 397B A17B means 397 billion in total, 17 billion active.