Distillation
Training a small model to imitate a large one, so it answers nearly as well at a fraction of the cost.
what it is made of · the vocabulary
distilled modelstudent model
Most of the cheap models worth using are distilled from something much larger. It is also why a small model can beat a bigger one on a leaderboard: it was taught by a better teacher.