Eval set
Your own collection of cases with known good answers, used to decide whether a change helped.
how it is measured · the vocabulary
golden setregression suite
Fifty examples from your own traffic will tell you more about which model to buy than every public leaderboard combined. It is also the only thing that catches a regression when a seller silently updates a model behind the same name.