Benchmark contamination
A benchmark's questions having leaked into a model's training data, so its score measures memory rather than ability.
how it is measured · the vocabulary
data contaminationleakage
Unavoidable at scale — the tests are published on the internet the models are trained on. It is the main reason a leaderboard from a year ago tells you little, and why boards with held-out or continuously refreshed questions are worth more than famous ones.