Benchmark Contamination

Benchmark contamination refers to a problem where an AI model has already encountered the questions used to measure its performance during training, causing its score to come out higher than its real ability. It's similar to a student who happened to see the answer sheet the night before an exam and got a perfect score—that doesn't mean they actually know the subject well.

The issue is that this "answer sheet" isn't secretly handed over on purpose. AI models learn from vast amounts of text gathered from the internet, and it often happens that some exam questions and their correct answers end up mixed into that training material by chance. As models get bigger and the training data grows, it becomes increasingly difficult to check one by one which questions were already included.

Because of this, comparing different models' abilities based on scores alone can be misleading. A high score from a contaminated test may not reflect genuine skill—it may simply be the result of having "already seen the questions."

The article explains, "If a model has already encountered benchmark questions during training, its score doesn't reflect its actual ability. This is called 'benchmark contamination.'" To prevent this, Google DeepMind has piloted a double-blind approach, encrypting both the test questions and the model weights so that neither the evaluating organization nor the model developer can see them. A common misunderstanding: benchmark contamination doesn't mean the model intentionally cheated—in most cases, it's simply the result of exam questions accidentally being mixed into the training data.