이미지: Google DeepMind Blog · METAL LAB 편집
Summary
- Google DeepMind, working with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, ran a pilot of the world's first double-blind AI evaluation on Gemini Flash Lite.
- Using Google Cloud's confidential-computing technology "Confidential Space," the setup encrypted things so evaluators couldn't see the model weights and Google couldn't see the test questions.
- Google DeepMind explained that benchmark evaluations had always forced a choice: either evaluators disclose their questions and risk the model having trained on them beforehand, or model providers hand over their weights and risk exposing their intellectual property.
If the model already saw the exam
A perfect score means nothing if the student saw the exam questions beforehand. AI model evaluation runs into the same problem. If a model has already encountered benchmark questions during training, its score doesn't reflect real ability. This is called "benchmark contamination," and as AI models grow larger and more complex, it's gotten harder to catch.
Google DeepMind announced on its official blog on August 27th that it had piloted the world's first double-blind AI evaluation. Researchers William Isaac, Sol Messing, and Christian Rumm of Google DeepMind presented the pilot. Rather than a frontier-scale model, the test subject was a lightweight one — Gemini Flash Lite.
Why it's always been an either/or choice
Until now, outside organizations testing AI models had to choose between two options. If the evaluator handed its test questions over to the model's provider, there was a risk the model would train on those questions beforehand and inflate its score. Conversely, if the provider handed the model weights themselves over to the evaluator, it had to accept the risk of exposing its intellectual property. Google DeepMind said that while no-logging measures and contractual safeguards have been used to prevent leaks, this is the first time encryption technology has been folded into the process as well.
| Method | Test questions exposed | Model weights exposed | Remaining risk |
|---|---|---|---|
| Question-disclosure (existing) | Exposed | Not exposed | Model trains on questions, inflating its score |
| Weight-sharing (existing) | Not exposed | Exposed | IP leakage |
| Double-blind (this pilot) | Not exposed | Not exposed | Only encrypted results are checked mutually |
A test run inside an encrypted box
For this pilot, Google used "Confidential Space," part of Google Cloud's confidential computing portfolio. Inside this environment, the evaluator can't see the weights of Gemini Flash Lite, and Google can't see the test questions the evaluator wrote. Each side's material exists only inside an encrypted box, and only the resulting output ever leaves that box. Google Cloud describes this approach as a privacy-preserving environment.
The pilot brought together the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. Google DeepMind said it has kept working with outside partners — civil-society groups and national AI safety and security institutes — to surface blind spots that internal evaluation alone might miss. The detailed methodology and results are laid out separately in a technical report.
Why this matters more for cybersecurity and government evaluations
Google DeepMind said encryption-based evaluation like this will matter especially in high-sensitivity areas such as cybersecurity testing and government assessments. It opens a path for independent bodies to rigorously verify frontier models without compromising data sovereignty or security. Policymakers, researchers, and companies all need to be able to trust AI benchmark scores at face value — but until now, the technical machinery to support that trust simply hasn't existed, according to Google DeepMind's assessment.
Editor's Take
This pilot matters because benchmark-contamination concerns are no longer hypothetical. On August 10th, a paper claiming to have decoded hidden reasoning tokens from Claude- and GPT-series models — and to have found responses that looked as if the models had memorized AIME questions in advance — spread through the community. Suspicion that benchmark scores can't simply be taken at face value had already been building, and Google DeepMind is now stepping in with an encryption method meant to cut that suspicion off at the source.
Still, it's worth noting that the model tested here was Gemini Flash Lite, a lightweight model. Scaling this up to frontier-scale models would inevitably mean far higher computational cost and much more complex verification. For this approach to actually become an industry standard, other developers like OpenAI and Anthropic would need to agree to put their own model weights inside this kind of encrypted environment — and that's not purely a technical question. It's also a question of trust and competition.
There's a clear takeaway here for companies and research institutions adopting AI models: don't judge a model by a single benchmark score — ask what conditions produced it. If a score came without any contamination check, it's worth assuming real-world performance might fall short of what that number suggests.
What's worth watching over the next few months is whether other AI labs try building similar encrypted-evaluation systems of their own. Once Google DeepMind extends this approach to frontier-scale models, that's when we'll actually start getting a real answer to the trust problem surrounding benchmark scores.
Comments