ExploitBench

ExploitBench is OpenAI's test for checking how good an AI model is at hacking. Like a test paper that gives a student several problems to solve and then scores them, it presents the model with a series of tasks that involve finding hidden weaknesses in simulated computer systems and actually breaking in, then tallies up how many it succeeds at.

Getting a perfect score on this test means the model can find unknown weaknesses in computer systems and carry out actual attacks without human help. OpenAI has said its new model, Astra, scored perfectly on this test — but that same capability could also be misused for dangerous purposes, which is why the score is used as a basis for assigning a safety rating.

However, this test was created and scored by OpenAI itself. It's important to note in the article that this is not a result independently verified by an outside organization.

The article states, "OpenAI said Astra achieved a perfect score on ExploitBench, which evaluates hacking capability." A common misunderstanding here is assuming this perfect score came from an independent verification body — but it should be read with the distinction that this was actually an internal evaluation designed and conducted by OpenAI itself.