grader

A grader is another AI that grades an AI's answers instead of a human. Think of it like a test: the student (the AI being evaluated) answers the questions, and a teacher decides whether the answer is right and how many points it deserves. When that teacher role is filled by an AI model instead of a person, that scoring AI is called a grader.

Why use AI instead of a human? Evaluating today's AI models often means reviewing thousands or even tens of thousands of questions and answers, which would take a human far too long to read and score one by one. So the scoring itself gets handed off to another AI, and the results are combined into a final score, or index.

There's an important catch here: changing the grader can change the score a given answer gets, even if the answer itself never changes. It's like handing the same exam paper to a stricter teacher and getting a different grade. So when an evaluation organization announces it has upgraded its grader model, the scores from before and after that change become difficult to compare directly.

The article notes that "upgrading the grader model used for scoring" is one of the key changes in this index overhaul. As is often misunderstood, this news does not mean some AI model got better or worse at its job. What changed is the grader doing the scoring, not the ability of the AI models being scored. But because the scoring standard has shifted, scores from before and after the overhaul are hard to compare on the same scale.