Psychometrics

Psychometrics is the field and set of techniques used to measure things you can't see directly, like intelligence, personality, or aptitude, through test questions and scores. When schools build exams, they need to statistically check whether a question actually distinguishes stronger students from weaker ones, and whether a set of questions measures one single ability or ends up mixing several different abilities together. Psychometrics is the toolbox for exactly this kind of verification.

For example, if every single student answers a question correctly, or every single student gets it wrong, that question is useless for telling who is better at the subject. Psychometrics helps identify and filter out such useless items, and it also reveals whether several tests that look similar on the surface are actually measuring different things underneath.

Recently, these techniques have started being applied not just to people but to evaluations of AI language models. Psychometric analysis has revealed that safety benchmarks—which score how well an AI refuses dangerous requests—were often not measuring one single, unified notion of "safety," but were instead mixing together several unrelated properties.

The article reports that "the same psychometric techniques used to build human IQ tests and aptitude exams were applied directly to analyzing language model safety benchmarks." A common misunderstanding here is that psychometrics is not a technology that makes AI smarter or safer. Rather, it's a statistical methodology for checking whether a test itself was properly constructed and what it's actually measuring.