Natural Language Rubric

A natural language rubric is a scoring guide for AI answers, written in plain sentences by a person. Think of a teacher deciding ahead of time: "meet this condition for 3 points, meet that one too for 5 points." That kind of pre-set standard makes grading fairer and faster. A natural language rubric works the same way. If you write down criteria like "did the agent confirm the user's request first" or "did it pass along all the necessary information without missing anything," you can then use those criteria to automatically grade answers.

This approach is especially useful when there isn't a single fixed correct answer. Take a voice AI, for example — the same meaning can be expressed in hundreds of different phrasings. "Yes, got it" and "Confirmed, I'll take care of it" use different words but could both be valid responses. In cases like this, comparing against one fixed "correct" sentence doesn't work. Instead, a human writes the judging criteria once, and those criteria can be automatically applied over and over to every new conversation.

In the end, a natural language rubric is a tool that takes over the work a human once had to do by listening to and judging each case by hand. The standards a human set stay exactly as they are — it's just the repetitive verification work that gets handed off to a machine.

The article explains that "grading is done through natural language rubrics… once a human writes down the judging criteria in sentence form, the same standard is automatically applied to every conversation afterward." A common misunderstanding here is that this means the AI decides for itself what counts as correct. It doesn't — the standard for what counts as correct is still set by a human in sentence form. The AI's role is only to apply that standard consistently to each new conversation.

To get a feel for how a natural language rubric actually works, try asking a chatbot something like this:

Here's a transcript of a conversation between a customer service agent and a user. Based on the criteria below, answer yes or no for each one and explain your reasoning in one line.

1. Did the agent clearly confirm the user's request?
2. Did it convey all necessary information without leaving anything out?
3. Did it avoid ignoring anything the user interjected?

[Paste the conversation here]

By setting the criteria sentences first and applying them the same way across multiple conversations, you can directly experience how a human-defined grading standard gets applied automatically and repeatedly.