replay

Replay is a way to evaluate AI by pausing a real conversation or scenario transcript at a midpoint, letting an AI take over and continue from that point, and then scoring what it produces. It's a bit like replaying just the decisive moment in a baseball broadcast to ask "what would a different player have done here?"

Say there's a real transcript of a math tutoring session between a student and a tutor. At the exact moment the student gets stuck and needs help, the transcript is paused. From that point on, an AI takes over the tutor's role and continues the conversation — and the student's role is also played by another AI, not a real person. The newly generated conversation is then graded on whether the AI tutor gave just the right hint at that moment, or gave away the answer too easily.

This approach is useful because instead of evaluating an AI on an entire conversation from start to finish, it lets you zero in on the exact moments where a human made a consequential judgment call. By placing the human's actual response and the AI's response side by side at the same point, you can examine the AI's in-the-moment judgment much more precisely.

The article explains: "The evaluation method is 'replay.' The transcript is paused at a key decision point, and a language model takes over as the tutor, carrying out a five-turn conversation with a simulated student played by another language model." The key point — often misunderstood — is that this isn't simply replaying the old recording as-is; rather, the AI generates an entirely new conversation from that point onward, which is then graded.