action label

An action label is a log that notes, moment by moment, 'this control was pressed at this instant.' Take a game as an example: from video alone, an AI can't tell why a character moved the way it did. But if the footage is paired with records of 'which button was pressed at which second,' the AI can learn the relationship between what's on screen and the controls behind it. That control record is the action label.

Think of it like cooking: there's a difference between showing only a photo of the finished dish and providing a recipe that lists which ingredient was added when and in what amount. Just as a recipe lets a chef reproduce the process, action labels let an AI learn to translate movements seen in video into actual actions.

Recently, action labels have been used in attempts to collect gameplay records from players and transfer those patterns into physical movements for real robots. Finger movements on a game controller differ from a robot arm's movements, but the underlying pattern of 'what action is chosen in what situation' is thought to be transferable.

The article reports that General Intuition used video collected from a game-clip sharing platform, along with 'action labels' recording when and which buttons players pressed, as its initial training data. The key point, often misunderstood, is that this isn't the video itself but the 'control record' attached to the video. Video alone can't tell an AI the cause behind on-screen movement, but the control record lets it learn the relationship between actions and outcomes.