mental-world/mentis

★ 37PythonMIT

An AI system that predicts what a person will do next by modeling not just the physical scene but also what they believe, want, and feel

Mentis is a reference implementation of 'Mental World Modeling,' a framework that tracks a character's hidden mental state (beliefs, desires, feelings, and sense of social norms) alongside the physical scene to predict their next action. It works with any off-the-shelf LLM without extra training, breaking the prediction process into inspectable steps: parsing the scene, rendering what the target character can actually perceive, simulating candidate actions, and scoring the outcomes. The accompanying Menti-Bench benchmark provides 448 test scenarios across text, images, and video to measure how well current LLMs handle these hidden mental variables.

What it does

Standard world models only track physical facts (what is where), so they fail on scenarios where hidden mental states matter, like someone searching for an object that was secretly moved.
Mentis runs a six-stage pipeline: parse the scene into a joint physical+mental state, render only what the target agent can actually see/know, split each candidate action into physical and mental effects, simulate the resulting state for each option, score the branches on three criteria (mental consistency, physical plausibility, social appropriateness) plus a safety check, then pick an action with a deterministic rule outside the LLM.
Every intermediate result is logged, so when a prediction is wrong, developers can pinpoint which stage failed rather than treating the model as a black box.
Menti-Bench, the paired benchmark, has 320 text scenarios, 100 image-sequence scenarios, and 28 sounding-video scenarios, each with deliberately hardened wrong answers that violate some clue in the scene.
The system requires no fine-tuning and works through the standard OpenAI-compatible chat API; a sample with 4 options costs 12 LLM calls, versus 1 call for a simple direct-answer baseline.

Why it matters

This matters for anyone building AI agents or assistants that need to reason about other people's intentions, false beliefs, or social norms, not just physical facts. It also gives researchers a transparent, step-by-step tool to diagnose exactly where LLMs go wrong when reasoning about other minds.

Terms in this repo

World model · An internal representation an AI uses to predict how a scene or situation will change over time
LLM (Large Language Model) · A large AI language model such as GPT that generates text based on patterns learned from data
Macro-F1 · An accuracy-like score that averages performance equally across all answer categories, useful when categories are imbalanced
Training-free · Works directly with an existing AI model without any additional training or fine-tuning
Safety veto · A rule that blocks selection of an action if it fails a safety check, regardless of its other scores

Repository description (English)

A Baseline Implementation of Mental World Model

Open on GitHub

Coverage

월드모델에 '마음' 변수 더하니 정확도 63→88점으로 뛰었다

Trending repos

All repos →