NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

arXiv:2608.132102026-08-12

A benchmark testing whether AI can watch hours of Japanese video and actually 'read the room'

NARU is a benchmark of 1,481 questions built from 155 long Japanese videos totaling 146.8 hours, testing whether AI models can both track an unfolding story and pick up on Japan's often-unspoken cultural cues. Because manually annotating such long videos is too costly, the team built a pipeline where an AI drafts structured annotations segment by segment, which 68 native Japanese speakers then verify. When tested, even top commercial models struggled badly with reading implicit cultural meaning, and open-source models lagged far behind across the board.

What they did

The team gathered 155 Japanese long-form videos (30 minutes to several hours each, 146.8 hours total) and created 1,481 questions split between 'narrative' tasks (tracking characters and plot over time) and 'cultural' tasks, like inferring that a host offering a guest more coffee can actually be a polite signal that it's time to leave.
A hierarchical pipeline breaks each video into roughly 5-minute chunks, has an AI model summarize entities and events chunk by chunk while referencing earlier context to keep the story consistent, and then generates candidate questions that are repeatedly checked to remove 'shortcuts' answerable without watching the video, before 68 native Japanese verifiers confirm them in two review stages.
Gemini-3-Flash scored highest at 76.2% accuracy, followed by Gemini-3-Pro (70.0%) and Gemini-2.5-Flash (51.4%), while open-source models scored only 29.6-39.8%.
Stronger models struggled most with grasping a video's overarching theme (N.4), while weaker models failed even at basic tasks like tracking who a character is and how they change (N.1).
Feeding models more frames (8 to 128) clearly improved narrative accuracy (up to a 20.5 percentage-point gain), but had little to no consistent effect on cultural understanding, sometimes even lowering it, showing that culture-reading depends more on background knowledge and reasoning than on simply seeing more of the video.

TABLE I: Statistics of NARU.
Level	Task	Type of Evidence	Code	#
Narrative (N, 745)	Character/Entity Evolution	A character/entity across segments	N.1	185
	Sequential/Topical Flow	Events or topics over time	N.2	187
	Plot/Conflict Progression	A causal thread or conflict	N.3	186
	Idea/Thematic Development	Motifs, claims, or narrative cues	N.4	187
Cultural (C, 736)	Aizuchi (Conversational Mechanics)	Backchannels and response timing	C.1	143
	Kuuki wo Yomu (Situational Awareness)	Social atmosphere or implicit norms	C.2	147
	Subtext Interpretation	Surface utterance plus context	C.3	148
	Cultural Context Recognition	Culturally specific references	C.4	149
	Sentiment Analysis	Verbal, visual, and social cues	C.5	149

(b) Cultural Understanding: The model is asked to infer the host’s implicit intent when offering another cup of coffee to the guest.22 2 In certain situations in Japan, such offers are conventionally used as an indirect and polite way to signal the end of a visit. Understanding the host’s true intent requires awareness of the context and the surrounding social atmosphere, beyond the literal semantic content.

TABLE II: MCQ accuracy results on NARU. The best and the 2nd best results are marked.
Model	Sampling Rate	N.1	N.2	N.3	N.4	Narr. Avg	C.1	C.2	C.3	C.4	C.5	Cult. Avg	Overall
Gemini-3-Flash	0.25 FPS	83.8	90.9	83.9	78.1	84.2	71.3	69.4	57.4	73.2	69.8	68.2	76.2
Gemini-3-Pro	0.25 FPS	74.6	78.6	76.3	66.8	74.1	60.1	68.7	64.2	69.8	67.1	66.0	70.0
Gemini-2.5-Flash	0.25 FPS	51.3	63.1	58.1	50.8	55.8	37.8	49.7	48.6	50.3	48.3	46.9	51.4
Qwen3.5-9B	128 Frames	34.6	48.7	34.9	34.2	38.1	41.3	49.0	34.5	40.9	41.6	41.4	39.8
Qwen3VL-8B	0.25 FPS	35.7	46.5	39.8	35.8	39.5	32.2	40.1	33.1	38.9	32.9	35.4	37.4
Qwen2.5VL-7B	128 Frames	25.4	42.2	30.6	26.2	31.1	22.4	40.8	25.7	24.2	28.2	28.2	29.7
MiniCPM-o-2.6	128 Frames	23.2	41.2	27.4	31.6	30.8	25.9	37.4	27.7	22.1	28.2	28.3	29.6
InternVL3.5	64 Frames	20.0	36.4	25.8	31.0	28.3	32.2	41.5	31.8	35.6	32.2	34.6	31.5

Fig. 2: Workflow overview of NARU. Video collection and filtering (Sec. III-B) select 155 long videos from the candidate set. A MLLM-centric pipeline (Sec. III-C1) is used to produce taxonomy-aligned evidence for narrative intelligence and cultural understanding annotation. Multiple-choice questions (Sec. III-C2) are subsequently generated and refined through a multi-agent pipeline. Verification (Sec. III-D) by 68 native Japanese experts yields the final 1,481 QA items.

TABLE III: Open-ended correctness on NARU. Scores are FActScore recall (0–1): fraction of reference atomic facts covered by the answer.
Model	N.1	N.2	N.3	N.4	C.1	C.2	C.3	C.4	C.5	Avg
Gemini-3-Flash	0.78	0.66	0.72	0.75	0.69	0.87	0.93	0.85	0.80	0.78
Gemini-3-Pro	0.77	0.61	0.71	0.69	0.65	0.87	0.88	0.80	0.72	0.75
Gemini-2.5-Flash	0.62	0.50	0.58	0.65	0.56	0.77	0.84	0.68	0.73	0.66
Qwen3.5-9B	0.54	0.49	0.47	0.50	0.52	0.66	0.65	0.60	0.59	0.56
Qwen3-VL-8B	0.39	0.23	0.34	0.36	0.47	0.64	0.62	0.39	0.53	0.44
Qwen2.5-VL-7B	0.35	0.24	0.27	0.28	0.43	0.47	0.40	0.28	0.40	0.35
MiniCPM-o-2.6	0.21	0.12	0.15	0.16	0.23	0.26	0.30	0.15	0.31	0.21
InternVL3.5	0.40	0.14	0.27	0.35	0.43	0.49	0.47	0.39	0.51	0.38

Fig. 3: Overall multiple-choice accuracy as the number of sampled frames increases from 8 to 128; the dotted gray line denotes the 25% chance level.

Why it matters

For anyone building AI tools to summarize long videos or moderate culturally sensitive content, models need to grasp context and social nuance, not just list what happens on screen. NARU exposes exactly where today's models fall short at this 'high-context' understanding, pointing to what the next generation of video-understanding AI needs to fix.

Fig. 4: Changes in narrative and cultural accuracy between 8 and 128 frames, reported in percentage points (pp).

Terms in this paper

MLLM · a multimodal large language model trained to jointly process video/image and text
kuuki wo yomu (空気を読む) · a Japanese communication norm of picking up on unspoken social atmosphere, literally 'reading the air'
aizuchi (相槌) · short listener responses like 'uh-huh' in Japanese conversation, signaling attention rather than necessarily agreement
tatemae / honne (建前・本音) · the distinction between one's public-facing expression (tatemae) and private true intent (honne)
FActScore recall · a scoring method measuring what fraction of a reference answer's individual facts a model's response actually covers

Original abstract (English)

Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimensions. To construct the benchmark at this scale, we propose a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal. The construction process includes two native-speaker verification stages involving 68 annotators. Evaluations across eight model configurations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning. By exposing these persistent gaps, NARU offers a systematic testing ground for developing MLLMs capable of reliably interpreting long-form, high-context video.

Authors · Yuheng Huang

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Yuheng Huang et al., arXiv:2608.13210, arxiv-nonexclusive