NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

arXiv:2608.132102026-08-12

测试AI能否看懂几小时日语长视频里那些没说出口的言外之意的基准

NARU 是一个基于155段日语长视频(总时长146.8小时)构建的基准测试,包含1,481道题目,用来考察AI模型能否既追踪不断发展的剧情,又读懂日本文化中常常不明说的社交暗示。由于人工逐帧标注如此长的视频代价太高,团队设计了一套流程,先让AI按片段生成结构化标注,再由68位日语母语者分两轮验证。测试结果显示,即便是顶尖商用模型在理解文化隐含意义上也表现吃力,开源模型则全面落后。

他们做了什么

团队收集了155段日语长视频(单个时长30分钟到数小时不等,总计146.8小时),据此设计出1,481道题目,分为考察人物和剧情演变的'叙事理解'题,以及类似'主人再给客人续一杯咖啡其实是在委婉暗示送客'这类需要读懂弦外之音的'文化理解'题。
构建流程把每段视频切成约5分钟一段,先由AI模型逐段生成人物、事件等结构化摘要,并参考前面片段的内容保持叙事连贯,再据此批量生成候选问题,并反复检查、剔除那些不看视频也能靠文字线索猜对的'投机取巧'题,最后由68位日语母语审核者分两轮确认。
Gemini-3-Flash 以76.2%的准确率表现最好,其次是Gemini-3-Pro(70.0%)和Gemini-2.5-Flash(51.4%),而开源模型的准确率只有29.6%到39.8%。
能力越强的模型,越容易在把握视频整体主题(N.4)上犯难;能力较弱的模型则连基本的人物追踪(比如某个角色是谁、状态如何变化)都常常出错。
把输入的采样帧数从8帧增加到128帧后,叙事理解的准确率明显提升(最高提升20.5个百分点),但文化理解的准确率几乎没有稳定改善,有时甚至下降,说明理解文化内涵更依赖背景知识和推理能力,而不只是看到更多画面。

TABLE I: Statistics of NARU.
Level	Task	Type of Evidence	Code	#
Narrative (N, 745)	Character/Entity Evolution	A character/entity across segments	N.1	185
	Sequential/Topical Flow	Events or topics over time	N.2	187
	Plot/Conflict Progression	A causal thread or conflict	N.3	186
	Idea/Thematic Development	Motifs, claims, or narrative cues	N.4	187
Cultural (C, 736)	Aizuchi (Conversational Mechanics)	Backchannels and response timing	C.1	143
	Kuuki wo Yomu (Situational Awareness)	Social atmosphere or implicit norms	C.2	147
	Subtext Interpretation	Surface utterance plus context	C.3	148
	Cultural Context Recognition	Culturally specific references	C.4	149
	Sentiment Analysis	Verbal, visual, and social cues	C.5	149

(b) Cultural Understanding: The model is asked to infer the host’s implicit intent when offering another cup of coffee to the guest.22 2 In certain situations in Japan, such offers are conventionally used as an indirect and polite way to signal the end of a visit. Understanding the host’s true intent requires awareness of the context and the surrounding social atmosphere, beyond the literal semantic content.

TABLE II: MCQ accuracy results on NARU. The best and the 2nd best results are marked.
Model	Sampling Rate	N.1	N.2	N.3	N.4	Narr. Avg	C.1	C.2	C.3	C.4	C.5	Cult. Avg	Overall
Gemini-3-Flash	0.25 FPS	83.8	90.9	83.9	78.1	84.2	71.3	69.4	57.4	73.2	69.8	68.2	76.2
Gemini-3-Pro	0.25 FPS	74.6	78.6	76.3	66.8	74.1	60.1	68.7	64.2	69.8	67.1	66.0	70.0
Gemini-2.5-Flash	0.25 FPS	51.3	63.1	58.1	50.8	55.8	37.8	49.7	48.6	50.3	48.3	46.9	51.4
Qwen3.5-9B	128 Frames	34.6	48.7	34.9	34.2	38.1	41.3	49.0	34.5	40.9	41.6	41.4	39.8
Qwen3VL-8B	0.25 FPS	35.7	46.5	39.8	35.8	39.5	32.2	40.1	33.1	38.9	32.9	35.4	37.4
Qwen2.5VL-7B	128 Frames	25.4	42.2	30.6	26.2	31.1	22.4	40.8	25.7	24.2	28.2	28.2	29.7
MiniCPM-o-2.6	128 Frames	23.2	41.2	27.4	31.6	30.8	25.9	37.4	27.7	22.1	28.2	28.3	29.6
InternVL3.5	64 Frames	20.0	36.4	25.8	31.0	28.3	32.2	41.5	31.8	35.6	32.2	34.6	31.5

Fig. 2: Workflow overview of NARU. Video collection and filtering (Sec. III-B) select 155 long videos from the candidate set. A MLLM-centric pipeline (Sec. III-C1) is used to produce taxonomy-aligned evidence for narrative intelligence and cultural understanding annotation. Multiple-choice questions (Sec. III-C2) are subsequently generated and refined through a multi-agent pipeline. Verification (Sec. III-D) by 68 native Japanese experts yields the final 1,481 QA items.

TABLE III: Open-ended correctness on NARU. Scores are FActScore recall (0–1): fraction of reference atomic facts covered by the answer.
Model	N.1	N.2	N.3	N.4	C.1	C.2	C.3	C.4	C.5	Avg
Gemini-3-Flash	0.78	0.66	0.72	0.75	0.69	0.87	0.93	0.85	0.80	0.78
Gemini-3-Pro	0.77	0.61	0.71	0.69	0.65	0.87	0.88	0.80	0.72	0.75
Gemini-2.5-Flash	0.62	0.50	0.58	0.65	0.56	0.77	0.84	0.68	0.73	0.66
Qwen3.5-9B	0.54	0.49	0.47	0.50	0.52	0.66	0.65	0.60	0.59	0.56
Qwen3-VL-8B	0.39	0.23	0.34	0.36	0.47	0.64	0.62	0.39	0.53	0.44
Qwen2.5-VL-7B	0.35	0.24	0.27	0.28	0.43	0.47	0.40	0.28	0.40	0.35
MiniCPM-o-2.6	0.21	0.12	0.15	0.16	0.23	0.26	0.30	0.15	0.31	0.21
InternVL3.5	0.40	0.14	0.27	0.35	0.43	0.49	0.47	0.39	0.51	0.38

Fig. 3: Overall multiple-choice accuracy as the number of sampled frames increases from 8 to 128; the dotted gray line denotes the 25% chance level.

为什么重要

对于想做长视频摘要或文化敏感内容审核的AI应用来说,模型不能只是罗列画面里发生了什么,还要读懂上下文和社交分寸。NARU具体揭示了当前模型在这种'高语境'理解上的薄弱环节,为下一代视频理解AI指明了需要改进的方向。

Fig. 4: Changes in narrative and cultural accuracy between 8 and 128 frames, reported in percentage points (pp).

本文术语

MLLM · 同时处理视频/图像与文本的多模态大语言模型
空気を読む(读空气) · 日本文化中不靠明说、而是通过察言观色理解现场氛围的沟通方式
相槌(aizuchi) · 日语对话中类似'嗯嗯'的简短回应,主要表示在听,不一定代表同意
建前与本音 · 建前指对外的场面话,本音指内心真实想法,两者常常并不一致
FActScore 召回率 · 一种评分方法,衡量模型回答覆盖了参考答案中多少个具体事实点

论文原文摘要（英文）

Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimensions. To construct the benchmark at this scale, we propose a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal. The construction process includes two native-speaker verification stages involving 68 annotators. Evaluations across eight model configurations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning. By exposing these persistent gaps, NARU offers a systematic testing ground for developing MLLMs capable of reliably interpreting long-form, high-context video.

作者 · Yuheng Huang

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Yuheng Huang et al., arXiv:2608.13210, arxiv-nonexclusive

NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道