工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

arXiv:2607.273802026-07-28

让视频生成AI先用可执行的Blender代码把物理过程画成草稿,再变成逼真视频

文本生成视频模型常常在物理动态上出错,因为一句压缩的提示词要求模型隐式推断整个时间演化过程。VideoCoCo让编码智能体编写并运行Blender程序,渲染出画质低但物理上逐帧一致的草稿视频,再由编辑模型把这个草稿转化为逼真的最终视频。在两个物理一致性基准上,把VideoCoCo加到基线模型OmniWeaving上,PhyGenBench平均分从0.475提升到0.558,VBench-2.0平均分从52.18%提升到77.88%。

METAL LAB 解读图

VideoCoCo双引擎流程

证据状态已报告实测结果

  1. 文本提示词用户输入的压缩物理事件描述
  2. 编码智能体与仿真引擎编写并在沙盒中运行Blender Python程序,渲染出时间上密集但画质低的白模草稿视频
  3. 指令智能体同时阅读提示词和草稿,写出聚焦主体、材质、光照与风格的编辑指令
  4. 生成视频引擎(编辑模型)在VideoCoCo-3K上用LoRA适配的编辑模型,以草稿和指令为条件生成逼真的最终视频
  5. 基准验证在PhyGenBench和VBench-2.0上测得相对OmniWeaving基线的平均分提升
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 论文指出此前的思维链方法(文本计划、关键帧或候选搜索)要么不可执行,要么时间上过于稀疏,难以完整控制时空过程。
  2. 第一阶段可执行仿真引擎:编码智能体根据提示词编写一个自包含的Blender Python程序,明确场景及其随时间的演变,再由沙盒执行渲染出画质低但每一帧都物理确定的白模草稿视频。
  3. 第二阶段生成视频引擎:指令智能体同时读取原始提示词和草稿视频,写出聚焦外观(主体、材质、光照、镜头风格)的编辑指令,连同草稿一起输入视频编辑模型,生成逼真的最终视频。
  4. 团队构建了VideoCoCo-3K数据集,包含草稿-指令-目标三元组,由智能体流水线和高保真教师模型Seedance 2.0生成,用于让编辑模型学会处理仿真草稿。
  5. 编辑器适配方式的消融实验显示,即使不做任何微调,仅靠草稿条件就能提升分数(0.475到0.506),而轻量级LoRA微调效果优于全参数微调(0.558对0.535)。
Figure 1: Comparison of chain-of-thought paradigms for video generation. (1) Planning CoT externalizes reasoning as textual plans, keyframes, or layouts that condition a video generator. (2) Test-Time Search CoT samples multiple candidate videos and revises or selects among them at inference time. (3) Visual-State CoT reasons through a sequence of intermediate visual states within the generated video. In contrast, VideoCoCo (Ours) uses a code agent to synthesize executable code, renders a deterministic sandbox draft, and conditions the video generator on this draft, yielding a complete and inspectable process-level chain of thought rather than sparse or non-executable intermediates.
Figure 1: Comparison of chain-of-thought paradigms for video generation. (1) Planning CoT externalizes reasoning as textual plans, keyframes, or layouts that condition a video generator. (2) Test-Time Search CoT samples multiple candidate videos and revises or selects among them at inference time. (3) Visual-State CoT reasons through a sequence of intermediate visual states within the generated video. In contrast, VideoCoCo (Ours) uses a code agent to synthesize executable code, renders a deterministic sandbox draft, and conditions the video generator on this draft, yielding a complete and inspectable process-level chain of thought rather than sparse or non-executable intermediates.
Table 1: Physical-consistency comparison on PhyGenBench across closed- and open-source video generators. We report per-category consistency scores in [0,1] for mechanics, optics, thermal, and material dynamics, together with their average; higher is better. Adding VideoCoCo to the OmniWeaving base generator lifts every category and raises the average from 0.48 to 0.56, the best overall result, with the largest gains on material and thermal dynamics, where appearance-driven priors are weakest. Orange and Champagne cells denote the best (top-1) and second-best (top-2) results per column.
MethodMechanics (↑)Optics (↑)Thermal (↑)Material (↑)Average (↑)
ClosedPika [28]0.350.560.430.390.44
Gen-3 [29]0.450.570.490.510.51
Kling [15]0.450.580.500.400.49
OpenCogVideoX [48]0.390.550.400.420.45
Open-Sora V1.2 [54]0.430.500.440.370.44
LaVie [39]0.300.440.380.320.36
Vchitect-2.0 [6]0.410.560.440.370.45
HunyuanVideo [13]0.330.390.260.300.33
Wan2.2-TI2V-5B [34]0.550.580.530.500.54
Cosmos-Predict2.5 [25]0.300.330.410.390.35
LTX-Video-2B [7]0.510.580.480.450.51
OursOmniWeaving [27]0.480.560.430.390.48
+ VideoCoCo0.560.610.510.530.56
Figure 2: Overview of the VideoCoCo dual-engine framework. Stage 1 (Executable Simulation Engine): given a user prompt, a coding agent plans and writes a self-contained Blender program that specifies the scene and its temporal evolution; executing this program in a sandbox renders a low-fidelity spatiotemporal draft. Stage 2 (Generative Video Engine): conditioned on the stage-1 prompt and draft video, an instruction agent composes an editing instruction, which together with the draft is passed to a video editing model to produce the final high-fidelity, physically consistent video. The executable draft supplies the process-level dynamics, while the editing model supplies photorealistic appearance.
Figure 2: Overview of the VideoCoCo dual-engine framework. Stage 1 (Executable Simulation Engine): given a user prompt, a coding agent plans and writes a self-contained Blender program that specifies the scene and its temporal evolution; executing this program in a sandbox renders a low-fidelity spatiotemporal draft. Stage 2 (Generative Video Engine): conditioned on the stage-1 prompt and draft video, an instruction agent composes an editing instruction, which together with the draft is passed to a video editing model to produce the final high-fidelity, physically consistent video. The executable draft supplies the process-level dynamics, while the editing model supplies photorealistic appearance.
Table 2: Physical-plausibility comparison on VBench-2.0, reported as per-dimension plausibility percentages over mechanics, thermotics, and material, with their average; higher is better. Coupling VideoCoCo with OmniWeaving improves the average from 52.18% to 77.88% (a gain of 25.70 points) and attains the best mechanics and thermotics scores among all systems, trailing only CogVideoX-1.5 on material. Orange and Champagne cells denote the best (top-1) and second-best (top-2) results per column.
MethodMechanics (↑)Thermotics (↑)Material (↑)Average (↑)
ClosedSora [3]62.22%43.36%64.94%56.84%
Kling 1.6 [14]65.55%59.46%68.00%64.34%
OpenHunyuanVideo [13]76.09%56.52%64.37%65.66%
CogVideoX-1.5 [48]80.80%67.13%83.19%77.04%
OursOmniWeaving [27]62.79%52.08%41.67%52.18%
+ VideoCoCo92.31%72.92%68.42%77.88%
Figure 3: Qualitative comparison on representative physical processes. OmniWeaving often produces visually plausible videos that violate the requested dynamics, whereas VideoCoCo follows the executable spatiotemporal draft and better preserves the intended processes, including sublimation, vacuum-induced collapse, impact shattering, and buoyancy.
Figure 3: Qualitative comparison on representative physical processes. OmniWeaving often produces visually plausible videos that violate the requested dynamics, whereas VideoCoCo follows the executable spatiotemporal draft and better preserves the intended processes, including sublimation, vacuum-induced collapse, impact shattering, and buoyancy.
Table 3: Ablation of video-editor adaptation strategies on PhyGenBench. All variants share the same executable draft-generation pipeline and differ only in how the editor is adapted: Tuning-Free, Full-Tune, or LoRA-Tune. The Tuning-Free variant already surpasses OmniWeaving (0.48 → 0.51), isolating the contribution of executable drafting, while LoRA-Tune yields the best average (0.56), showing that drafting and editor adaptation are complementary. Bold and underline denote the best (top-1) and second-best (top-2) results per column.
MethodMech. (↑)Opt. (↑)Therm. (↑)Mat. (↑)Avg. (↑)
OmniWeaving [27]0.480.560.430.390.48
+ VideoCoCo (Tune-Free)0.500.530.480.510.51
+ VideoCoCo (Full-Tune)0.510.610.510.490.54
+ VideoCoCo (LoRA-Tune)0.560.610.510.530.56

研究结果

  • 在PhyGenBench上,把VideoCoCo加到OmniWeaving上后平均物理一致性得分从0.475升至0.558,是表中所有系统里的最好成绩,其中材料动态(+0.133)和热学动态(+0.078)提升最大。
  • 在VBench-2.0上,OmniWeaving单独得分为52.18%,加入VideoCoCo后升至77.88%,提升25.70个百分点,并在力学(92.31%)和热学(72.92%)上取得最高分,材料维度以68.42%排名第二,仅次于CogVideoX-1.5的83.19%。
  • 在PhyGenBench上对编辑器适配方式的消融实验中,不做任何微调、仅用草稿作条件的版本已将平均分从0.475提升到0.506;全参数微调达到0.535;LoRA微调取得最佳平均分0.558。

可应用场景

  • 需要精确呈现升华、蒸发/凝固、撞击破碎、浮力等明确物理过程的视频内容制作流程。
  • 先根据提示词生成并检查一个物理仿真草稿,再将其转化为逼真视频的工作流设计。
  • 教育或可视化场景中,需要将物理现象的中间步骤以代码形式检查或修改后再渲染的场合。

局限与待验证事项

  • 作者指出,像湍流这类高度复杂的现象超出了Blender仿真器的表达能力,难以零样本合成。
  • 草稿渲染加编辑的两阶段流程比单阶段生成增加了额外的推理延迟。
  • 作者表示未来工作将探索整合Taichi等专用物理引擎,并研究通过知识蒸馏把这些可执行先验直接内化到端到端视频模型中。

为什么重要

视频生成模型在物理动态上的不一致一直是把它们用作世界模型或内容生产工具的主要障碍,这项工作说明先用可执行代码固定过程、再单独处理外观的方式能明显缩小这一差距。由于程序、草稿视频和编辑指令都是可读的中间产物,整条流程比端到端的黑箱生成更容易检查、调试和复现。

本文术语

  • 思维链(Chain-of-Thought, CoT) · 让模型先走过中间推理步骤,而不是直接给出结果
  • Blender · 一款可以搭建三维场景并运行物理仿真的开源软件
  • 沙盒(Sandbox) · 一个隔离、安全且行为确定的程序运行环境
  • LoRA · 一种只更新少量低秩参数、而非整个模型的轻量级微调方法
  • OmniWeaving · 本研究用作对比基线的现有视频生成模型

论文原文摘要（英文）

Text-to-video models have achieved remarkable visual quality, yet they still struggle to generate physically consistent dynamics because the temporal evolution of a scene must be inferred implicitly from a highly compressed text prompt. Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-executable or temporally sparse, limiting their ability to instantiate and control the complete spatiotemporal process. To address this

作者 · Haodong Li

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Haodong Li et al., arXiv:2607.27380, CC BY 4.0