VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

arXiv:2607.273802026-07-28

Teaching video-generation AI to draft physics in runnable Blender code before making the final realistic clip

Text-to-video models struggle with physically consistent motion because a short prompt forces them to infer the entire timeline implicitly. VideoCoCo has a coding agent write and run a Blender program to render a low-fidelity but physically dense draft video, then has an editing model turn that draft into a photorealistic final video. On two physics benchmarks, adding VideoCoCo to the OmniWeaving base model raised average scores from 0.475 to 0.558 on PhyGenBench and from 52.18% to 77.88% on VBench-2.0.

METAL LAB explanatory visual

VideoCoCo dual-engine pipeline

Evidence statusMeasured results reported

Text promptUser's compressed description of a physical event
Coding agent & simulation engineWrites and runs a Blender Python program in a sandbox to render a temporally dense, low-fidelity white-clay draft video
Instruction agentReads the prompt and draft together to write an appearance-focused editing instruction covering subjects, materials, lighting, and style
Generative video engine (editor)A video editor adapted with LoRA on VideoCoCo-3K turns the draft plus instruction into the final photorealistic video
Benchmark validationAverage scores measured against OmniWeaving baseline on PhyGenBench and VBench-2.0

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The paper argues that prior chain-of-thought approaches for video (textual plans, keyframes, or candidate search) are either non-executable or temporally sparse, limiting control over the full spatiotemporal process.
Stage 1, the executable simulation engine: a coding agent writes a self-contained Blender Python program specifying the scene and its temporal evolution, then a sandbox renders it into a deterministic, low-fidelity 'white-clay' draft video where every frame is physically instantiated.
Stage 2, the generative video engine: an instruction agent reads both the original prompt and the draft to compose an appearance-focused editing instruction (subjects, materials, lighting, style), which together with the draft conditions a video editor to produce the final photorealistic video.
The team built VideoCoCo-3K, a dataset of draft-instruction-target triplets generated via an agentic pipeline and a high-fidelity teacher editor (Seedance 2.0), used to adapt the editor to consume simulated drafts.
An ablation on editor adaptation found that even with no fine-tuning, conditioning on the draft alone improved scores (0.475 to 0.506), and lightweight LoRA fine-tuning beat full fine-tuning (0.558 vs 0.535).

Figure 1: Comparison of chain-of-thought paradigms for video generation. (1) Planning CoT externalizes reasoning as textual plans, keyframes, or layouts that condition a video generator. (2) Test-Time Search CoT samples multiple candidate videos and revises or selects among them at inference time. (3) Visual-State CoT reasons through a sequence of intermediate visual states within the generated video. In contrast, VideoCoCo (Ours) uses a code agent to synthesize executable code, renders a deterministic sandbox draft, and conditions the video generator on this draft, yielding a complete and inspectable process-level chain of thought rather than sparse or non-executable intermediates.

Table 1: Physical-consistency comparison on PhyGenBench across closed- and open-source video generators. We report per-category consistency scores in [0,1] for mechanics, optics, thermal, and material dynamics, together with their average; higher is better. Adding VideoCoCo to the OmniWeaving base generator lifts every category and raises the average from 0.48 to 0.56, the best overall result, with the largest gains on material and thermal dynamics, where appearance-driven priors are weakest. Orange and Champagne cells denote the best (top-1) and second-best (top-2) results per column.
	Method	Mechanics (↑)	Optics (↑)	Thermal (↑)	Material (↑)	Average (↑)
Closed	Pika [28]	0.35	0.56	0.43	0.39	0.44
Gen-3 [29]	0.45	0.57	0.49	0.51	0.51
Kling [15]	0.45	0.58	0.50	0.40	0.49
Open	CogVideoX [48]	0.39	0.55	0.40	0.42	0.45
Open-Sora V1.2 [54]	0.43	0.50	0.44	0.37	0.44
LaVie [39]	0.30	0.44	0.38	0.32	0.36
Vchitect-2.0 [6]	0.41	0.56	0.44	0.37	0.45
HunyuanVideo [13]	0.33	0.39	0.26	0.30	0.33
Wan2.2-TI2V-5B [34]	0.55	0.58	0.53	0.50	0.54
Cosmos-Predict2.5 [25]	0.30	0.33	0.41	0.39	0.35
LTX-Video-2B [7]	0.51	0.58	0.48	0.45	0.51
Ours	OmniWeaving [27]	0.48	0.56	0.43	0.39	0.48
+ VideoCoCo	0.56	0.61	0.51	0.53	0.56

Figure 2: Overview of the VideoCoCo dual-engine framework. Stage 1 (Executable Simulation Engine): given a user prompt, a coding agent plans and writes a self-contained Blender program that specifies the scene and its temporal evolution; executing this program in a sandbox renders a low-fidelity spatiotemporal draft. Stage 2 (Generative Video Engine): conditioned on the stage-1 prompt and draft video, an instruction agent composes an editing instruction, which together with the draft is passed to a video editing model to produce the final high-fidelity, physically consistent video. The executable draft supplies the process-level dynamics, while the editing model supplies photorealistic appearance.

Table 2: Physical-plausibility comparison on VBench-2.0, reported as per-dimension plausibility percentages over mechanics, thermotics, and material, with their average; higher is better. Coupling VideoCoCo with OmniWeaving improves the average from 52.18% to 77.88% (a gain of 25.70 points) and attains the best mechanics and thermotics scores among all systems, trailing only CogVideoX-1.5 on material. Orange and Champagne cells denote the best (top-1) and second-best (top-2) results per column.
	Method	Mechanics (↑)	Thermotics (↑)	Material (↑)	Average (↑)
Closed	Sora [3]	62.22%	43.36%	64.94%	56.84%
Kling 1.6 [14]	65.55%	59.46%	68.00%	64.34%
Open	HunyuanVideo [13]	76.09%	56.52%	64.37%	65.66%
CogVideoX-1.5 [48]	80.80%	67.13%	83.19%	77.04%
Ours	OmniWeaving [27]	62.79%	52.08%	41.67%	52.18%
+ VideoCoCo	92.31%	72.92%	68.42%	77.88%

Figure 3: Qualitative comparison on representative physical processes. OmniWeaving often produces visually plausible videos that violate the requested dynamics, whereas VideoCoCo follows the executable spatiotemporal draft and better preserves the intended processes, including sublimation, vacuum-induced collapse, impact shattering, and buoyancy.

Table 3: Ablation of video-editor adaptation strategies on PhyGenBench. All variants share the same executable draft-generation pipeline and differ only in how the editor is adapted: Tuning-Free, Full-Tune, or LoRA-Tune. The Tuning-Free variant already surpasses OmniWeaving (0.48 → 0.51), isolating the contribution of executable drafting, while LoRA-Tune yields the best average (0.56), showing that drafting and editor adaptation are complementary. Bold and underline denote the best (top-1) and second-best (top-2) results per column.
Method	Mech. (↑)	Opt. (↑)	Therm. (↑)	Mat. (↑)	Avg. (↑)
OmniWeaving [27]	0.48	0.56	0.43	0.39	0.48
+ VideoCoCo (Tune-Free)	0.50	0.53	0.48	0.51	0.51
+ VideoCoCo (Full-Tune)	0.51	0.61	0.51	0.49	0.54
+ VideoCoCo (LoRA-Tune)	0.56	0.61	0.51	0.53	0.56

Findings

On PhyGenBench, adding VideoCoCo to OmniWeaving raised the average physical-consistency score from 0.475 to 0.558, the best result among all systems compared, with the largest gains in material (+0.133) and thermal (+0.078) dynamics.
On VBench-2.0, OmniWeaving alone scored 52.18% on average, and adding VideoCoCo raised it to 77.88% (a 25.70-point gain), attaining the best mechanics (92.31%) and thermotics (72.92%) scores and ranking second on material behind CogVideoX-1.5 (83.19%).
In the editor-adaptation ablation on PhyGenBench, the tuning-free variant (draft conditioning only) already improved the average from 0.475 to 0.506, full fine-tuning reached 0.535, and LoRA fine-tuning achieved the best average of 0.558.

Where it can be used

Video content pipelines that need explicit physical processes such as sublimation, vaporization/freezing, impact collision, or buoyancy rendered correctly.
Workflows where a prompt-only simulation draft is generated and checked first, then converted into a photorealistic clip.
Educational or visualization settings where intermediate physical steps need to be inspected or edited as code before final rendering.

Limits and open work

The authors note that highly complex phenomena like turbulent fluids remain hard to synthesize zero-shot because they exceed the expressiveness of the Blender simulator.
The two-stage draft-then-edit process adds extra inference latency compared to single-stage generation.
The authors state future work will explore integrating specialized physics engines like Taichi and distilling these executable priors directly into end-to-end video models.

Why it matters

Physically implausible motion is a persistent weak point of video generators aiming to serve as world models or production tools, and this work shows that fixing the process first in inspectable code, then handling appearance separately, meaningfully narrows that gap. Because the program, draft, and instruction are all readable artifacts, the whole pipeline is easier to verify, debug, and reproduce than opaque end-to-end generation.

Terms in this paper

Chain-of-Thought (CoT) · Having a model work through intermediate reasoning steps instead of jumping straight to an answer
Blender · Open-source 3D software that can define scenes and run physics simulations
Sandbox · An isolated environment where a program is run safely and deterministically
LoRA · A lightweight fine-tuning method that updates only small low-rank parameter additions instead of the whole model
OmniWeaving · The baseline video generator used as the comparison point in this study

Original abstract (English)

Text-to-video models have achieved remarkable visual quality, yet they still struggle to generate physically consistent dynamics because the temporal evolution of a scene must be inferred implicitly from a highly compressed text prompt. Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-executable or temporally sparse, limiting their ability to instantiate and control the complete spatiotemporal process. To address this

Authors · Haodong Li

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Haodong Li et al., arXiv:2607.27380, CC BY 4.0