INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models

arXiv:2607.260562026-07-27

无需昂贵搜索,世界模型也能把想要的目标直接变成可执行的动作

以往的潜在世界模型只能预测某个动作会如何改变场景,而要找到能实现期望变化的动作,通常需要在测试时用CEM等方法反复采样、模拟、筛选,代价很高。INTACT用同一个预测器,以相同的输入格式同时学习真实的下一状态变化和目标状态变化,使得部署时只给出目标就能直接输出动作,无需搜索。在PushT、Cube、Reacher、TwoRoom四个视觉任务上用一个共享编码器联合训练后,这种免搜索的Direct控制在成功率上优于或接近基于搜索的基线,同时速度大幅提升。

METAL LAB 解读图

INTACT的训练与控制流程

证据状态已报告实测结果

共享编码器将PushT、Cube、Reacher、TwoRoom四个视觉领域的图像压缩为同一潜在表征z。
局部意图输入真实的下一状态变化z(t+1)-z(t)输入INTACT Predictor,并用示范动作监督,锚定物理可达性。
目标意图输入目标状态与当前状态之差sg(z_goal)-z(t)输入同一预测器,但目标一侧梯度被截断,形成部署时的动作接口。
Forward Predictor单独模块继续预测未来潜在状态并用SIGReg正则化,保留接触、障碍物等超出即时动作所需的信息。
Direct控制/可选验证推理时只给目标即可直接输出动作块(Direct模式),也可用小规模局部CEM(Guarded A)进行验证或重新规划。

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

问题背景:标准的潜在世界模型只能在给定动作后预测场景如何变化,要反推出能实现期望变化的动作,通常需要昂贵的测试时搜索(如CEM/MPPI),即采样大量候选动作序列、在模型中模拟并挑选最优。
核心思路:把两种意图输入同一个共享预测器——真实的下一状态变化(z(t+1)-z(t),局部意图)和目标状态变化(sg(z_g)-z(t),目标意图),两者都用示范中实际执行的动作来监督,但目标一侧的梯度被截断(stop-gradient),不干扰局部学习。
方法实现:一个共享视觉编码器加上各任务专用的小型预测头,联合训练于PushT、Cube、Reacher、TwoRoom四个领域;另设一个Forward Predictor继续预测未来状态并用SIGReg正则化,保留超出即时动作所需的世界信息(如接触、障碍物等)。
结果:免搜索的Direct控制每次规划仅需约2.9到5.5毫秒,相比实测的CEM 300x30平均1.48秒,速度提升约300倍;若再加入轻量级局部验证(Guarded A,128x3候选),四任务宏平均成功率达96.86%,最差任务成功率达92.22%,比匹配的纯CEM 300x30高16个百分点。
额外验证:即使去掉动作输出头、只用表征本身重新跑CEM(actor-disabled CEM),性能依然优于基线世界模型,说明提升的不仅是策略头本身,底层表征也变得更有用。

Figure 1: Shared-encoder INTACT training and recurrent control. Four visual domains use one encoder and task-specific predictor pairs. Within each task, graph-isomorphic local and goal calls enter the same INTACT Predictor through one matched input grammar, using attached local intent zt+1k−ztk or detached goal intent sg⁡(zgk)−ztk. The Forward Predictor retains latent-dynamics and SIGReg supervision. At inference, the goal-conditioned call emits one action chunk for search-free Direct control; the world rollout is retained only for optional verification and replanning.

Table 1: Positioning among closely related methods. “Action→E” means that action supervision updates the visual/world encoder, and “Shared-4T E” means one encoder is jointly trained over all four official LeWM domains. The final column states each method’s actual training-to-control path; a cross denotes an unreported interface, not an impossible extension.
Method	Action→E	Shared-4T E	Condition	Action obj.	Direct	Search	Training-to-control path
DINO-WM [40]	✗	✗	action	none	✗	✓	Frozen image encoder; CEM ranks latent rollouts
C-JEPA [22]	✗	✗	action/proprio.	none	✗	✓	Frozen object features; CEM searches masked rollouts
LeWM [21]	✗	✗	action	none	✗	✓	Joint JEPA prediction; CEM inverts the forward model
Fast-LeWM [9]	✗	✗	action prefix	none	✗	✓	LeWM training; parallel action prefixes are searched
SMWM [16]	✓	✗	successor	local action	✗	✓	Local inverse shapes the encoder; deployment still uses CEM
GC-IDM [23]	✗	✗	goal + horizon	next action	✓	✗	Frozen LeWM; a separate goal/horizon actor acts directly
PRISM [36]	✗	✗	state/goal	Gaussian prior	✗	✓	Frozen LeWM; a separately trained prior initializes MPPI
Qantara [28]	✓	✗	dispatch-dependent	action flow	✓	✓	Joint JEPA; bridge-flow heads serve dispatch modes
QuoVLA [35]	✓	✗	visual/language	action chunk	✓	✗	VLM action quotient; quantized intent drives a VLA actor
INTACT	✓	✓	local/goal intent	action block	✓	opt.	One shared JEPA/actor law; Direct by default, search verifies

Figure 2: INTACT converts mandatory search into optional verification. LeWM samples raw actions from an uninformed Gaussian and uses forward rollouts to become goal-directed. INTACT alternates the raw goal displacement, the shared conditional action mean, and the unchanged Forward Predictor to produce a Direct plan. It can execute immediately or locally verify that plan with K=128, I=3, and σ0=0.25.

Table 4: Theory-linked evidence for shared four-task INTACT. Correlations use official Direct SR; the actor-disabled and gauge rows are matched interventions that test representation and correspondence directly.
Evidence	Measured result	Interpretation
Pred.–expert kNN (45 eligible E1–E5 checkpoints)	Pooled 𝒓=.954 [.928,.969]; adjusted 𝒓=.902. Within Waypoint/Goal: .968/.981.	Local action-family neighborhoods track deployable control.
Pred.–expert linear CKA (same 45 checkpoints)	Pooled 𝒓=.897 [.837,.930]. Within Waypoint/Goal: .979/.986. Leave-one-epoch-out: [.888,.914].	Global family geometry remains predictive across interfaces and epochs.
Pointwise action R2 (same 45 checkpoints)	r=.815.	Recovering one expert action is weaker than preserving the family relation.
Effective-rank inversion (controlled E5 cells)	Rank 93.87>89.26, while SR 74.22<89.39.	Latent spread is a capacity check, not a semantic certificate.
Actor-disabled planning (matched E5 checkpoints)	Pure-CEM macro 66.17→70.08.	Action losses improve the Encoder–Forward stack before actor execution.
Paired gauge intervention (21,600 CLEAR Moderate episodes)	Shuffled 9.46%→68.04% paired (+58.58 pp).	Correct task-local correspondence, rather than coordinate identity, restores control.

Figure 3: Single-task action likelihoods shape the representation. Actor-disabled CEM evaluates the encoder–Forward stack under a fixed optimizer; episode-disjoint probes measure readable state and transition action. Physical inverse supplies attached successor shaping, whereas the goal branch updates current-state coordinates and the shared actor with a stop-gradient goal anchor. Error bars are sample standard deviations over three PushT training seeds.

Table A7: Controlled shared-encoder representation snapshots. σ¯ is mean per-dimension latent standard deviation, cosine is mean pairwise cosine, and action R2 is an episode-disjoint frozen probe.
Training cell	n	Rank	σ¯	Cos.	Act. R2
LeWM	3	25.53	.830	.554	.309
Inverse only	3	24.03	.902	.590	.372
Waypoint intent only	3	24.83	.846	.582	.328
Goal intent only	3	24.45	.839	.604	.349
Waypoint INTACT	3	25.73	.877	.587	.383
Goal INTACT	3	25.34	.888	.616	.382

Figure 4: Controlled E5 success across four tasks. Bars visualize the native rows of Table 3: LeWM uses CEM 300×30, learned heads use Direct, and the final bar uses Guarded A 128×3. Values are means ± sample standard deviations over three training seeds. The ordinate starts at 30% to expose high-SR differences and is not a zero-based effect size.

Table A10: Task-centered Pearson correlation with E5 Direct SR. These are diagnostic associations, not independent causal effects.
Metric	Moderate	Strict
Effective rank	0.888	0.846
Gsep	0.819	0.731
Deployment NLL	−0.640	−0.766

Figure 5: Intent–action relation, rather than task clustering, tracks control. Columns separate waypoint and goal-displacement interfaces; rows compare task purity, pointwise action R2, predicted–expert CKA, and local kNN overlap. Each point is one checkpoint and each annotation is a pooled checkpoint-level Pearson correlation with official four-task Direct SR. Historical runs are lower opacity and never rank the controlled objectives.

Table A13: PushT Direct SR (%) on the held-out 10% episode split. Each cell contains 100 paired test-only episodes.
Training objective	eval 0	eval 1	eval 42	pooled
Transition inverse	35.67	44.33	40.67	40.22
+ deployment goal intent	75.33	81.33	76.00	77.56

Figure A1: Controls used to select the intent grammar. (a) Coordinate and interaction interventions. (b) Re-encoding every 1, 2, or 5 action blocks. (c) Detaching the goal anchor and explicitly reducing chord error separate action-field learning from Euclidean latent straightening. Error bars are training-seed standard deviations.

Table A14: PushT successes on held-out expert episodes (100 per cell).
	Transition inverse	Goal intent
Training seed	e0	e1	e42	e0	e1	e42
3072	35	42	41	74	83	74
3073	36	44	41	81	81	79
3074	36	47	40	71	80	75

Figure A2: Verifier-selection controls. The complete three-seed audit compares Direct, guarded and unguarded plan-centered search, trust penalties, actor particles, and actor-disabled CEM.

Table A15: Historical automatic-success audit and CLEAR Moderate calibration. Historical random is the mean of three runs under the released stack. CLEAR values use canonical seeds 0/1/42 and 100 episodes per seed.
Task	Initially solved	Historical random	Moderate random
PushT	0.22	4.00	4.00
Cube	38.38	49.00	15.67
Reacher	0.57	13.00	4.33
TwoRoom	8.82	26.00	6.67

Figure A3: Exploratory E5 collapse morphologies from fixed samples. These legacy non-Math-SDPA runs are qualitative diagnostics and do not enter performance ranking or headline correlations.

Table A16: CLEAR-LeWM v0.5.1 task contracts. “Hold” is the number of consecutive successful environment steps. The 24-fold Cube comparison minimizes orientation error over the proper rotational symmetry group of a cube.
Task	Moderate: minimal compatibility correction	Strict: task-semantic precision
PushT	Released pusher + T position <20 px and T angle <20∘; first hit.	T object only, <10 px and <10∘; hold 3.
Cube	Initially unsolved, cube-center distance ≤4 cm; first hit.	Cube center ≤3 cm and symmetry-aware orientation ≤15∘; hold 3.
Reacher	Periodic unbounded shoulder, raw bounded wrist, joint error <0.05 rad; first hit.	Physical fingertip endpoint ≤1 cm; hold 2.
TwoRoom	Clean cross-room pair, continuous swept-disk collision, endpoint <16 px.	Legal doorway crossing, valid route, goal-side arrival, endpoint <8 px.

Figure A4: Search-free control and optional verification. Left: physical-successor inversion versus goal-displacement INTACT after one epoch. Right: Direct uses no candidate sequence, Guarded A locally verifies 384, and broad actor-disabled CEM evaluates 9,000. More search is not automatically better once the action conditional is aligned.

研究结果

在单任务PushT上,去掉动作头的CEM 30x10成功率从基线LeWM的42.22%提升到逆动力学监督下的57.67%、匹配的目标意图监督下的61.44%,INTACT(路点变体,SIGReg 0.03)结果更高。
免搜索的Direct控制每次规划耗时2.9到5.5毫秒(匹配的PushT为4.8毫秒),相比实测的以actor初始化的CEM 300x30平均1.48秒,约快300倍。
在全部12个最终检查点(共576项作业审计)中,Guarded A(Direct方案加局部128x3 CEM验证,σ0=0.25)取得宏平均成功率96.86%、最差任务成功率92.22%,比匹配的纯CEM 300x30高16.00个宏平均百分点,同时候选序列数从9000降至384(减少23.44倍)。
在四个任务上用单一共享编码器训练的Goal-displacement INTACT宏平均Direct成功率为±0.77%,相比匹配的共享编码器LeWM在四个任务上分别提升5.66/32.23/12.56/42.44个百分点,并超过了已发表的任务专属LeWM宏平均及Cube/Reacher分数。
在45个检查点上,预测动作与专家动作的kNN重叠度和CKA与官方成功率的Pearson相关系数分别为0.954和0.897,均高于逐点动作R2的0.815。

Figure A5: Gauge controls isolate correspondence from actor functionality. Correctly paired calibration restores control; recovery saturates near 64/full episodes; same-objective swaps define a ceiling; reverse swaps show that a coordinate map cannot create an untrained deployment conditional. The 72-cell audit contains 21,600 CLEAR Moderate episodes and excludes all evaluation episodes from calibration.

可应用场景

需要快速响应、无法承受穷举式搜索的目标导向机器人操作任务(如PushT类推物任务)
仅用无奖励标签的离线示范数据构建目标条件动作策略的场景
在现有CEM/MPPI规划器上加入轻量验证步骤(Guarded),兼顾速度与可靠性的系统

局限与待验证事项

已报告的结果仅限于PushT、Cube、Reacher、TwoRoom四个仿真任务,尚未验证在真实机器人或其他领域的泛化能力。
Direct控制的速度对比是论文自身的规划器侧测量结果,并非完整的视觉-语言-动作(VLA)系统端到端延迟基准。
在另一个经过校正的基准CLEAR-LeWM下,Reacher成功率从官方的97.67%大幅降至Moderate标准下的49.56%,暗示原基准可能存在天花板效应。
INTACT尚未完成Strict模式评估,目前只报告了Moderate结果。
部分辅助实验(如流匹配潜在预测器、双独立actor对比)属于探索性诊断,未纳入主要性能排名。

为什么重要

基于搜索的规划方法是机器人操作等需要快速响应场景中的主要计算瓶颈,这项工作表明搜索可以从“必须步骤”降级为“可选验证”,对延迟敏感的控制系统设计具有参考价值。同时它说明仅靠无奖励标签的离线示范数据,也能重组出可直接部署的目标条件控制器,为依赖奖励标注的强化学习提供了一种更省成本的替代思路。

本文术语

潜在世界模型(latent world model) · 将图像等原始输入压缩成低维向量,并在这个压缩空间中预测未来状态的模型
JEPA · 一种自监督架构,在压缩表征之间进行预测,而不是重建原始像素
CEM/MPPI · 基于采样搜索的规划算法,通过反复采样候选动作序列、模拟并挑选评分最高的方案来逐步优化
SIGReg · 一种正则化方法,使潜在分布接近各向同性的高斯分布,防止表征坍缩
stop-gradient(梯度截断) · 在前向计算中使用某个值,但训练时阻止梯度沿该路径反向传播

论文原文摘要（英文）

Forward latent world models predict how actions change a scene, but recover actions for a desired change only through expensive test-time search. We introduce INTACT (INtent-To-ACTion), an end-to-end JEPA that turns action-labeled, reward-free trajectories into a deployable intent-to-action interface. Each transition supplies physical intent z_{t+1}-z_t, while a future goal supplies deployment intent sg(z_g)-z_t. The architecture is isomorphic between the local and goal motion-intent backbone-in

作者 · Junhan Sun

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Junhan Sun et al., arXiv:2607.26056, arxiv-nonexclusive