INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models

arXiv:2607.260562026-07-27

A world model that turns a desired outcome directly into an action, without expensive trial-and-error search

Latent world models can predict how a scene changes after an action, but finding which action produces a desired change has usually required costly test-time search like CEM. INTACT trains one predictor on both the real next-state change and a goal-state change using the same input format, so at deployment a goal alone yields an action with no search. Trained jointly across four visual tasks (PushT, Cube, Reacher, TwoRoom) with one shared encoder, this search-free 'Direct' control matched or beat search-based baselines while running far faster.

METAL LAB explanatory visual

How INTACT is trained and used for control

Evidence statusMeasured results reported

Shared encoderOne encoder compresses images from all four visual domains (PushT, Cube, Reacher, TwoRoom) into a latent representation z.
Local intent inputThe real change to the next state, z(t+1)-z(t), is fed into the INTACT Predictor along with the demonstrated action, anchoring physical reachability.
Goal intent inputThe change to a goal state, sg(z_goal)-z(t), is fed into the same predictor with gradients stopped on the goal side, forming the deployment-time action interface.
Forward PredictorA separate module keeps predicting future latents with SIGReg regularization, preserving contacts and obstacles beyond what's needed for immediate action.
Direct control / optional verificationAt inference, a goal alone produces an action chunk with no search (Direct mode); a small local CEM (Guarded A) can optionally verify or replan.

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

Problem: standard latent world models only predict the next scene given an action; recovering the action for a desired change requires test-time search (CEM/MPPI) that samples many candidate actions, rolls them through the model, and keeps the best.
Core idea: feed the same shared predictor two kinds of 'intent'—the real next-state change (z_{t+1}-z_t, local intent) and the goal-state change (sg(z_g)-z_t, goal intent)—both supervised by the demonstrated action, with the goal branch's gradient stopped so it doesn't disturb local learning.
Method: one shared visual encoder plus task-specific small predictor heads are trained across four domains (PushT, Cube, Reacher, TwoRoom); a separate Forward Predictor keeps predicting future states with SIGReg regularization to preserve world information beyond immediate action.
Result: search-free Direct control took about 2.9-5.5 ms per plan versus a measured 1.48 s for CEM 300x30 (roughly 300x faster), and adding a light local verification step (Guarded A, 128x3 candidates) reached 96.86% macro success rate and 92.22% worst-task success rate across four tasks, 16 points above matched pure CEM 300x30.
Additional check: even after removing the action head and running CEM only on the representation (actor-disabled CEM), performance still improved over the baseline world model, indicating the representation itself, not just the policy head, became more usable.

Figure 1: Shared-encoder INTACT training and recurrent control. Four visual domains use one encoder and task-specific predictor pairs. Within each task, graph-isomorphic local and goal calls enter the same INTACT Predictor through one matched input grammar, using attached local intent zt+1k−ztk or detached goal intent sg⁡(zgk)−ztk. The Forward Predictor retains latent-dynamics and SIGReg supervision. At inference, the goal-conditioned call emits one action chunk for search-free Direct control; the world rollout is retained only for optional verification and replanning.

Table 1: Positioning among closely related methods. “Action→E” means that action supervision updates the visual/world encoder, and “Shared-4T E” means one encoder is jointly trained over all four official LeWM domains. The final column states each method’s actual training-to-control path; a cross denotes an unreported interface, not an impossible extension.
Method	Action→E	Shared-4T E	Condition	Action obj.	Direct	Search	Training-to-control path
DINO-WM [40]	✗	✗	action	none	✗	✓	Frozen image encoder; CEM ranks latent rollouts
C-JEPA [22]	✗	✗	action/proprio.	none	✗	✓	Frozen object features; CEM searches masked rollouts
LeWM [21]	✗	✗	action	none	✗	✓	Joint JEPA prediction; CEM inverts the forward model
Fast-LeWM [9]	✗	✗	action prefix	none	✗	✓	LeWM training; parallel action prefixes are searched
SMWM [16]	✓	✗	successor	local action	✗	✓	Local inverse shapes the encoder; deployment still uses CEM
GC-IDM [23]	✗	✗	goal + horizon	next action	✓	✗	Frozen LeWM; a separate goal/horizon actor acts directly
PRISM [36]	✗	✗	state/goal	Gaussian prior	✗	✓	Frozen LeWM; a separately trained prior initializes MPPI
Qantara [28]	✓	✗	dispatch-dependent	action flow	✓	✓	Joint JEPA; bridge-flow heads serve dispatch modes
QuoVLA [35]	✓	✗	visual/language	action chunk	✓	✗	VLM action quotient; quantized intent drives a VLA actor
INTACT	✓	✓	local/goal intent	action block	✓	opt.	One shared JEPA/actor law; Direct by default, search verifies

Figure 2: INTACT converts mandatory search into optional verification. LeWM samples raw actions from an uninformed Gaussian and uses forward rollouts to become goal-directed. INTACT alternates the raw goal displacement, the shared conditional action mean, and the unchanged Forward Predictor to produce a Direct plan. It can execute immediately or locally verify that plan with K=128, I=3, and σ0=0.25.

Table 4: Theory-linked evidence for shared four-task INTACT. Correlations use official Direct SR; the actor-disabled and gauge rows are matched interventions that test representation and correspondence directly.
Evidence	Measured result	Interpretation
Pred.–expert kNN (45 eligible E1–E5 checkpoints)	Pooled 𝒓=.954 [.928,.969]; adjusted 𝒓=.902. Within Waypoint/Goal: .968/.981.	Local action-family neighborhoods track deployable control.
Pred.–expert linear CKA (same 45 checkpoints)	Pooled 𝒓=.897 [.837,.930]. Within Waypoint/Goal: .979/.986. Leave-one-epoch-out: [.888,.914].	Global family geometry remains predictive across interfaces and epochs.
Pointwise action R2 (same 45 checkpoints)	r=.815.	Recovering one expert action is weaker than preserving the family relation.
Effective-rank inversion (controlled E5 cells)	Rank 93.87>89.26, while SR 74.22<89.39.	Latent spread is a capacity check, not a semantic certificate.
Actor-disabled planning (matched E5 checkpoints)	Pure-CEM macro 66.17→70.08.	Action losses improve the Encoder–Forward stack before actor execution.
Paired gauge intervention (21,600 CLEAR Moderate episodes)	Shuffled 9.46%→68.04% paired (+58.58 pp).	Correct task-local correspondence, rather than coordinate identity, restores control.

Figure 3: Single-task action likelihoods shape the representation. Actor-disabled CEM evaluates the encoder–Forward stack under a fixed optimizer; episode-disjoint probes measure readable state and transition action. Physical inverse supplies attached successor shaping, whereas the goal branch updates current-state coordinates and the shared actor with a stop-gradient goal anchor. Error bars are sample standard deviations over three PushT training seeds.

Table A7: Controlled shared-encoder representation snapshots. σ¯ is mean per-dimension latent standard deviation, cosine is mean pairwise cosine, and action R2 is an episode-disjoint frozen probe.
Training cell	n	Rank	σ¯	Cos.	Act. R2
LeWM	3	25.53	.830	.554	.309
Inverse only	3	24.03	.902	.590	.372
Waypoint intent only	3	24.83	.846	.582	.328
Goal intent only	3	24.45	.839	.604	.349
Waypoint INTACT	3	25.73	.877	.587	.383
Goal INTACT	3	25.34	.888	.616	.382

Figure 4: Controlled E5 success across four tasks. Bars visualize the native rows of Table 3: LeWM uses CEM 300×30, learned heads use Direct, and the final bar uses Guarded A 128×3. Values are means ± sample standard deviations over three training seeds. The ordinate starts at 30% to expose high-SR differences and is not a zero-based effect size.

Table A10: Task-centered Pearson correlation with E5 Direct SR. These are diagnostic associations, not independent causal effects.
Metric	Moderate	Strict
Effective rank	0.888	0.846
Gsep	0.819	0.731
Deployment NLL	−0.640	−0.766

Figure 5: Intent–action relation, rather than task clustering, tracks control. Columns separate waypoint and goal-displacement interfaces; rows compare task purity, pointwise action R2, predicted–expert CKA, and local kNN overlap. Each point is one checkpoint and each annotation is a pooled checkpoint-level Pearson correlation with official four-task Direct SR. Historical runs are lower opacity and never rank the controlled objectives.

Table A13: PushT Direct SR (%) on the held-out 10% episode split. Each cell contains 100 paired test-only episodes.
Training objective	eval 0	eval 1	eval 42	pooled
Transition inverse	35.67	44.33	40.67	40.22
+ deployment goal intent	75.33	81.33	76.00	77.56

Figure A1: Controls used to select the intent grammar. (a) Coordinate and interaction interventions. (b) Re-encoding every 1, 2, or 5 action blocks. (c) Detaching the goal anchor and explicitly reducing chord error separate action-field learning from Euclidean latent straightening. Error bars are training-seed standard deviations.

Table A14: PushT successes on held-out expert episodes (100 per cell).
	Transition inverse	Goal intent
Training seed	e0	e1	e42	e0	e1	e42
3072	35	42	41	74	83	74
3073	36	44	41	81	81	79
3074	36	47	40	71	80	75

Figure A2: Verifier-selection controls. The complete three-seed audit compares Direct, guarded and unguarded plan-centered search, trust penalties, actor particles, and actor-disabled CEM.

Table A15: Historical automatic-success audit and CLEAR Moderate calibration. Historical random is the mean of three runs under the released stack. CLEAR values use canonical seeds 0/1/42 and 100 episodes per seed.
Task	Initially solved	Historical random	Moderate random
PushT	0.22	4.00	4.00
Cube	38.38	49.00	15.67
Reacher	0.57	13.00	4.33
TwoRoom	8.82	26.00	6.67

Figure A3: Exploratory E5 collapse morphologies from fixed samples. These legacy non-Math-SDPA runs are qualitative diagnostics and do not enter performance ranking or headline correlations.

Table A16: CLEAR-LeWM v0.5.1 task contracts. “Hold” is the number of consecutive successful environment steps. The 24-fold Cube comparison minimizes orientation error over the proper rotational symmetry group of a cube.
Task	Moderate: minimal compatibility correction	Strict: task-semantic precision
PushT	Released pusher + T position <20 px and T angle <20∘; first hit.	T object only, <10 px and <10∘; hold 3.
Cube	Initially unsolved, cube-center distance ≤4 cm; first hit.	Cube center ≤3 cm and symmetry-aware orientation ≤15∘; hold 3.
Reacher	Periodic unbounded shoulder, raw bounded wrist, joint error <0.05 rad; first hit.	Physical fingertip endpoint ≤1 cm; hold 2.
TwoRoom	Clean cross-room pair, continuous swept-disk collision, endpoint <16 px.	Legal doorway crossing, valid route, goal-side arrival, endpoint <8 px.

Figure A4: Search-free control and optional verification. Left: physical-successor inversion versus goal-displacement INTACT after one epoch. Right: Direct uses no candidate sequence, Guarded A locally verifies 384, and broad actor-disabled CEM evaluates 9,000. More search is not automatically better once the action conditional is aligned.

Findings

On single-task PushT, actor-disabled CEM 30x10 success rate rose from 42.22% (baseline LeWM) to 57.67% with inverse-dynamics supervision, 61.44% with matched goal-intent supervision, and higher still for INTACT (waypoint variant, SIGReg 0.03).
Search-free Direct control took 2.9-5.5 ms per plan (matched PushT: 4.8 ms), about a 300x reduction versus a measured actor-initialized CEM 300x30 average of 1.48 seconds.
Across all 12 final checkpoints (576-job audit), Guarded A (Direct plan plus local 128x3 CEM verification at sigma0=0.25) reached 96.86% macro and 92.22% worst-task success rate, gaining 16.00 macro points over matched pure CEM 300x30 while cutting sampled candidate sequences from 9,000 to 384 (23.44x fewer).
A single shared-encoder Goal-displacement INTACT trained across all four tasks reached macro Direct success rate of ±0.77%, improving over the matched shared-encoder LeWM baseline by 5.66/32.23/12.56/42.44 points across the four tasks, and exceeding published task-specific LeWM macro and Cube/Reacher scores.
Across 45 checkpoints, predicted-versus-expert action-family kNN overlap and CKA correlated with official success rate at Pearson r=0.954 and r=0.897 respectively, higher than pointwise action R2 at r=0.815.

Figure A5: Gauge controls isolate correspondence from actor functionality. Correctly paired calibration restores control; recovery saturates near 64/full episodes; same-objective swaps define a ceiling; reverse swaps show that a coordinate map cannot create an untrained deployment conditional. The 72-cell audit contains 21,600 CLEAR Moderate episodes and excludes all evaluation episodes from calibration.

Where it can be used

Goal-directed robotic manipulation tasks (e.g., pushing objects as in PushT) where fast reaction time makes exhaustive search impractical
Building goal-conditioned action policies purely from reward-free offline demonstration data
Adding a lightweight verification step (Guarded) on top of existing CEM/MPPI planners to balance speed and reliability

Limits and open work

Reported results are limited to four simulated tasks (PushT, Cube, Reacher, TwoRoom); generalization to real robots or other domains is not verified.
Direct control's speed comparison is a planner-side measurement from this paper, not an end-to-end latency benchmark for full vision-language-action systems.
Under a separate corrected benchmark (CLEAR-LeWM), Reacher success rate drops sharply from 97.67% (official) to 49.56% (Moderate), suggesting the original benchmark may have had a ceiling effect.
Strict-mode evaluation has not been completed for INTACT; only Moderate results are reported.
Some auxiliary experiments (e.g., flow-matching latent predictors, independent dual-actor comparisons) are exploratory diagnostics and not part of the headline performance ranking.

Why it matters

Search-based planning is a major computational bottleneck for real-time control like robotic manipulation, and this work shows search can become optional verification rather than a mandatory step, which matters for latency-sensitive systems. It also demonstrates that reward-free, offline demonstration data alone can be reorganized into a deployable goal-conditioned controller, offering a practical alternative to reward-labeled reinforcement learning.

Terms in this paper

latent world model · a model that compresses raw input like images into a lower-dimensional vector and predicts future states within that compressed space
JEPA · a self-supervised architecture that predicts between compressed representations rather than reconstructing raw pixels
CEM/MPPI · search-based planning algorithms that sample many candidate action sequences, simulate them, and iteratively refine toward the best-scoring one
SIGReg · a regularization technique that keeps the latent distribution close to an isotropic Gaussian to prevent representation collapse
stop-gradient · a technique that uses a value in a forward computation but blocks gradients from flowing back through that path during training

Original abstract (English)

Forward latent world models predict how actions change a scene, but recover actions for a desired change only through expensive test-time search. We introduce INTACT (INtent-To-ACTion), an end-to-end JEPA that turns action-labeled, reward-free trajectories into a deployable intent-to-action interface. Each transition supplies physical intent z_{t+1}-z_t, while a future goal supplies deployment intent sg(z_g)-z_t. The architecture is isomorphic between the local and goal motion-intent backbone-in

Authors · Junhan Sun

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Junhan Sun et al., arXiv:2607.26056, arxiv-nonexclusive