AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents
arXiv:2608.004862026-07-31
An AI predicts how an object will move by peeking inside a half-finished video, without ever generating the video
DreamTraj takes just one photo and a text instruction and predicts how an object will move over the next two seconds, in full 3D position and orientation (6-DoF). Instead of running a video-generation AI all the way to a finished video and then analyzing it, it secretly reads the internal signals of that model while it is only partway through denoising. This makes it more accurate than prior methods while running 4.6x faster.
METAL LAB explanatory visual
How DreamTraj reads a trajectory out of a half-finished video
Evidence statusMeasured results reported
InputOne RGB image (I0) and a text instruction (ℓ); the object is located in the first frame only once.
Frozen video-generation modelWan2.2 denoises toward an imagined future but is stopped at step 16 of 40, so the video is never fully generated. Its weights are never updated.
Reading internal signalsAn attention track that follows the object (block 28, head 32) and pooled hidden states carrying scene context (blocks 0/19/39) are combined into a 13x132 signal.
Small Reader networkA 7.1M-parameter flow-matching model converts the signal into 13 relative 9-D position/orientation tokens.
Restoring real-world scaleA depth estimate from the input image and the camera's intrinsics convert the relative values into a metric 6-DoF trajectory.
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
Problem: predicting object motion previously required special inputs like depth sensors, 3D CAD meshes, or multi-view video, or else required generating a full future video and running a costly, error-prone perception pipeline on it.
Method: a frozen (never fine-tuned) video-generation model, Wan2.2, is given one image and an instruction and stopped after only 16 of its 40 denoising steps. From that half-denoised internal state, the system reads an attention signal that implicitly tracks the object plus a signal that captures overall scene context, and a small 7.1-million-parameter network called the Reader converts these into 13 relative position/orientation values.
Data: to fix a lack of fine-grained language-to-motion data, the authors built the Move dataset, 5,038 human-inspected trajectories drawn from six existing hand-object interaction video corpora.
Results: under 5-fold cross-validation over all 5,038 samples, DreamTraj reached 6.4 cm average displacement error (ADE), 9.5 cm final-step error (FDE), and 24.6-degree rotation error, beating prior forecasters (ObjectForesight, EgoScaler) on both translation and rotation even though those baselines receive extra privileged inputs like video context, meshes, or depth.
Efficiency: on the same GPU (an RTX PRO 6000), a generate-then-extract pipeline that fully renders and decodes a video before analyzing it takes about 348 seconds per clip (280s generation plus 68s perception), while DreamTraj finishes in 76 seconds — a 4.6x speedup.
Figure 1: Trajectory extraction pipeline used to label generated videos (Sec. 3.3). Amber marks the two blocks that depart from RigVid and ObjectForesight: DA3 metric depth and a SAM 3D Objects mesh.
Table 1: Main comparison on held-out folds. Block (a) restricts to clips captured by a real sensor, so the reference trajectory is the dataset’s own 6-DoF annotation; block (b) is the full evaluation set, whose labels come from the extraction pipeline of Sec. 3.3. ADE/FDE in cm, rotation in degrees. Baselines marked † consume inputs DreamTraj does not (Sec. 5.1). Each baseline predicts on its own timestamps—EgoScaler a fixed 20-step trajectory, ObjectForesight a horizon fixed by its context requirement—so each is reported against DreamTraj evaluated at that baseline’s timestamps, and blocks are not comparable to one another.
Method
ADE↓
FDE↓
Rot↓
Rot-final↓
EgoScaler protocol — its native 2.0 s window, 20 timestamps
(a) real capture, real GT (n=537)
EgoScaler†
7.52
10.71
28.6
45.6
DreamTraj
3.04
4.71
7.9
12.9
(b) generated video, pipeline GT (n=414)
EgoScaler†
6.55
9.52
31.7
50.0
DreamTraj
3.01
4.72
11.5
18.3
ObjectForesight — its native 1.17 s horizon
(a) real capture, real GT (n=593)
ObjectForesight†
2.71
4.58
7.8
11.1
DreamTraj
1.97
3.32
6.5
10.5
(b) generated video, pipeline GT (n=414)
ObjectForesight†
2.80
4.42
8.6
14.7
DreamTraj
1.97
3.20
7.1
11.6
ObjectForesight — extended to the whole action
(a) real capture, real GT (n=594)
ObjectForesight†
10.29
12.74
29.8
30.2
DreamTraj
6.76
8.39
18.2
20.4
(b) generated video, pipeline GT (n=366)
ObjectForesight†
10.26
13.51
37.3
51.0
DreamTraj
7.26
9.34
33.0
41.4
Figure 2: Overview of DreamTraj. One RGB frame and one instruction go in. The object is grounded once in that frame (GroundingDINO + SAM2) to give query points; a frozen Wan2.2-I2V-A14B is stopped at step 16 of 40, and from that latent we read q⋅k attention tracks (block 28, head 32) and anchor-pooled hidden states (blocks 0/19/39) as a 13×132 sequence. A 7.1M-parameter flow-matching trainable Reader, turns it into 13 relative 9-D pose tokens, which, with a depth anchor z0 and the intrinsics K lift to metric 6-DoF.
Table 2: Inference cost vs. the generate-then-extract route, identical hardware. Guided forwards counts both classifier-free-guidance branches per denoising step. All times are measured on one GPU, averaged over three clips.
Method
Fwd.
Decode
Percep.
Generate
Total↓
Generate-then-extract
80/80
yes
4 models
280 s
348 s
DreamTraj
𝟏𝟖/𝟖𝟎
no
none
𝟕𝟔 s
𝟕𝟔 s
Figure 3: Qualitative comparison. Each row is one clip, labelled with the instruction the model was given (abridged for space). Panels overlay 7 of the 13 predicted poses on the anchor frame, each outlined in green, with the dashed line tracing the object’s centre. Red marks error: a centroid off by more than the object’s own width, or an orientation off by more than 45∘. Methods marked † receive privileged input: ObjectForesight is given the first three poses as ground-truth context, so its overlay starts from the correct pose by construction. EgoScaler predicts over its native 2 s horizon, the other columns over the whole action.
Table 3: Readout step. Track margin is the permutation-controlled score of Eq. 3 between the attention centroid and the recorded 2D object track (955 clips). ADE comes from retraining the Reader at each step on the generated-video path (321 clips); its scale is not comparable to Table 1. Bold marks the adopted step, the earliest at which the margin saturates.
Read step k
12
14
16
18
20
Expert
high-noise
low-noise
Track margin↑
0.123
0.129
0.191
0.196
0.197
ADE (cm)↓
8.05
8.52
7.89
8.20
8.29
Figure 5: Move statistics. (a) source corpus, (b) action verb, (c) object category, each split into recorded and generated trajectories; (d) joint distribution of per-clip translation and rotation magnitude over all 5,038 trajectories.
Table 4: Ablations. ADE/FDE in cm, rotation in degrees, over all 5,038 out-of-fold samples. All four arms are the deployed configuration retrained from the same features, differing only in the conditioning removed. Depth corr. is the median over clips of |r| between the predicted and ground-truth log-depth ratio st=log(zt/z0), each linearly detrended against frame index.
Variant
ADE↓
FDE↓
Rot↓
Depth corr.↑
Control (deployed)
6.42
9.54
24.63
0.640
w/o pooled hidden states
6.91
10.47
26.00
0.565
w/o q⋅k track
7.13
9.90
26.15
0.637
w/o instruction conditioning
6.45
9.70
26.20
0.639
Figure 6: Readout step sweep. (a) permutation-controlled track margin against the recorded 2-D object track; (b) end-task ADE from retraining the Reader at each step. The shaded region is the backbone’s high-noise expert. The signal appears at the expert switch, not gradually.
Table 5: Backbone and readout configuration.
Setting
Value
Backbone
Wan2.2-I2V-A14B (frozen)
Blocks / heads / width
40 / 40 / 5120
Video resolution
832×480
Frames
49 @ 16 fps
VAE stride
(4,8,8); patch (1,2,2)
Latent grid
13×60×104
Token grid
13×30×52=20,280
Sampler
FlowUniPCMultistep
Denoising steps
40
Shift
5.0
Guidance scale
3.5
Seed
0
Readout step
16 of 40 (t≈882)
Flow caching
stride-2 (compute one step, reuse one)
q⋅k cell
block 28, head 32
Pooled-hidden blocks
0, 19, 39
Anchor grid
4×4, anchor-mean pooled
PCA dimension
128 (over the concatenated 3×5120)
Reader input
13 tokens × (128+4) =132 D
Figure 7: Extended qualitative comparison. Each row is one clip, labelled with the instruction the model was given (abridged for space). Panels overlay 7 of the 13 predicted poses on the anchor frame, each outlined in green, with the dashed line tracing the object’s centre. Red marks error: a centroid off by more than the object’s own width, or an orientation off by more than 45∘. Methods marked † receive privileged input: ObjectForesight is given the first three poses as ground-truth context, so its overlay starts from the correct pose by construction. EgoScaler predicts over its native 2 s horizon, the other columns over the whole action.
Table 6: Reader architecture.
Setting
Value
Width d
256
Blocks
4
Attention heads
8
MLP ratio
4.0
Pose tokens
13
Token dimension
9 (2 bearing + 1 log-depth + 6 rotation)
Conditioning
AdaLN-Zero, three gates per block
Context memory
13×256, cross-attention
Extra conditions
instruction 768 D; domain flag 2 D
Positional encoding
fixed sinusoidal (pose and context)
Output head
zero-initialised linear
Parameters
7,078,165 (7.1M)
Inference
20 explicit Euler steps
Figure 8: Generalization beyond the training distribution. Scenes, objects and actions absent from every training corpus, rendered as in Fig. 7. The shampoo bottle panel is a video-game frame; the others are phone photographs. Instructions abridged.
Table 7: Reader optimisation. Identical across every arm of Table 4 in the main paper; the arms differ only in which conditioning signal is removed.
Setting
Value
Optimiser
AdamW
Learning rate
2×10−4 (constant)
Betas
(0.9,0.95)
Weight decay
0.01
Batch size
128
Steps per epoch
29
Deployed snapshot
5,799 steps (epoch 200)
Stage-1 warm-up
40 epochs, domain B only
Stage-2 mixture
60% domain A per batch
Early stopping
none (fixed budget)
Precision
fp32
Flow-matching loss
velocity MSE
Endpoint term
0.1
SO(3) chordal term
0.1
Smoothness term
0.05
Table 8: Realised fold sizes under grouping by source clip. Folds are balanced in recorded samples by construction; the generated counts vary because a clip contributes as many generated samples as it has surviving generations.
Fold
1
2
3
4
5
Recorded (A)
596
597
595
593
594
Generated (B)
366
432
355
445
465
Total
962
1,029
950
1,038
1,059
Table 9: ObjectForesight retraining. Two horizon configurations are trained separately; each is evaluated only against DreamTraj read at its own timestamps.
Setting
Value
Configuration
released HOT3D config
Horizon H / frame stride
8 / 4 (short)
13 / 8 (whole action)
Context length
3 poses
Object library
disabled (mesh supplied per clip)
Depth
cached, anchor frame only
Batch size
32
Learning rate
1×10−4
Split
grouped folds (Sec. B)
Table 10: Move composition by source corpus. Recorded trajectories come from the corpus’s own 6-DoF annotation; generated ones are labelled by the extraction pipeline of Sec. 3.3 and screened by a human.
Corpus
Recorded
Generated
Total
HOI4D
1,325
1,037
2,362
TACO
421
508
929
HOT3D
720
7
727
OakInk2
244
226
470
H2O
173
108
281
FPHA
92
177
269
Total
2,975
2,063
5,038
Table 11: Which middle block to read. Both rows retrain the Reader with only the middle block changed, under the grouped five-fold protocol of Table 4 in the main paper. Requiring every block to be available on every sample leaves 5,036 of the 5,038 samples; the folds are redrawn on that pool, so absolute values differ slightly from Table 4 while the comparison between the two rows is exact.
Pooled-hidden blocks
ADE↓
FDE↓
Rot↓
Depth corr.↑
{0,19,39} (adopted)
6.26
9.46
24.15
0.660
{0,20,39}
6.53
9.92
24.39
0.638
Findings
Under 5-fold cross-validation over all 5,038 samples, DreamTraj achieved 6.4 cm ADE, 9.5 cm FDE, and 24.6-degree rotation error.
It beat EgoScaler on every metric even though EgoScaler receives depth and the object's initial pose, and the lead held even when EgoScaler was given its best-performing input convention out of five tested and evaluated on DreamTraj's own timestamps.
ObjectForesight, which receives video context, an object mesh, and three ground-truth initial poses, trailed DreamTraj on both translation and rotation after being retrained on the Move dataset.
On identical hardware, the generate-then-extract cascade took about 348 seconds per clip (280s generation + 68s perception) versus DreamTraj's 76 seconds, a 4.6x speedup.
On 50 out-of-distribution scenes (40 real phone photos, 10 video-game frames) not present in any training corpus, ten human annotators judged that the predicted trajectory correctly carried out the instruction 80% of the time.
Where it can be used
Planning robot grasps and placements before performing a manipulation
Serving as a training signal for imitating human demonstrations at the object level rather than pixel level
Verifying in advance whether an intended action would achieve its goal
Attempting motion prediction from ordinary phone photos in settings without depth sensors or 3D models
Limits and open work
Many ground-truth trajectories used for evaluation, even when human-screened, come from the paper's own extraction pipeline rather than a real sensor, so quality may differ from directly captured trajectories.
The 50 out-of-distribution scenes had no ground truth, so only human judgment (80% success) was reported, not a numeric error score.
For rotationally symmetric objects like bottles and cups, rotation about the symmetry axis is unobservable from RGB alone and was excluded from supervision and evaluation.
Driving an actual robot arm using the predicted trajectories has not yet been tested; the paper names it as the next step.
Results depend on the object being correctly identified in the first frame by a grounding step, and performance when that grounding fails was not evaluated.
Why it matters
Being able to anticipate how an object will move lets a robot plan grasps and placements in advance, but until now this required expensive or hard-to-get inputs like depth sensors or 3D models. This work shows that a single ordinary photo and a sentence can achieve the same goal faster, suggesting a cheaper, more practical path toward planning robot manipulation.
Terms in this paper
6-DoF (six degrees of freedom) · the six numbers needed to fully describe an object's position (x, y, z) and orientation (three rotation axes)
image-to-video (I2V) diffusion model · an AI model that turns one still image into a video by gradually removing noise over many steps
query-key (q·k) attention · an internal value the model computes to relate different parts of its input; here it is repurposed to secretly track where the object is
flow matching · a way of training a generative model to learn a smooth path from random noise to the target output
ADE/FDE · ADE is the average position error across the whole predicted trajectory; FDE is the error only at the final predicted point
Figures we cannot republish
Figure 4: Generalization beyond the training distribution. Scenes, objects and actions absent from every training corpus, rendered as in Fig. 3. Panel (a) is a video-game frame; (b–d) are phone photographs. Instructions abridged.
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajec
