FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

arXiv:2608.010492026-08-01

让AI在人车混杂、拥挤混乱的“全球南方”城市场景中,把未来预测拆成布局、行人车辆、互动关系三条通道分别处理

研究者在印度22个城市拍摄了约1000小时视频,建立了名为DENSEWORLD的拥挤城市数据集,发现现有的JEPA世界模型(一种预测未来场景的AI架构)在这种高密度、强遮挡环境下表现不佳。为此他们提出FactorJEPA,不再用单一的“大杂烩”表示去预测未来,而是把未来拆成布局、带可见度门控的行人车辆、稀疏互动关系三条通道,再重新组合成最终预测。无论是在1万条片段的对照实验还是11.5万条片段的全量训练中,FactorJEPA在未来预测准确度、对场景干预的反应一致性、以及对遮挡的鲁棒性上都超过了最强的对比方法。

METAL LAB 解读图

FactorJEPA如何拆分并重新组合对未来的预测

证据状态已报告实测结果

DENSEWORLD原始视频来自印度22个城市的约1000小时行车、步行、航拍视频,涵盖市场、交叉口、高架路等高密度城市场景
DINOv2自动拆分冻结的预训练视觉模型自动提取区域、行人车辆轨迹、可见度和距离信息,生成布局、个体、互动的训练目标
三条预测通道分别预测缓慢变化的布局、带可见度权重的行人车辆、以及邻近主体之间稀疏的互动关系
重新组合的未来表示三条通道的结果重新组合成一个统一的未来JEPA嵌入,并通过分离损失约束不同通道之间不互相泄露信息
四项指标评测分别测量未来嵌入预测准确度、对干预的反应一致性、对遮挡的鲁棒性、以及线性可读取的运动信息,并与其他适配方法对比

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

研究团队在印度22个一二线城市拍摄了约1000小时的行车记录、步行拍摄和航拍视频,涵盖市场、居民区、交通枢纽、高架路、海滩等场景,建成DENSEWORLD 1.0数据集。
与常用自动驾驶基准BDD100K和nuScenes按相同场景类型对比,DENSEWORLD中每个画面里的行人车辆数量和画面占用面积明显更高,在市场、商业街、高架/地下通道场景中差距最大。
把现有的V-JEPA世界模型直接冻结使用时,动作预测准确率只停留在37.5%到44.4%之间(多数类基线19.5%),即便用LoRA、DoRA、Auto-RGN等参数高效微调方法,也只能提升到50.3%到53.2%。
FactorJEPA借助一个冻结的DINOv2(预训练图像理解模型)自动提取布局、个体、互动的监督目标,再用三条独立通路分别预测缓慢变化的布局、带可见度门控的行人车辆、以及稀疏的两两互动关系,最终重新组合成对未来的预测。
在1万条片段的对照实验中,FactorJEPA在未来预测误差和干预反应一致性上领先,但在线性可读取的运动信息上不如常规微调;而在11.5万条片段全量训练后,FactorJEPA在全部四项指标上都反超对比方法,遮挡鲁棒性提升43.3倍、未来预测误差提升33.2倍、运动信息提升20.0倍、干预反应一致性提升13.9倍(均为置信区间单位,而非直接的性能倍数)。

Figure 1: Same question, two encoders. Each row is a three-frame filmstrip of one held-out clip posed as a multiple-choice question, answered from an identical probe head over the frozen V-JEPA 2.1 encoder vs. ours (factor-view predictor surgery); only the backbone differs. Top: motion speed (ours 69.8% vs frozen 60.9%). Bottom: turn direction.

Table 1: Frozen encoders collapse. Ten frozen encoders on the DENSEWORLD motion probe (ntest=1,825, ±95% BCa CI). Action top-1 (A) stays in a 37.5–44.4% band (19.5% majority), while adapted encoders reach 50.3–53.2%. M = motion-cos, T = taxonomy F1 (↑ better). In each column the best value is shaded green and the worst red.
Frozen encoder	A (%)	M	T
V-JEPA 2.1 (2B)	44.4	0.009	0.793
V-JEPA 2.1 ViT-L	44.2	0.004	0.788
V-JEPA 1 ViT-H	40.5	0.007	0.702
LeJEPA ViT-L	40.1	0.014	0.740
V-JEPA 1 ViT-L	39.9	0.008	0.660
I-JEPA ViT-H	39.1	0.016	0.781
V-JEPA 2.0 (SSv2)	38.8	0.007	0.776
DINOv2	38.5	0.016	0.816
V-JEPA 2 ViT-L	37.9	0.013	0.778
I-JEPA ViT-G/16	37.5	0.019	0.787

Figure 2: DENSEWORLD 1.0 scene-type coverage. Representative examples from the dataset illustrate the breadth of urban environments covered by DENSEWORLD 1.0, including market, residential, commercial, promenade, transit, highway, heritage, junction, flyover, and beach scenes. This diversity reflects the spatial, social, and infrastructural heterogeneity of populous, crowded, and chaotic urban environments.

Table 2: Protocol-matched V-JEPA adaptation baselines. All methods retain the JEPA objective and train on the same raw clips; only the adaptation mechanism changes.
Method	Adaptation mechanism	Question tested
LoRA (5)	Adds low-rank updates, ΔW=(α/r)BA, to selected frozen projections.	Is parameter-efficient low-rank adaptation sufficient?
DoRA (8)	Separates weight magnitude from direction and applies low-rank updates to the directional component.	Does weight decomposition recover structure missed by LoRA?
Auto-RGN (7)	Selects transformer blocks using their relative gradient norms and updates only the selected subset.	Can gradient-guided selection localize the required adaptation?

Figure 3: DENSEWORLD exhibits higher multi-agent density than standard driving benchmarks. We compare agent count density and agent occupancy across matched scene types. DENSEWORLD shows large gaps in market, commercial, and flyover/underpass scenes, indicating stronger interaction pressure and heavier visual competition.

Table 3: Composition, information flow, and independence constraints of the DENSEWORLD partitions. Allocation is performed over source-video groups. Durations are approximate because grouped recordings vary in length and usable-shot yield. DINOv2-based test measurements are reported as teacher-relative; headline prediction, motion, and RGB metrics use fixed evaluators that do not provide FactorJEPA’s factor-supervision targets.
Partition	Share	Duration	Gradient updates	Model selection	Threshold calibration	Signal exposed	Isolation constraint
Training	80%	≈800 h	Yes	No	No	JEPA prediction targets and DINOv2-derived layout, agent, and interaction targets.	Source-video disjoint from validation and test; all descendants of a source remain grouped.
Validation	10%	≈100 h	No	Yes	Yes	Prespecified validation metrics and DINOv2-derived targets used only for selection and calibration.	Source-video disjoint from training and test; never used for gradient optimization.
Test	10%	≈100 h	No	No	No	Frozen headline evaluators and DINOv2-derived targets used only for explicitly teacher-relative diagnostics.	Source-video disjoint from training and validation; all configurations are frozen before test access.
Total	𝟏𝟎𝟎%	≈1,000 h	–	–	–	–	–

Figure 4: DENSEWORLD 1.0 scene-type coverage. Twelve representative scene types (market, temple, commercial, transit, residential, promenade, ghat, heritage, highway, junction, flyover, beach) across 22 Indian cities, shown from ground-level pedestrian viewpoints.

Table 4: Scene-matched agent density and occupancy. Count density is measured in agents per sampled frame; occupancy is the percentage of valid image area covered by the union of visible agent masks. Δ=DW−reference, and R is the ratio of the two means. DENSEWORLD results are aggregated over K executed folds. The final row gives the unweighted mean across the six matched scene strata.
	Agent count density (agents/frame)	Agent occupancy (% valid area)
Matched scene stratum	Matched reference	DENSEWORLD [K-fold]	𝚫	𝑹	Matched reference	DENSEWORLD [K-fold]	𝚫 pp	𝑹
Residential lane	1.8	3.3	+1.5	1.83×	2.0%	3.5%	+1.5	1.75×
Promenade	1.4	2.5	+1.1	1.79×	1.5%	2.8%	+1.3	1.87×
Market	4.6	12.4	+7.8	2.70×	4.2%	11.1%	+6.9	2.64×
Heritage / tourist	1.7	3.0	+1.3	1.76×	0.9%	1.4%	+0.5	1.56×
Flyover / underpass	2.6	5.0	+2.4	1.92×	8.8%	20.8%	+12.0	2.36×
Commercial	4.1	11.0	+6.9	2.68×	3.9%	9.0%	+5.1	2.31×
Equal-stratum mean	2.70	6.20	+3.50	2.30×	3.55%	8.10%	+4.55	2.28×

Figure 5: Factorized decomposition of the same urban scene. (a) Agents are isolated from scene context. (b) Layout highlights persistent spatial structure while retaining suppressed agent silhouettes. (c) Interactions show tracklets, residual motion, and sparse pairwise coupling; arrow direction and length encode motion direction and magnitude, while solid and dashed edges denote stronger and weaker interactions.

Table 5: Frozen teacher-to-training contract for FactorJEPA. DENSEWORLD contains no manually annotated factor labels. All targets are generated automatically by the fixed DINOv2-based pipeline before optimization. The table identifies, for each factor, the teacher evidence, deterministic target operator, output support, reliability contract, and trainable consumer. Here, pL,pA,pI are the executed target dimensions, NnA is the number of retained agent tracklets, and ℰn is the candidate interaction graph.
Factor	Frozen teacher-side target contract	Training side
Target	Teacher evidence	Executed operator	Output support	Reliability contract	Consumer
Layout Tn,L	DINOv2 spatial-feature grid, valid-image support, dynamic-region union, and temporal feature persistence.	ΨL suppresses pixels assigned to retained dynamic tracklets, aggregates the remaining temporally persistent support, and applies the fixed layout-target projection.	Tn,L∈ℝpL clip-level	ωn,L∈[0,1] combines valid background coverage, feature confidence, and temporal stability. The target is retained only when rn,L=𝟏[ωn,L≥τL].	PL(CL) via ℒfactor
Agent Tn,A	Retained masks, boxes, region descriptors, confidences, and temporally associated tracklets {τn(i)}i=1NnA.	ΨA constructs an object-centric state for each valid tracklet and aggregates the retained states after confidence, visible-support, and temporal-continuity filtering.	Tn,A∈ℝpA clip-level	ωn,A∈[0,1] combines retained-region confidence, visible mask support, track continuity, and valid temporal coverage. Missing or rejected tracks are not converted into negative agents.	PA(CA) via ℒfactor
Visibility Tn,V	Visible-mask support, valid-image-boundary truncation, region confidence, track continuity, and short interior track gaps.	ΨV produces a soft visibility value for every retained agent. Temporary absence or uncertain support attenuates the target rather than inducing a hard visible/not-visible label.	{Tn,V(i)}i=1NnA Tn,V(i)∈[0,1]	Each agent receives ωn,V(i)∈[0,1], determined by region confidence, temporal support, and track consistency. Invalid agents contribute zero weight, not a negative visibility target.	gV via weighted ℒV
Interaction Tn,I	Pairs of retained tracklets, relative position, relative motion, visible support, temporal overlap, and pairwise proximity.	ΨI constructs ℰn⊆{(i,j):i≠j}, forms normalized pair descriptors, and aggregates valid pair states into the interaction target.	ℰn,Tn,I∈ℝpI graph + clip target	Pair reliability ωn,I(ij)∈[0,1] combines the two endpoint confidences, joint temporal support, and pair validity. These weights induce the clip-level reliability ωn,I.	PI(CI) via ℒfactor; gI,gW use ℰn
Shared reliability contract. All reliability variables are stop-gradient quantities in [0,1]. For k∈{L,A,I}, the effective loss weight is rn,kωn,k, where rn,k=𝟏[ωn,k≥τk]. Factor losses are normalized by the total effective weight within the minibatch, so variations in retained coverage do not directly rescale their contribution. Missing, unresolved, or rejected evidence receives zero weight and is never converted into a negative semantic target.
Gradient and provenance contract. The DINOv2 encoder, structural operators ΨL,ΨA,ΨV,ΨI, tracklets, candidate graph, targets, and reliability weights remain frozen. Gradients propagate only through the FactorJEPA branches, factor heads, synthesis dictionaries, and the selected online-encoder blocks. Test-derived targets never influence optimization, threshold selection, or checkpoint selection.

Figure 6: Factorized prediction. FactorJEPA composes the target embedding from layout, agents, and interactions, with a soft visibility gate applied only to entity terms.

Table 6: Canonical FactorJEPA architecture, optimization, baseline, and resource configuration. The table specifies a complete reproducible reference configuration for the V-JEPA 2.1 ViT-G and ViT-g experiments. Backbone identifiers and architectural scales follow the official V-JEPA 2.1 release. All project-specific optimization, factorization, adaptation, and profiling settings are fixed below. Resource entries are prespecified execution ceilings; they must not be described as measured consumption unless verified against profiling logs.
Configuration item	V-JEPA 2.1 ViT-G	V-JEPA 2.1 ViT-g	Control, interpretation, or measurement rule
Panel A: Backbone, teacher, and video preprocessing
Backbone identifier	vjepa2_1_vit_gigantic_384	vjepa2_1_vit_giant_384	Official V-JEPA 2.1 PyTorch-Hub model identifiers. Both checkpoints remain the unique initialization source for all compared methods.
Backbone scale	Approximately 2.0B encoder parameters.	Approximately 1.0B encoder parameters.	Parameter counts exclude the momentum copy when reporting trainable parameters because the momentum encoder receives no gradient.
Encoder architecture	Embedding dimension d=1664; 48 transformer blocks; 16 attention heads; MLP ratio 4.	Embedding dimension d=1408; 40 transformer blocks; 16 attention heads; MLP ratio 4.	Both models use pre-norm transformer blocks, patch size 16, and rotary positional encoding.
Video tokenizer	Patch size 16×16; tubelet size 2.	Patch size 16×16; tubelet size 2.	No backbone-specific spatial or temporal interpolation is introduced during the controlled comparison.
Model input	384×384 RGB; 16 sampled frames.	384×384 RGB; 16 sampled frames.	All clips undergo the same resize, center-preserving crop, privacy filter, temporal sampling, and normalization.
Temporal sampling	4 fps; 4.0-s sampled window.	4 fps; 4.0-s sampled window.	A window is sampled entirely inside one shot. No sample crosses a shot or source-video boundary.
Context and target	Frames 1:12 form context; frames 13:16 form the future target; horizon 1.0 s.	Frames 1:12 form context; frames 13:16 form the future target; horizon 1.0 s.	Context duration is 3.0 s. Temporal indices are identical across all adaptation methods within each replication.
Pixel normalization	ImageNet mean (0.485,0.456,0.406) and standard deviation (0.229,0.224,0.225).	Same.	Normalization is applied after privacy filtering and spatial resampling.
DINOv2 checkpoint	dinov2_vitg14_reg; patch size 14; descriptor dimension 1536; final normalized patch-token representation.	The teacher is frozen and evaluated in BF16 without stochastic augmentation.
Teacher input	518×518 RGB; ImageNet normalization; the same 16 temporal indices used by the JEPA branch.	Teacher preprocessing is deterministic and partition independent.
Target-cache precision	DINOv2 descriptors and factor targets stored in FP16; reliabilities stored in FP32.	Targets are computed once before training. No gradient enters DINOv2, the structural operators, cached targets, or reliability weights.
Region retention	Minimum confidence 0.55; minimum area 0.1% of valid image support; duplicate suppression at mask IoU 0.70.	Thresholds are fixed on training/validation data and reused without modification on test data.
Temporal association	Hungarian matching with 0.45 box-IoU cost, 0.35 mask-IoU cost, and 0.20 descriptor-cosine cost; maximum gap 4 frames; minimum track length 3 frames.	Association is restricted to a single source video and shot. Unmatched regions are retained only after satisfying the track initiation criterion.
Reliability range	ωn,k∈[0.10,1.00], computed from region confidence, temporal support, track continuity, and factor-specific validity.	Reliability weights are stop-gradient quantities and are normalized within factor before minibatch aggregation.
Panel B: Masking and predictor architecture
Small-block masks	8 blocks per clip; spatial scale 0.15; aspect-ratio range [0.75,1.50]; temporal scale 1.0.	The same mask realization is reused across methods for a given replication and sampled clip.
Large-block masks	2 blocks per clip; spatial scale 0.70; aspect-ratio range [0.75,1.50]; temporal scale 1.0.	Small- and large-block policies are jointly applied. Complement masks are not forced.
Predictor backbone	24 transformer layers; width 384; 12 heads; MLP ratio 4; RoPE; learned mask tokens; no predictor registers.	The monolithic and factorized variants use the same predictor depth, width, attention count, positional encoding, and target-query support.
Factor target dimensions	pL=256, pA=256, pI=256; one scalar visibility target per retained agent.	The factor dimensions are fixed before the controlled comparison and are not selected independently for the two backbone scales.
Factor ranks	Layout rank rL=64; agent rank rA=96; interaction rank rI=64.	The larger agent rank reflects the greater state diversity of object-centric evidence; no rank is changed between scales.
Visibility head	Two-layer MLP: 384→256→1; GELU; sigmoid output.	Same.	Visibility targets and predictions lie in [0,1]. Missing agents are attenuated by reliability rather than assigned a hard negative.
Interaction head	Two-layer pair MLP: 768→256→1; GELU; sigmoid strength.	Same.	Candidate pairs are formed within normalized image-plane radius 0.25. At most 12 nearest valid neighbors are retained per agent.
Trainable encoder depth	Top KG=2 online-encoder blocks.	Top Kg=1 online-encoder block.	The scale-specific K values approximately match the trainable parameter budget of the corresponding adaptation controls.
Frozen components	Bottom 46 online blocks; complete momentum encoder; DINOv2 teacher; structural operators; cached targets.	Bottom 39 online blocks; complete momentum encoder; DINOv2 teacher; structural operators; cached targets.	The momentum encoder is updated only by EMA.
Panel C: Optimization, regularization, and model selection
Optimizer	AdamW, β1=0.9, β2=0.95, ϵ=10−8.	Same.	Optimizer state is retained only for trainable parameters.
Learning rates	Factorized predictor and heads: 1.0×10−4; top encoder blocks: 1.0×10−5.	Same.	Encoder learning rate is 0.1× the predictor learning rate. No layer-wise decay is applied within the selected top-K blocks.
Weight decay	0.04.	0.04.	Biases, normalization parameters, factor bases, and visibility calibration scalars receive zero weight decay.
Schedule	Linear warm-up for 1,000 steps, followed by cosine decay to 1.0×10−6.	Same.	Schedule is indexed by optimizer updates rather than processed clips.

Figure 7: Depth-wise factor realization. Lightweight probes show a staged profile: layout emerges early, agents grow gradually, and interactions peak in deeper layers.

Table 7: Executed attribution contract. FactorJEPA-RAW and FactorJEPA share the factorized predictor, separation objective, interaction-sparsity objective, trainable encoder scope, and parameter budget. They differ in access to the DINOv2-derived factor and visibility targets. Auto-RGN is the primary parameter-matched monolithic reference; full fine-tuning is a non-matched capacity ceiling.
Method	Predictor	Factor targets	Executed objective	Trainable scope	Attribution role
Frozen V-JEPA	Monolithic	None	No optimization	None	No-adaptation reference.
V-JEPA Full-FT	Monolithic	None	ℒJEPA	Complete online encoder and predictor	Non-matched capacity ceiling.
V-JEPA LoRA	Monolithic	None	ℒJEPA	Matched low-rank projection updates	Parameter-efficient adaptation control.
V-JEPA DoRA	Monolithic	None	ℒJEPA	Matched decomposed low-rank updates	Parameter-efficient adaptation control.
V-JEPA Auto-RGN	Monolithic	None	ℒJEPA	Monolithic predictor and selected top-K blocks	Primary matched monolithic reference.
FactorJEPA-RAW	Factorized	None	ℒJEPA+λsepℒsep+λsparseℒsparse	Factorized predictor and selected top-K blocks	Factorized structural package without teacher targets.
FactorJEPA	Factorized	TL,TA,TV,TI	ℒJEPA+λsupℒfactor+λVℒV+λsepℒsep+λsparseℒsparse	Factorized predictor and selected top-K blocks	Complete model: structural package plus structured supervision.

Figure 8: FactorJEPA vs. the strongest competitor across scale and data regimes. Bars report FactorJEPA’s advantage in units of the paired-difference 95% confidence interval; the dashed line marks statistical separation at 1×. Under matched stratified 10k protocol, FactorJEPA separates on Future-frame L1 and Causal L1 at both scales, and on Mask-ratio slope at 1B, while trailing on Motion cosine (revealing a consistent prediction–motion trade-off). With full 115k-clip training at 1B, it separates strongly on all four primary diagnostics, reaching 43.3× for Mask-ratio slope, 33.2× for Future-frame L1, 20.0× for Motion cosine, and 13.9× for Causal L1.

Table 8: Evaluation metric registry. The table records the operational definition, preferred direction, evaluation unit, frozen signal or evaluator, and inferential status of every reported diagnostic. “Intervention L1” corresponds to the quantity previously labeled Causal L1; the revised name reflects that the metric measures intervention-response consistency rather than causal identification.
Metric	Operational definition	Dir.	Unit	Frozen signal or evaluator	Status
Primary predictive diagnostics
Future-frame MSE	Mean squared error between predicted and target future-token embeddings.	↓	Clip–horizon	Momentum target encoder	Primary
Intervention L1	Normalized L1 discrepancy between predicted and target latent changes under the same automatically generated intervention.	↓	Clip–intervention	Intervention generator and target encoder	Primary
Mask-ratio slope	OLS slope of Future-frame MSE as the visible context is progressively reduced.	↓	Clip–mask curve	Target encoder and fixed mask sampler	Primary
Motion cosine	Cosine similarity between linearly decoded and target motion descriptors.	↑	Clip	Motion estimator and fixed-capacity probe	Primary
Semantic diagnostics
Action top-1	Top-1 accuracy of a fixed-capacity action probe.	↑	Clip	Frozen action labels and probe protocol	Secondary
Taxonomy F1	F1 over the shared dynamic-agent taxonomy.	↑	Clip–class	Frozen taxonomy evaluator	Secondary
Prediction-stability diagnostics
Rollout-drift slope	Increase in prediction error over autoregressive rollout depth.	↓	Clip–rollout	Target encoder	Secondary
L1-vs-Δt decay	Increase in latent error over the evaluated future horizons.	↓	Clip–horizon	Target encoder	Secondary
Exposure-bias gap	Difference between free-running and teacher-conditioned prediction error.	↓	Clip	Target encoder	Secondary
Temporal diagnostics
Frame-order sensitivity	Prediction-error increase after controlled frame-order corruption.	↑	Clip	Fixed temporal permutation	Secondary
Arrow-of-Time	Accuracy for distinguishing forward from reversed clips.	↑	Clip	Fixed temporal classifier	Secondary
Temporal-order accuracy	Accuracy for detecting frame permutations.	↑	Clip	Fixed temporal-order classifier	Secondary
Playback-pace accuracy	Accuracy for identifying the applied temporal-rate transformation.	↑	Clip	Fixed pace classifier	Secondary
TCC cycle-back error	Temporal distance between a source frame and its cycle-consistent match.	↓	Frame pair	Frozen correspondence features	Secondary
TCC Kendall τ	Rank agreement between predicted and true temporal correspondences.	↑	Clip pair	Frozen correspondence features	Secondary

Figure 9: Evaluation scorecard across 2B and 1B scales. Frozen, conventionally adapted, parameter-efficient, and factorized V-JEPA 2.1 variants are compared across predictive, motion, semantic, and temporal diagnostics. Bars report performance with 95% BCa intervals where available.

Table 9: Latent-to-RGB decoder contract. One scale-specific decoder is trained for each JEPA embedding width. Within a backbone scale, every method uses exactly the same transport operator, Cosmos component, preprocessing, and evaluation settings. Fields marked from log must be copied verbatim from the executed decoder configuration.
Component	Executed configuration	Comparison control
JEPA latent source	ViT-G: dJ=1664; ViT-g: dJ=1408; future frames 13:16 from the common 384×384, 16-frame protocol.	Target and predicted latents use the same temporal indices, normalization, token ordering, and spatial support.
Transport operator	Layer normalization, input projection, spatial reshaping, spatiotemporal transport stack, output projection, and positional encoding. Exact depth, width, heads, and dC: [from log].	One transport operator per backbone scale; no method-specific projection or normalization.
Cosmos initialization	Checkpoint identifier and revision: [from log]. Checkpoint hash: [from log].	The same initialization is used for all methods evaluated at a fixed backbone scale.
Cosmos update scope	Trainable modules: [from log]. Frozen modules: [from log].	The update scope is fixed before decoder training and is not selected separately for individual prediction methods.
RGB output	Frames, resolution, color range, and frame rate: [from log].	All outputs undergo the same clipping, resizing, and inverse normalization before evaluation.
Frozen upstream models	Online encoder, momentum target encoder, monolithic or factorized predictor, DINOv2 teacher, and factor-target operators.	RGB reconstruction gradients never enter a world model or pseudo-target generator.
Frozen RGB evaluators	LPIPS network, optical-flow estimator, and agent detector. Exact checkpoints and thresholds: [from log].	No RGB evaluator supplies FactorJEPA supervision or participates in decoder optimization.

Figure 10: Causal future-block rankings replicate across model scales. Each point represents one adaptation method, comparing its causal future-block L1 score with the ViT-G 2B backbone on the horizontal axis and the ViT-g 1B backbone on the vertical axis; lower values are better. FactorJEPA variants are shown in green, and the dashed line denotes equal scores across scales. The strong Spearman correlation (ρ=0.979) shows that the relative method ordering is highly preserved when scaling from 2B to 1B.

Table 10: Latent-to-RGB evaluation under a common frozen decoder. Entries should be reported as held-out mean [95%clusteredCI]. Values in parentheses denote the direction-normalized deficit relative to target-latent oracle decoding; positive deficits indicate worse performance than the oracle. No predicted-latent source is used to train or select the decoder.
Latent source	PSNR ↑	SSIM ↑	LPIPS ↓	Flow EPE ↓	Agent F1 ↑
Target latent (oracle)	[result]	[result]	[result]	[result]	[result]
V-JEPA Auto-RGN	[result]	[result]	[result]	[result]	[result]
FactorJEPA-RAW	[result]	[result]	[result]	[result]	[result]
FactorJEPA	[result]	[result]	[result]	[result]	[result]

Figure 11: Staged factor-curriculum surgery on the full corpus. Training JEPA loss for FactorJEPA predictor surgery on the full DENSEWORLD corpus (∼115k clips; ViT-g 1B backbone, batch 32, learning rate 5×10−5). Surgery proceeds in four shaded phases (dashed boundaries): a head-only warmup, then progressive unfreezing of the top encoder blocks while the predictive emphasis cycles through layout, agents, and interactions. The loss falls from ≈0.41 to ≈0.38; each transition briefly raises the loss as the target distribution shifts, after which training re-descends, with the interaction stage settling at a slightly higher floor consistent with its harder prediction target. The learning-rate schedule is dashed (right axis).

Table 11: Operational taxonomy and attribution rules for decoded-future failures. Let 𝖷=x+ denote the observed future, 𝖮=x~⋆ the target-latent oracle reconstruction, and 𝖯=x^M a decoded prediction. Panel A uses the matched triplet (𝖷,𝖮,𝖯) to localize error; Panel B defines observable failure modes and corroborating diagnostics. Attribution is diagnostic rather than causal because an off-manifold predicted latent may expose nonlinear decoder sensitivities.
Class	Operational signature	Oracle comparison	Interpretation or diagnostic
Panel A: Oracle-based attribution rules
Oracle-present	The same artifact is visible in both 𝖮 and 𝖯 relative to 𝖷.	𝖮 already fails to reconstruct the relevant content.	Primarily a decoder-capacity or target-latent-invertibility limitation; it must not be attributed solely to forecasting.
Prediction-conditioned	𝖮 preserves the relevant content, whereas 𝖯 does not.	The artifact appears only after replacing the target latent with Y^M.	Localized to the prediction-conditioned pathway: inaccurate predicted latent, distribution shift at the decoder input, or their interaction.
Mixed or amplified	The artifact is present in 𝖮 but becomes materially stronger in 𝖯.	Both oracle reconstruction error and oracle-to-prediction deviation are non-negligible.	Decoder and forecasting effects coexist; report oracle error, end-to-end error, and visible forecast gap separately.
Panel B: Observable decoded-future failure modes
Layout drift	Static boundaries, façades, road geometry, or background support move despite being stable in 𝖷.	Check whether the same displacement occurs in 𝖮.	Static-region error, boundary displacement, and spurious background flow distinguish geometric drift from local texture variation.
Agent omission	An agent visible in 𝖷 is absent, severely attenuated, or merged into the background in 𝖯.	Determine whether the corresponding agent is recoverable in 𝖮.	Agent recall, Agent F1, and dynamic-region error quantify the failure; small or heavily occluded agents should be reported separately.
Agent duplication	One reference agent produces multiple overlapping instances or spatially inconsistent fragments in 𝖯.	If duplication is also present in 𝖮, classify it as oracle-present.	Agent precision, duplicate-detection rate, and connected-component fragmentation provide corroborating evidence.
Interaction inconsistency	Nearby agents exhibit incorrect relative ordering, separation, direction, or collision geometry.	Compare pairwise geometry in 𝖷, 𝖮, and 𝖯.	Pairwise displacement, relative-motion error, and interaction-edge agreement distinguish interaction failure from independent agent misplacement.
Motion under-dispersion	Moving agents become blurred, nearly stationary, or displaced toward an average future.	An accurate 𝖮 with reduced motion only in 𝖯 indicates prediction-conditioned averaging.	Flow EPE, predicted-to-reference flow-magnitude ratio, and dynamic-region sharpness quantify the effect. It may reflect deterministic prediction under multimodal futures.
Appearance substitution	Coarse location and occupancy remain plausible, but local appearance or category-specific structure changes.	Determine whether the substitution is already visible in 𝖮.	Agent-crop LPIPS and frozen-descriptor similarity separate appearance loss from geometric or occupancy failure.
Unsupported content	𝖯 introduces an agent, boundary, texture, or motion pattern unsupported by 𝖷.	Oracle presence suggests decoder hallucination; prediction-only presence indicates an off-manifold latent or prediction–decoder interaction.	False-positive agent rate, static-region residuals, perceptual error, and oracle comparison jointly support classification.

Figure 12: Factorized predictive channels. Context is split into layout, visibility-gated agent, and sparse interaction coordinates, then recomposed in the future JEPA embedding, limiting cross-factor leakage.

研究结果

把V-JEPA冻结直接使用时,在DENSEWORLD的动作预测探测任务上准确率停留在37.5%到44.4%的区间(多数类基线为19.5%,测试片段1825条,95%置信区间),而LoRA、DoRA、Auto-RGN等参数高效微调方法也只能提升到50.3%到53.2%。
在按场景类型对齐的比较中,DENSEWORLD相比BDD100K和nuScenes始终表现出更高的行人车辆密度和画面占用率,市场、商业街、高架/地下通道场景中的差距最大。
在1万条片段的对照实验中,FactorJEPA在未来嵌入预测误差上(2B模型6.3倍、1B模型4.8倍,置信区间单位)和干预反应一致性上(分别2.3倍、2.7倍)领先最强对比方法,遮挡鲁棒性在2B模型上未达到统计显著分离(0.9倍)但在1B模型上分离(1.9倍),而在线性可读取的运动信息上,常规微调反而更优(-14.5倍/-9.2倍)。
在1B模型上用全部11.5万条片段训练后,FactorJEPA在全部四项核心指标上都与最强对比方法拉开差距:遮挡鲁棒性提升43.3倍、未来预测误差提升33.2倍、运动信息提升20.0倍、干预反应一致性提升13.9倍,这些数值均以配对置信区间为单位,而非直接的性能倍数。
比较2B模型和1B模型在四项关键指标上各方法的排名,斯皮尔曼相关系数达到0.895到0.979,说明不同适配方法之间的相对优劣排序在模型规模变化时基本保持一致。

可应用场景

该数据集可作为在车道秩序较弱、人车混行常见的发展中国家城市场景中训练或评测自动驾驶、机器人导航世界模型的基准。
把未来预测拆分为布局、个体、互动三条通道的思路,可为其他需要在密集多主体、部分遮挡场景下保持稳健的世界模型设计提供参考。
论文中用于将预测的潜在表示解码为可视化RGB图像、并用“预言家”对照分离预测误差与解码器局限的评测流程,可供其他需要目视检查预测表示质量的研究参考。

局限与待验证事项

互动关系的监督信号来自冻结的DINOv2自动提取的区域、运动轨迹和距离信息,目前还不能充分捕捉预判、让行、群体性移动等随时间延展的社会协商行为。
布局、个体、互动等目标由固定的自动流水线生成,而非人工逐一标注,在严重遮挡、模糊或光线不佳时可能不准确,且该分解方式并不保证是唯一或完全独立的划分方式。
实验集中在预录制视频的短时预测上,更长时段的滚动预测、基于动作条件的预测,以及闭环规划或在线适应尚未得到验证。
数据集仅覆盖印度22个城市,代表的是拥挤混乱“全球南方”城市环境的一种大规模实现,并不能直接推广到非洲、拉丁美洲、东南亚等其他地区,也不涉及室内、农村或灾难场景。
11.5万条片段的全量训练只在1B规模的模型上完成,2B规模模型仅在1万条片段的对照实验中评测,更大规模模型在全量数据下是否保持同样表现尚待验证。

为什么重要

这项研究首次用数据证明,在车道整齐、密度较低场景上训练评测的世界模型,面对人车混行、规则松散的“全球南方”城市场景时会明显失灵。这对想把自动驾驶或城市仿真系统落地到发展中国家城市的研究者和企业来说,提供了现有基准为何不够、以及一种具体架构改进方向的实证依据。

本文术语

JEPA(联合嵌入预测架构) · 一种不直接重建像素,而是在压缩后的表示空间里预测未来的AI模型结构
DINOv2 · 一个可以直接拿来用、无需额外训练即可提取图像区域和特征的预训练视觉模型
可见度门控 · 一种机制,对被遮挡、不确定的目标不是直接丢弃,而是柔性降低其权重
Causal L1 · 衡量对场景做一次受控编辑后,预测的未来表示与真实目标表示变化是否一致的指标;论文后来改称Intervention L1,以强调它衡量的是对干预的敏感度,而非因果证明
Mask-ratio slope(遮挡比例斜率) · 随着遮住画面比例增加,预测误差增长的速度;数值越小说明模型对信息缺失越稳健

论文原文摘要（英文）

World models have attracted significant attention for their ability to capture and predict the structure and dynamics of the physical world. In this emerging landscape, Joint Embedding Predictive Architectures (JEPA) offer a particularly compelling direction. We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments, which we call DENSEWORLD. Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft

作者 · Kapil Wanaskar

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Kapil Wanaskar et al., arXiv:2608.01049, CC BY 4.0