Figure 1. Given a casually captured monocular video, typically with mild camera motion and unknown camera intrinsics and poses, 4DAnyone generates multiview-consistent human videos, enabling 4DGS reconstruction rendered from free viewpoints. See the project page and Fig. 7 for more results.One continuous 3D scene: a smartphone at the lower left films a bass-playing subject, a ring of twenty-four generated camera views surrounds the stage, and the reconstructed subject stands at the center on a turntable, rendered from a free viewpoint.
Table 1. Training dataset statistics.
Dataset
Videos
Cameras
Actors
Resolution
Type
MVGameHuman
38k
24
318
2560×1440
Multi-view
SynCamVideo
34k
10
66
1280×1280
Multi-view
DNA-Rendering
51k
48
548
2048×2048
Multi-view
TedTalk
42k
1
413
2160×1620
Monocular
Pexels
20k
1
1,411
3840×2160
Monocular
Figure 2. Overview of 4DAnyone. Given a source video, an HMR model (GVHMR (33)) estimates a 3D skeleton sequence, which is rendered into depth-buffered skeleton videos for v target views. A 3D-aware skeleton encoder produces residual skeleton tokens that are added to the noisy target latents. The source tokens, RCP reference tokens, and skeleton-conditioned target tokens are concatenated along the view dimension and processed by the DiT with video attention, multiview attention, and text cross-attention. Generated target videos are packed back into the RCP context for subsequent generation rounds, while TCR improves consistency during grouped target-view generation. The final multi-view videos are used to train a 4DGS model with FreeTimeGS (39).Pipeline diagram. A source video is converted into skeleton-conditioned target-view tokens. Reference Context Packing supplies compact reference tokens, Target Context Routing exchanges information among target-view groups, and the generated multiview videos are reconstructed as a 4D Gaussian Splatting model.
Table 2. Quantitative comparison on DNA-Rendering and DyMVHumans. We evaluate 4DGS reconstruction, generated video consistency, and generated video reconstruction. Best results are in bold. † denotes our fine-tuned version.
Method
DNA-Rendering
DyMVHumans
PSNR↑
SSIM↑
LPIPS↓
PSNR↑
SSIM↑
LPIPS↓
Gen. Video Consistency
MV-Performer (59)
21.25
0.799
0.222
19.98
0.797
0.182
TrajectoryCrafter (52)
13.56
0.641
0.331
15.19
0.769
0.204
ReCamMaster† (2)
21.47
0.806
0.210
21.94
0.833
0.142
Ours
24.33
0.862
0.163
24.48
0.862
0.109
4DGS Reconstruction
MV-Performer (59)
20.38
0.826
0.191
18.69
0.787
0.158
TrajectoryCrafter (52)
14.81
0.718
0.331
15.11
0.748
0.217
ReCamMaster† (2)
20.55
0.807
0.214
19.86
0.795
0.159
Ours
24.15
0.863
0.159
23.28
0.846
0.117
Gen. Video Reconstruction
MV-Performer (59)
19.33
0.803
0.204
14.36
0.731
0.230
TrajectoryCrafter (52)
13.68
0.698
0.358
14.11
0.725
0.247
ReCamMaster† (2)
20.74
0.809
0.204
19.18
0.778
0.168
Ours
23.69
0.850
0.165
21.03
0.808
0.143
Figure 3. Progressive inference with Reference Context Packing and Target Context Routing. We first generate reference videos and pack them with the source video into a fixed RCP context. During target-view generation, this fixed RCP context is shared by all groups. TCR partitions v target views into m four-view groups, cyclically regroups them at high noise levels to propagate global structure, and fixes adjacent groups at low noise levels for stable detail refinement.Three-stage inference diagram. Initial views become a fixed multiscale reference context; target views are cyclically regrouped at high noise to share global structure and arranged into fixed adjacent groups at low noise to refine local details.
Table 3. Ablation study. Following the Gen. Video Consistency setting, we evaluate the consistency among generated videos.
Configuration
PSNR↑
SSIM↑
LPIPS↓
w/o TCR & RCP
21.09
0.766
0.216
w/o RCP
22.03
0.780
0.203
w/o TCR
22.21
0.788
0.196
Full (Random)
22.20
0.788
0.197
Full (Strided)
22.06
0.786
0.198
Full (Sliding)
22.63
0.796
0.191
Figure 4. Qualitative comparison with baselines. We show target-view generated videos (Gen.) and their corresponding 4DGS renderings (Rend.) across diverse human-centric videos. 4DAnyone produces geometrically accurate and visually detailed results across viewpoints, while the baselines suffer from inaccurate camera control (our fine-tuned ReCamMaster†) or geometric distortions (MV-Performer). See the project page for dynamic results.A grid compares generated target views and reconstructed 4D renderings from several methods on diverse human videos. The proposed method preserves body geometry, clothing appearance, and camera viewpoint more consistently than ReCamMaster and MV-Performer.
Table 4. Training data sampling. Source cameras are uniformly sampled from the listed options.
Dataset
No. Tgt Cam
No. Src Cam
No. Frame
Weight
MVGameHuman
6
1 / 4 / 8
41
0.4
MVGameHuman
4
1 / 4 / 8
61
0.4
MVGameHuman
1
1 / 4 / 8
121
0.2
SynCamVideo
4
1 / 4
61
0.8
SynCamVideo
1
1 / 4 / 8
81
0.2
DNA-Rendering
6
1 / 4 / 8
41
0.4
DNA-Rendering
4
1 / 4 / 8
61
0.4
DNA-Rendering
1
1 / 4 / 8
121
0.2
Pexels
1
1
121
1.0
TedTalk
1
1
121
1.0
Figure 5. Qualitative ablation results. Left: 3D-aware skeleton conditioning resolves the inherent ambiguity of 2D skeletons, guiding the model to generate geometrically correct content. Right: RCP and TCR maintain an effective cross-view context, leading to consistent appearance across generated views.Side-by-side ablation grids compare outputs with and without 3D-aware skeleton conditioning, Reference Context Packing, and Target Context Routing. The complete model has more accurate body geometry and more consistent appearance across views.
Table 5. Stage-specific training settings. “Indep. Src Prob” denotes the probability of independently sampling the source frame range. Skeleton: B=body, H=hands, F=feet, Fi=fingers.
Stage
Dataset
Bg. Removal
Indep. Src Prob
Skeleton
1
DNA-Rendering
✓
0.2
B+H+F+Fi
2
+ MVGameHuman, SynCam.
–
0.0
B+H+F+Fi
3
+ Pexels, TedTalk
–
0.0
B+H+F
Figure 6. Challenging cases. 4DAnyone robustly handles back-view source videos (left), complex subject appearance (right), and complex human motions.Two challenging examples show source frames and generated viewpoints for a person initially seen from the back and a person with complex clothing and appearance.
Table 6. Multi-GPU inference configurations. Configurations are shown for different camera setups.
No. Layers
No. Cam / Layer
No. GPUs
No. Tgt Cam / GPU
1
16
4
4
2
16
8
4
3
16
8
6
Figure 7. Robust generalization to diverse in-the-wild human videos. For each example, we show the source video (left), generated target-view videos (middle, 4 of 16 views), and a 4DGS novel-view rendering (right). See the project page for dynamic results.Sixteen in-the-wild examples are arranged in two columns. Each example shows a source human video frame, four generated target viewpoints, and a novel-view rendering of the reconstructed 4D Gaussian avatar.
Table 7. TCR switching-time sweep. Gen. Video Consistency when varying the number of sliding denoising steps.
ts/T
Sliding steps
PSNR↑
SSIM↑
LPIPS↓
1.00
0
22.2079
0.7880
0.1964
0.75
5
22.3030
0.7898
0.1947
0.50
10
22.4575
0.7925
0.1933
0.25
15
22.6093
0.7955
0.1912
0.20
16
22.6294
0.7963
0.1906
0.15
17
22.6023
0.7961
0.1908
0.10
18
22.6217
0.7967
0.1905
0.05
19
22.6367
0.7972
0.1903
0.00
20
22.6414
0.7971
0.1903
Figure 8. Representative MVGameHuman samples. Each row shows one frame from four evenly spaced cameras in a synchronized 24-camera sequence captured with our in-house game data engine. MVGameHuman provides diverse actors, clothing, motions, lighting conditions, virtual scenes, and backgrounds for training multi-view human video generation.Eight rows of synthetic human scenes, each viewed simultaneously from four evenly spaced virtual cameras. The samples vary in actor, clothing, pose, lighting, environment, and background.
Figure 9. Single image to 4D avatar. Given a single input image, we first generate a source video via pose-driven video generation (Wan-Animate), then produce multi-view target videos with 4DAnyone, and finally reconstruct a 4DGS avatar via FreeTimeGS.A left-to-right pipeline turns one portrait into a pose-driven source video, generates synchronized target-view videos with 4DAnyone, and reconstructs an animatable 4D Gaussian avatar.
Figure 10. Failure cases. Left: skeleton guidance is uninformative for the large flowing fabric, which is generated inconsistently across views and yields a degraded reconstruction (red box). Right: HMR mis-estimates the en-pointe pose (green circle in the source view) as flat feet, and all generated views inherit the pose error (red box).Two failure examples. Red boxes highlight inconsistent large flowing fabric in one reconstruction and incorrect feet in generated views caused by a source-pose estimation error marked with a green circle.
论文原文摘要（英文）
We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as O(N), weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with O(1) reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.
