Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

What they did

1. The team collected about 1,940 hours of first-person human hand manipulation videos from four sources (ANT, EgoDex, ViTRA, EgoVerse), then built a pipeline with three stages: action alignment (converting 21 hand keypoints into robot end-effector trajectories), visual alignment (removing human arms via SAM 3 segmentation and inpainting, then compositing a rendered robot arm using inverse kinematics), and multi-level quality curation to filter bad frames and trajectories.
2. They ran this pipeline separately for 15 different robot arm morphologies (Panda, UR5e, xArm7, and 12 others), turning each source video into 15 parallel training streams and yielding a total of 18,561 hours of synthesized robot data, described as the largest ego-to-robot dataset so far.
3. To measure generalization precisely, the authors extended the RoboTwin2.0 benchmark so that four factors -- visual appearance (background, lighting, robot color), scene layout (table height, clutter, camera offset), embodiment (swapping in different robot arms), and task semantics (unseen objects, paraphrased instructions) -- could each be tested independently instead of all mixed together.
4. Compared with pretraining on real robot data alone (about 6,565 hours from DROID, AgibotWorld, and InternData), mixing the synthesized data with real data at a 1:1 ratio raised success rate on RoboTwin's randomized setting to 53.5% (+2.6 points), and produced gains across background (+4), lighting (+8), robot color (+6), camera offset (+6), unseen objects (+11 at a 3:1 ratio), and paraphrased instructions (reaching 69% at 1:1).
5. On a real ARX ACone robot across five tasks (putting fruit in a basket, placing blocks in a drawer, folding a towel, sweeping trash, inserting a screw), the model trained with both teleoperated demonstrations and the synthesized ego-derived data outperformed the robot-only model on every task, with the biggest gains on placing blocks (+14 points) and inserting a screw (+13 points).

Findings

• Mixing Ego2R-synthesized data with real robot data at a 1:1 ratio reached 53.5% success on RoboTwin's randomized setting, 2.6 points above robot-only pretraining, while still keeping 68.1% on the clean setting.
• At the 1:1 mixing ratio, per-factor visual improvements were background +4, lighting +8, robot color +6, and camera offset +6 compared to robot-only pretraining.
• Unseen-object generalization improved from 29% to 40% (+11 at a 3:1 ratio), and success on paraphrased instructions reached 69% at the 1:1 ratio.
• In cross-embodiment tests swapping in different robot arms, ARX improved from 44% to 51% and UR5 peaked at 31% (3:1 ratio), while Franka Panda stayed below 7%.
• On the real ARX ACone robot across five tasks, the model trained with both teleoperated demonstrations and the pipeline-converted egocentric data outperformed the robot-only model on every task, with the largest gains on placing blocks and inserting a screw.

Where it can be used

• Teams that lack large amounts of real demonstration data for a new robot arm or new task could consider converting existing human manipulation footage into supplementary pretraining data.
• Service robot settings where backgrounds, lighting, objects, and phrasing change frequently could consider using diverse human video as an additional training source to improve robustness.
• Projects supporting multiple robot arm models at once could reference this approach of rendering a single human video into several robot embodiments in parallel.

Limits and open work

• Hand motions are converted only into simple parallel-jaw gripper movements, so the approach does not directly extend to dexterous multi-fingered manipulation.
• Visual synthesis relies on inpainting and depth-aware compositing, which can introduce artifacts under heavy occlusion or difficult lighting conditions.
• Evaluation is confined to the task scope of RoboTwin2.0, so it remains untested whether the benefits hold for a broader range of tasks and robot configurations.
• For robots with a large kinematic gap from the training embodiments, such as Franka Panda, cross-embodiment success stayed below 7%, showing the gains are not uniform across all robot types.

Why it matters

Collecting real robot demonstrations is expensive and slow, so being able to automatically convert the enormous existing supply of first-person human manipulation videos into usable robot training data could substantially cut data-collection costs. The fact that this specifically improves robustness to unseen backgrounds, robot types, objects, and phrasing matters directly for deploying robots in real, unpredictable environments.

Terms in this paper

• Egocentric video · Video recorded from a person's own point of view, typically with a head- or body-mounted camera, showing their hands manipulating objects
• VLA model (vision-language-action model) · An AI model that takes camera images and language instructions as input and directly outputs the robot actions to perform
• Inverse kinematics (IK) · A method for computing the joint angles a robot arm needs so its end-effector reaches a desired position and orientation
• Out-of-distribution (OOD) generalization · The ability to keep performing well under backgrounds, objects, robot types, or phrasing that were not seen during training
• End-effector (EEF) · The part at the tip of a robot arm, such as a gripper, that actually makes contact with objects

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctlyarXiv:2608.194082026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3