ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine
一套设备同时记录一个人做家务时的第一人称视角、全身动作、手部动作、声音和触觉，用来做机器人训练数据
ACE把真实的家变成录制棚,在同一时间轴和空间坐标下,同步记录一个人做饭或整理房间时的第一人称视频、多视角第三人称视频、全身和手部动作、物体六自由度轨迹、声音和触觉压力。由此收集到的ACE-Data-0数据集包含150小时录像、200种任务、50名参与者、1700万帧和7.5万个交互片段。研究团队据此建立了三层基准测试,评估了30多种现有方法在触觉预测、身体/手部姿态恢复、手物交互估计上的表现。
METAL LAB 解读图
ACE数据采集与结构
证据状态实测结果与计划中的工作并存
- 两套互补采集配置桌面级(8台近距相机+16台动捕相机)用于手部操作,房间级(8台广基线相机+12台动捕相机)用于全身移动,两者并行运作
- 多传感器同步录制第一人称头戴视频、多视角第三人称视频、全身及手部动捕、物体六自由度轨迹、音频、触觉手套压力数据全部对齐到同一时间轴和空间坐标系
- 自动标注生成根据追踪到的真实物理状态自动生成相机标定、姿态投影、物体网格与轨迹、事件文字描述
- 三层基准评测依次评测触觉信号预测(信号层)、身体/手部姿态恢复(组件层)、第一/第三人称手物交互估计(交互层),覆盖30余种方法
他们做了什么
- 现有数据集存在割裂问题:Ego4D、EPIC-Kitchens等大规模第一人称数据集缺少身体和物体的真实动作标注,而BEHAVE、GRAB、ARCTIC等动捕数据集又缺少第一人称视角、声音或触觉信号。
- 为此团队搭建了两套互补装置:桌面级配置(8台近距离相机加16台动捕相机)用于精细手部操作,房间级配置(8台广基线相机加12台动捕相机)用于覆盖整间公寓的全身移动。
- 参与者穿戴动捕服、触觉手套,并佩戴带有四个鱼眼镜头的头戴设备,所有传感器数据流通过硬件或软件方式同步,并统一注册到同一空间坐标系。
- 最终得到的ACE-Data-0数据集附带自动生成的丰富标注,包括相机标定、投射到每个摄像机画面上的身体和手部姿态、每个物体的三维网格与六自由度姿态,以及事件文字描述。
- 研究团队据此建立了三层基准:触觉信号预测、人体/手部姿态估计、以及从第一人称和第三人称视频中估计手物交互动作,并评估了30多种现有先进方法。
| Dataset | Year | Hours | #Frames | #Subj | #Obj | #Tasks | Ego | #Exo | Body | Hand | Obj. 6D | Tactile | Audio | Sync | Setup | LH |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Egocentric video | ||||||||||||||||
| Ego4D [33] | 2022 | 3670 | – | 931 | – | Open | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | – | In-the-wild | ✓ |
| EPIC-KITCHENS-100 [19] | 2021 | 100 | 20M | 37 | – | Open | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | – | Kitchens | ✓ |
| HoloAssist [95] | 2023 | 166 | – | 222 | 16 | 20 | ✓ | ✗ | ✗ | (✓) | ✗ | ✗ | ✓ | – | Desktop | ✓ |
| Ego-Exo4D [34] | 2024 | 1286 | – | 740 | – | 8 Dom. | ✓ | 4–5 | (✓) | ✗ | ✗ | ✗ | ✓ | ✓ | In-the-wild | ✓ |
| EgoLife [100] | 2025 | 266 | – | 6 | – | Open | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | Shared house | ✓ |
| HD-EPIC [73] | 2025 | 41 | 4.46M | 9 | – | 69 | ✓ | ✗ | ✗ | ✗ | (✓) | ✗ | ✓ | – | Home kitchens | ✓ |
| Hand–object interaction | ||||||||||||||||
| ContactPose [7] | 2020 | – | 2.9M | 50 | 25 | 2 | ✗ | 3 | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | Table-top | ✗ |
| HO-3D [36] | 2020 | – | 78K | 10 | 10 | – | ✗ | 1–5 | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | Table-top | ✗ |
| GRAB [88] | 2020 | – | 1.6M | 10 | 51 | 4 | ✗ | ✗ | ✓ | ✓ | ✓ | (✓) | ✗ | – | Mocap lab | ✗ |
| DexYCB [15] | 2021 | – | 582K | 10 | 20 | 1 | ✗ | 8 | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | Table-top | ✗ |
| H2O [52] | 2021 | – | 571K | 4 | 8 | 36 | ✓ | 4 | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | Table-top | ✗ |
| OakInk [101] | 2022 | – | 230K | 12 | 100 | 5 | ✗ | 4 | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | Table-top | ✗ |
| HOI4D [61] | 2022 | 22 | 2.4M | 9 | 800 | 54 | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | – | Indoor rooms | ✗ |
| ARCTIC [22] | 2023 | – | 2.1M | 10 | 11 | 2 | ✓ | 8 | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | Mocap lab | ✗ |
| TACO [60] | 2024 | – | 5.2M | 14 | 196 | 151 | ✓ | 12 | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | Table-top | ✗ |
| HOT3D [3] | 2024 | 13.9 | 3.7M | 19 | 33 | Open | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | – | Lab rooms | ✗ |
| OakInk2 [110] | 2024 | – | 4.0M | 9 | 75 | 150 | ✓ | 3 | (✓) | ✓ | ✓ | ✗ | ✗ | ✓ | Table-top | (✓) |
| GigaHands [26] | 2025 | 34 | 183M | 56 | 417 | Open | ✗ | 51 | ✗ | (✓) | (✓) | ✗ | ✗ | ✗ | Table-top | ✗ |
| Full-body HOI, human-scene interaction, and daily motion | ||||||||||||||||
| BEHAVE [5] | 2022 | – | 15K | 8 | 20 | – | ✗ | 4 | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | Lab rooms | ✗ |
| InterCap [41] | 2022 | – | 67K | 10 | 10 | – | ✗ | 6 | ✓ | (✓) | ✓ | ✗ | ✗ | ✓ | Lab room | ✗ |
| CHAIRS [44] | 2022 | 17.3 | – | 46 | 81 | 32 | ✗ | 4 | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | Lab room | – |
| EgoBody [113] | 2022 | – | 220K | 36 | – | 5 Cat. | ✓ | 3–5 | (✓) | (✓) | ✗ | ✗ | ✗ | ✓ | Indoor rooms | ✗ |
| Aria Digital Twin [69] | 2023 | 6.6 | – | – | 398 | Open | ✓ | ✗ | (✓) | ✗ | ✓ | ✗ | ✓ | – | Apartment, office | ✗ |
| OMOMO [54] | 2023 | 10 | – | 17 | 15 | – | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | – | Lab room | ✗ |
| HIMO [65] | 2024 | – | 4.1M | 34 | 53 | – | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | – | Mocap lab | ✗ |
| TRUMANS [45] | 2024 | 15 | 1.6M | 7 | 20 | – | ✗ | ✗ | ✓ | ✗ | (✓) | ✗ | ✗ | – | Scene mockups | ✗ |
| ParaHome [50] | 2024 | 8.1 | – | 38 | 22 | – | ✗ | 70 | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | Home room | ✓ |
| Device | Role | Qty | Resolution / rate | Notes |
|---|---|---|---|---|
| OptiTrack PrimeX 22 | Optical motion capture | 28 | 2048×1088, 60 Hz | IR tracking: 41 body markers, objects, ego rig |
| ZED One | Exocentric RGB capture | 8 | 1920×1080, 30 FPS | GMSL2 to a single Jetson Orin host; shared frame trigger |
| GoPro | Exocentric RGB capture | 8 | 1920×1080, 30 FPS | Rigidly mounted on stands; audio-triggered recording |
| ACE-Ego-Head-V02 Lite | Egocentric capture | 4 cameras | 4×1088×1280, 20 FPS | One headset: front/back fisheye pairs, IMU, 5 markers |
| Manus | Hand pose | 2 gloves | 60 Hz | Per-finger articulation, both hands |
| ACE-Sense-Glove Lite | Contact pressure | 2 gloves | – | Full-palm pressure map, both hands |
| Jetson Orin | Recording host | 1 | – | Ingests all ZED One streams; NatNet-coordinated |
| Motive host (PC) | Mocap host, sync reference | 2 | – | Renders the optical clock for cross-system sync |
| Method | Temp Acc. ↑ | C-IoU ↑ | V-IoU ↑ | CoP ↓ |
|---|---|---|---|---|
| PressureVision [32] | 0.0093 | 0.0007 | 0.0000 | 10.9807 |
| EgoPressureDiff [109] | 0.2912 | 0.0197 | 0.0025 | 8.5152 |
| TouchAnything [115] | 0.7095 | 0.1646 | 0.1357 | 6.5846 |
| Method | PA-MPJPE ↓ | PA-PVE ↓ | MPJPE ↓ | PVE ↓ | WA-MPJPE ↓ |
|---|---|---|---|---|---|
| — Single-view exocentric, per-frame — | |||||
| Multi-HMR [4] | 110.5 | – | 115.3 | – | – |
| Multi-HMR2 [24] | 85.5 | – | 92.8 | – | – |
| SAM-3D-Body [103] | 71.0 | – | 76.4 | – | – |
| PyMAF-X [111] | 94.7 | 177.5 | 99.9 | 177.0 | – |
| PARE [51] | 98.7 | 132.3 | 102.0 | 134.8 | – |
| CameraHMR [70] | 78.7 | 98.7 | 85.7 | 105.2 | – |
| OSX [59] | 59.2 | 73.7 | 62.5 | 76.6 | – |
| — Single-view exocentric, temporal — | |||||
| SMPLer-X [11] | 57.0 | 70.2 | 59.8 | 71.7 | – |
| SMPLest-X [107] | 55.7 | 65.8 | 58.8 | 67.6 | – |
| Humans-in-4D [31] | 77.1 | 94.4 | 79.6 | 96.4 | – |
| GVHMR [84] | 88.0 | 104.1 | 96.3 | 112.0 | 217.1 |
| WHAM [85] | 131.1 | 160.2 | 144.0 | 166.4 | 243.1 |
| EasyMoCap [21, 86] | 103.4 | 155.8 | 106.0 | 156.1 | 256.2 |
| — Single-view exocentric, scene-aware — | |||||
| Phy-SIC [68] | 64.3 | 78.9 | 71.3 | 84.6 | – |
| UniSH [55] | 80.0 | 111.2 | 82.8 | 114.0 | 196.3 |
| JOSH [62] | 64.0 | 89.2 | 69.9 | 95.0 | 245.1 |
| Human3R [16] | 60.1 | 75.4 | 63.5 | 80.6 | 180.2 |
| — Multi-view exocentric — | |||||
| MAMMA [18] | 70.1 | 86.8 | 74.8 | 89.6 | 230.8 |
| U-HMR [57] | 134.4 | 176.0 | 150.2 | 179.2 | – |
| HSfM [67] | 92.8 | 133.8 | 93.8 | 134.8 | – |
| — Egocentric — | |||||
| EgoEgo [53] | 159.6 | 245.3 | 163.4 | 263.9 | 306.2 |
| EgoAllo [106] | 131.7 | 196.8 | 147.9 | 220.3 | 252.2 |
| Method | PA-MPJPE ↓ | MPJPE ↓ | F@5 ↑ | F@15 ↑ | AUCJ ↑ | Traj. err. ↓ |
|---|---|---|---|---|---|---|
| WildHands [76] | 11.2 | 12.6 | 0.175 | 0.774 | 0.776 | – |
| Dyn-HaMR [108] | 18.9 | 21.1 | 0.042 | 0.413 | 0.624 | 98.2 |
| HaWoR [112] | 13.8 | 17.4 | 0.130 | 0.666 | 0.729 | 102.1 |
| Method | PA-MPJPE ↓ | MPJPE ↓ | F@5 ↑ | F@15 ↑ | AUCJ ↑ | Traj. err. ↓ |
|---|---|---|---|---|---|---|
| HORT [17] | 10.8 | 12.3 | 0.276 | 0.776 | 0.784 | – |
| HaMeR [72] | 9.6 | 10.4 | 0.281 | 0.848 | 0.812 | – |
| HaPTIC [105] | 10.0 | 10.7 | 0.247 | 0.838 | 0.804 | 63.0 |
| WiLoR [75] | 9.1 | 9.9 | 0.313 | 0.854 | 0.819 | – |
| OmniHands [58] | 10.7 | 11.5 | 0.211 | 0.814 | 0.791 | – |
研究结果
- 在第一人称视频的手部动作估计中,逐帧方法WildHands的关节姿态精度最高,而视频类方法Dyn-HaMR和HaWoR的世界坐标轨迹误差高达98-102毫米,原因可能是相机运动估计误差。
- 在第三人称视频的手部动作估计中,五种方法的关节姿态精度相近,PA-MPJPE在9.1到10.8毫米之间;WiLoR表现最好,PA-MPJPE为9.1毫米,F@5为0.313,AUCJ为0.819,HaMeR紧随其后为9.6毫米。
- 使用固定第三人称相机的HaPTIC轨迹误差仅为63毫米,远低于第一人称世界坐标方法的98-102毫米,说明固定视角能提供更稳定的参考坐标系。
- 同时重建被操作物体的HORT在操作帧上的PA-MPJPE为10.8毫米,与其他方法相近,说明联合建模物体既没有明显提升也没有明显损害手部姿态精度。
- 将第一人称和第三人称方法直接比较后发现,第三人称方法在姿态和轨迹精度上都优于第一人称方法,这归因于固定相机坐标系比由头部运动推算出的坐标系更稳定。
可应用场景
- 作为同步的视觉-动作-触觉训练数据,用于家庭或人形机器人的模仿学习和策略学习
- 作为基准测试,检验仅凭第一人称视频估计手部动作或触觉信号的模型表现
- 为结合第一人称和第三人称视角的手物交互理解研究提供基础数据
局限与待验证事项
- 目前数据仅来自两处住宅场地,户型、家具和光照的多样性有限。
- 真实标注仅限于经过预先扫描并安装标记点的物体,门等关节机构、液体或可变形材料的状态变化未被标注。
- 动捕服、触觉手套、头戴设备和标记点在画面中始终可见,可能带来数据集特有的视觉痕迹。
- 结合第一人称与第三人称数据流、向第一人称方法提供实测头戴设备运动信息、或将物体姿态作为辅助输入等实验尚未开展,仅作为未来研究方向提出。
为什么重要
要让机器人或AI学会像人一样用手操作物体,需要能同时展示视觉、动作、声音和触觉如何在同一时刻交织的数据,而以往割裂的数据集难以满足这一需求。ACE提供了一种在真实家庭环境中同步精确记录这些信号的方法,为模仿学习和世界模型等机器人研究打下基础。
本文术语
- 第一人称/第三人称视频(egocentric/exocentric) · 佩戴在身上的相机拍到的第一人称画面,与固定在环境中的相机拍到的第三人称画面
- 六自由度姿态(6-DoF pose) · 用三个方向的位置和三个方向的旋转角度共六个数值来描述物体的完整三维姿态
- MANO · 一种表示手部关节和形状的标准三维手部模型
- PA-MPJPE · 在分别对齐尺度、位置和旋转后计算的关节位置误差,主要反映手指姿态的准确度
- 动作捕捉(OptiTrack) · 通过多台相机追踪身体或物体上贴附的标记点,记录精确三维动作的系统
论文原文摘要（英文）
Embodied intelligence faces a fundamental data bottleneck. Models must capture how first-person perception, whole-body motion, dexterous manipulation, object state, sound, and touch evolve together as humans pursue goals over time. Existing datasets fragment this experience across viewpoints, modalities, or spatial scales, leaving the full perception-action loop only partially observed. We introduce the Ambient Capture Engine (ACE), a human-centric data engine that transforms real home environme
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI助手在该向你提问的时候,却更愿意自己去核实事实
- SynFlow: A Multidimensional Diachronic Semantic Analysis Toolkit一款把单词意义变化拆解到语法细节的开源分析工具
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)用AI总结股市新闻发现:简单的摘要方法反而比时髦的检索增强技术更靠谱
METAL LAB 最新报道
图片来源： Yukang Cao et al., arXiv:2607.28625, arxiv-nonexclusive