每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

τ_0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation

arXiv:2608.168852026-08-16

机器人遇到难以判断的时刻,会先在脑中想象几种可能的下一步,再做决定

τ0-VLA是一个分层机器人系统,在打扫房间、做饭、做奶茶等耗时数分钟到12分钟的长任务中,遇到不确定的时刻不会只算一次就下结论,而是生成多个候选下一步,用世界模型预测每个候选执行后画面会变成什么样,再打分挑选,最后才决定实际要做的动作。该系统用40115小时的真实机器人数据训练而成,在实体机器人实验中,测试时投入越多计算,下一步预测准确率和最终任务成功率都随之提高。论文作者为Xiaowei Cai,发布于arXiv,编号2608.16885。

他们做了什么

  1. 以往大多数分层机器人AI只用一次前向计算就决定下一步该做什么,遇到困难或关键决策时也无法多花点计算力气
  2. τ0-VLA只在模型信心不足时才启动额外搜索:生成多个候选子任务,用世界模型预测每个候选执行完后的画面,再用价值模型打分并做集束搜索,最后生成最终决定
  3. 选定的子任务交给统一40维动作空间的下层执行模型完成,使同一套模型能操控多种不同结构的机器人本体
  4. 在打扫房间、备菜、番茄炒蛋、做奶茶、收衣服、整理书本等真实机器人任务中,测试时计算量增加,下一步预测准确率和闭环任务成功率都随之提升
  5. 在训练中从未见过的书本摆放顺序(分布外场景)下,同样的提升趋势依然成立,说明方法对陌生情况也有一定稳健性
Fig. 2: The hierarchical τ0-VLA architecture. (a) At a high-level inference step t, the proposal model P conditions on the latest multi-view observation ot, task instruction ℓ, carried execution memory ℳt−1, and previously generated subtask zt−1⋆. It produces observation-aligned memory ℳt and a direct proposal ztdir. (b) The low-level policy conditions on the generated subtask zt⋆, multi-view observation ot, proprioceptive state 𝐬t, and textual control metadata η. A vision-language backbone and Mixture-of-Transformers (MoT) action expert generate the action chunk 𝐚t:t+H−1 through conditional flow matching from a noisy action chunk. (c) On the TTC route, the proposal model is invoked N times for each retained branch to generate N candidates. Given the branch’s head-camera image and a candidate, the world model predicts the terminal head-camera image, and the value model assigns a candidate-quality score conditioned on the task instruction, candidate, and predicted image. Beam search globally retains the top-B branches by cumulative score and recursively expands them to depth D. The figure illustrates the root expansion with N=3 and B=2, where local and cumulative scores coincide. Deeper expansion is omitted for clarity. The reflective model then conditions on h¯t and the final branch summaries 𝒞t to generate the final subtask zt⋆. This output may coincide with a retained proposal but is not restricted to the retained set and is passed to the low-level policy for execution.
Fig. 2: The hierarchical τ0-VLA architecture. (a) At a high-level inference step t, the proposal model P conditions on the latest multi-view observation ot, task instruction ℓ, carried execution memory ℳt−1, and previously generated subtask zt−1⋆. It produces observation-aligned memory ℳt and a direct proposal ztdir. (b) The low-level policy conditions on the generated subtask zt⋆, multi-view observation ot, proprioceptive state 𝐬t, and textual control metadata η. A vision-language backbone and Mixture-of-Transformers (MoT) action expert generate the action chunk 𝐚t:t+H−1 through conditional flow matching from a noisy action chunk. (c) On the TTC route, the proposal model is invoked N times for each retained branch to generate N candidates. Given the branch’s head-camera image and a candidate, the world model predicts the terminal head-camera image, and the value model assigns a candidate-quality score conditioned on the task instruction, candidate, and predicted image. Beam search globally retains the top-B branches by cumulative score and recursively expands them to depth D. The figure illustrates the root expansion with N=3 and B=2, where local and cumulative scores coincide. Deeper expansion is omitted for clarity. The reflective model then conditions on h¯t and the final branch summaries 𝒞t to generate the final subtask zt⋆. This output may coincide with a retained proposal but is not restricted to the retained set and is passed to the low-level policy for execution.
TABLE I: Long-horizon task performance. Each method–task setting uses 10 independently collected physical-robot trials. SR reports successful trials as x/10, and Progress is the normalized milestone-completion score. Avg. is the unweighted mean of the four task-level rates. The first four rows use direct execution. The final row uses the Hierarchical System with Plan Once and no beam search.
MethodClean RoomPrepare IngredientsTomato and Egg Stir FryMake Milk TeaAvg.
SR ↑Progress ↑SR ↑Progress ↑SR ↑Progress ↑SR ↑Progress ↑SR ↑Progress ↑
GR00T N1.7 [26]0/1059.80%1/1068.57%0/1024.32%0/1028.46%2.50%45.29%
LingBot-VLA [40]0/1066.60%0/1035.00%0/1012.27%0/1063.85%0.00%44.43%
π0.5 [2]4/1086.20%2/1073.93%0/1049.77%3/1082.31%22.50%73.05%
τ0-VLA4/1092.80%2/1066.43%0/1065.00%5/1096.15%27.50%80.10%
τ0-VLA (Hierarchical System, Plan Once)5/1094.80%4/1082.86%4/1081.82%5/1091.92%45.00%87.85%
Fig. 3: Representative physical-robot evaluation tasks. (a) Clean Room requires collecting two dirty garments, placing them in a laundry basket, hanging a handbag, handing a blanket to a person, and disposing of table trash. (b) Prepare Ingredients requires retrieving a tomato and an egg, then cracking and stirring the egg while returning the tools. (c) Tomato and Egg Stir Fry requires chopping and transferring the tomatoes, cooking and seasoning the ingredients, plating the dish, and returning the cookware and utensils. (d) Make Milk Tea requires adding toppings, pouring milk and tea, sealing the cup, and inserting a straw. (e) Collect Laundry requires transferring a T-shirt from the bedside table to a laundry basket. (f) Tidy Makeup Table comprises three separately scored instruction-following groups that require different object selections and action sequences from matched visual states.
Fig. 3: Representative physical-robot evaluation tasks. (a) Clean Room requires collecting two dirty garments, placing them in a laundry basket, hanging a handbag, handing a blanket to a person, and disposing of table trash. (b) Prepare Ingredients requires retrieving a tomato and an egg, then cracking and stirring the egg while returning the tools. (c) Tomato and Egg Stir Fry requires chopping and transferring the tomatoes, cooking and seasoning the ingredients, plating the dish, and returning the cookware and utensils. (d) Make Milk Tea requires adding toppings, pouring milk and tea, sealing the cup, and inserting a straw. (e) Collect Laundry requires transferring a T-shirt from the bedside table to a laundry basket. (f) Tidy Makeup Table comprises three separately scored instruction-following groups that require different object selections and action sequences from matched visual states.
TABLE II: Direct-execution performance across embodiments. Tidy Makeup Table comprises three independently scored instruction-following groups. All methods execute the full task instruction without a high-level policy. Each task or task group is evaluated over 10 trials. SR reports successful trials as x/10, and Progress is the normalized milestone-completion score.
MethodCollect LaundryTidy Makeup Table
T-shirtCotton PadEyelash CurlerMakeup Puff
SR ↑Progress ↑SR ↑Progress ↑SR ↑Progress ↑SR ↑Progress ↑
GR00T N1.7 [26]4/1076.00%10/1087.50%8/1077.50%7/1052.50%
LingBot-VLA [40]2/1035.00%9/1067.50%3/1022.50%3/1033.75%
π0.5 [2]9/1088.00%9/1085.00%8/1085.00%7/1073.75%
τ0-VLA10/1097.00%10/1095.00%9/1092.50%10/1095.00%
Fig. 4: Next-subtask prediction accuracy under different high-level inference methods. We compare Plan Once, Best-of-N, and TTC (Ours) across four evaluation settings: Make Milk Tea, Book Organization (In-Domain), Book Organization (OOD), and Clean Room.
Fig. 4: Next-subtask prediction accuracy under different high-level inference methods. We compare Plan Once, Best-of-N, and TTC (Ours) across four evaluation settings: Make Milk Tea, Book Organization (In-Domain), Book Organization (OOD), and Clean Room.
TABLE III: Closed-loop physical-robot performance with test-time computation. Each entry uses 10 independently collected trials. Book Organization uses shuffled initial arrangements and is reported without the in-domain and OOD split used in the open-loop evaluation. SR denotes task success rate, and Progress is the normalized milestone-completion score.
MethodMake Milk TeaBook OrganizationClean Room
SR ↑Progress ↑SR ↑Progress ↑SR ↑Progress ↑
Plan Once5/1091.92%6/1066.67%5/1094.80%
TTC7/1095.38%9/1093.33%7/1097.60%
Fig. 5: Relationship between computational cost and subtask-prediction accuracy. The accuracy increases with additional computation for the Make Milk Tea and Book Organization tasks, shown in panels (a) and (b), respectively. Each point represents an experimental result obtained at a different computational cost. The orange dashed curves show saturation fits to these results, and the gray dashed lines indicate the Plan Once baselines.
Fig. 5: Relationship between computational cost and subtask-prediction accuracy. The accuracy increases with additional computation for the Make Milk Tea and Book Organization tasks, shown in panels (a) and (b), respectively. Each point represents an experimental result obtained at a different computational cost. The orange dashed curves show saturation fits to these results, and the gray dashed lines indicate the Plan Once baselines.
TABLE IV: Canonical 40-D state and action layout. Dimensions are one-indexed. For a rotation matrix 𝐑=[𝐫1,𝐫2,𝐫3], we use Rot6D⁡(𝐑)=[𝐫1⊤,𝐫2⊤]⊤.
CoordinatesDimensionsState representation
Left EEF position1–3Cartesian position in meters
Left EEF orientation4–9Rot6D⁡(𝐑L)
Right EEF position10–12Cartesian position in meters
Right EEF orientation13–18Rot6D⁡(𝐑R)
Left gripper19native opening coordinate
Right gripper20native opening coordinate
Waist21–22two native coordinates
Planar base velocity23–24two native coordinates
Left arm joints25–32q1L,…,q8L in radians
Right arm joints33–40q1R,…,q8R in radians
TABLE V: Maximum duration of each physical-robot trial.
TaskMaximum duration
Clean Room20 min
Prepare Ingredients20 min
Tomato and Egg Stir Fry20 min
Make Milk Tea10 min
Book Organization5 min
Collect Laundry5 min
Tidy Makeup Table (each group)5 min
TABLE VI: High-level instance families, synthesized by perturbing only the input memory while reading the corrected target from the demonstration. ℳn is the memory upon entering segment n. A single unified <think>/<memory>/<subtask> format instantiates all of them at zero extra annotation.
FamilySampling positionInput → target memoryTarget subtaskDeployment failure counteredMix
within-subtaskanywhere in seg. nℳn→ℳnseg. n— (aligned, normal progression)58%
transitiontail of seg. nℳn→ℳn+1seg. n+1starting a new subtask after completion15%
catch-uphead of seg. nℳn−1→ℳnseg. nmemory lag (behind the visual state)10%
rollbacklate in seg. nℳn+1​…​n+3→ℳnretry seg. nmemory run-ahead (over-optimistic)12%
error-thinkannotated failure frameℳn→ type-dependentrecovery stepunnoticed execution failure5%

为什么重要

在长时间的机器人任务中,一旦选错了下一步,即便动作执行得再精准也无法挽回,因此只在关键时刻多花计算力的方法对打造可靠的家用机器人很有意义。这也说明语言模型中流行的测试时算力扩展思路,可以迁移到真实机器人控制上并带来实际收益。

本文术语

  • VLA(视觉-语言-动作)模型 · 把摄像头画面和语言指令转换成机器人动作指令的AI模型
  • 测试时计算(Test-Time Computation) · 不重新训练模型,而是在实际使用时投入更多计算来提升答案质量的方法
  • 世界模型(World Model) · 预测某个动作执行后环境会变成什么样子的模型
  • 集束搜索(Beam Search) · 每一步只保留得分最高的几条候选路径继续往下扩展的搜索方法
  • 执行记忆(Execution Memory) · 记录机器人到目前为止任务进展到哪一步的摘要信息

论文原文摘要（英文）

Long-horizon robot manipulation requires a robot to both execute individual skills reliably and sequence them coherently over extended tasks. Most hierarchical vision-language-action (VLA) models make each such decision with a single forward pass, leaving no mechanism to allocate additional computation to difficult or consequential choices. We introduce τ_0-VLA, a hierarchical robot foundation model that formulates high-level subtask generation as a compute-scalable inference problem through world-model-guided test-time computation. At each inference step, the high-level policy uses execution memory to generate a subtask and, when needed, searches over alternatives before committing to its output. A low-level policy then executes the generated subtask across multiple robot embodiments. The policy is trained on 40,115 hours of heterogeneous real-world data with multimodal co-training. Across in-domain and distribution-shifted settings, allocating additional test-time computation substantially improves next-subtask prediction accuracy, and these gains translate into higher closed-loop success on long-horizon robot manipulation tasks.

作者 · Xiaowei Cai

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Xiaowei Cai et al., arXiv:2608.16885, arxiv-nonexclusive