每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

EXIMO: VLM Guided Exploration of VLA Policies

arXiv:2608.198912026-08-21

不用人工遥控演示,让会说话的AI来教机械臂做新家务

机器人操作策略(VLA模型)通常靠模仿人工遥控采集的数据来训练,每学一个新任务都要重新花大量人力,效率很低。EXIMO让一个视觉语言模型(VLM)把复杂的长任务拆成机器人已经会做的简单步骤,收集成功的执行数据,再用这些数据重新微调机器人策略,最后再用在线强化学习进一步打磨。在22项操作任务的实验中,该方法在数据效率和最终成功率上都明显超过现有方法。

他们做了什么

  1. 基础机器人策略是30亿参数的Gemini Robotics On-Device(GROD),而Gemini作为VLM'调度者',观察场景并用自然语言给出机器人本身就能执行的简短指令
  2. 只保留VLM指导下成功完成的操作轨迹,用这些数据对原始VLA模型做监督微调,让机器人把新技能内化,评估时不再需要VLM的实时协助
  3. 最后阶段用离线策略强化学习(MPO算法)训练一个小型残差策略,对VLA输出的动作做微调修正
  4. 在22个任务上,VLM引导的探索比单纯的机器人策略成功率更高,而用这些数据微调后的机器人甚至比评估时一直依赖VLM的效果还要好
  5. 经过完整流程(含强化学习)后,微调策略的最终成功率和完成速度都超过了给予更多训练步数进行强化学习的基础策略
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Table 1: Manipulation tasks used in our evaluation and their natural-language goals. The task identifiers (T2–T23) correspond to the per-task labels in Fig. 2 and Fig. 4; T1 in those plots denotes the average over all tasks. Reasoning variants replace the explicit object name with a semantic description that the agent must ground to the correct object, while the left/right caddy tasks additionally require spatial understanding.
IDTaskNatural-language goal
T2BowlGlassOnRackput the bowl and glass on the rack
T3BananaInBowl-Reasoning0put the item that a monkey can eat into the bowl
T4MugOnPlateput the mug on the plate
T5MugOnPlate-Reasoning0put the object you pour coffee in on the plate
T6MugOnPlate-Reasoning1put the object with a handle on top of the flat object
T7PenInContainerput the pen into the white container
T8PenInContainer-Reasoning0put the object you use to write into the white container
T9PenInContainer-Reasoning1put the thinnest object into the white container
T10CanOpenerInCaddy-Left-Reasoning0place the can opener in the left compartment of the caddy
T11CanOpenerInCaddy-Right-Reasoning0place the can opener in the right compartment of the caddy
T12MagnifierCanOpenerInCaddyput the magnifier and can opener in the caddy
T13MagnifierInCaddy-Left-Reasoning0place the magnifier in the left compartment of the caddy
T14MagnifierInCaddy-Right-Reasoning0place the magnifier in the right compartment of the caddy
T15ScissorsInCaddy-Left-Reasoning0place the scissors in the left compartment of the caddy
T16ScissorsInCaddy-Right-Reasoning0place the scissors in the right compartment of the caddy
T17ScissorsMagnifierInCaddyput the scissors and magnifier in the caddy
T18ScissorsScrewdriverInCaddyput the scissors and screwdriver in the caddy
T19ScrewdriverInCaddy-Left-Reasoning0place the screwdriver in the left compartment of the caddy
T20ScrewdriverInCaddy-Right-Reasoning0place the screwdriver in the right compartment of the caddy
T21ScrewdriverMagnifierInCaddyput the screwdriver and magnifier in the caddy
T22PlateBowlOnRackput the plate and bowl on the rack
T23PlateGlassOnRackput the plate and glass on the rack
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.

为什么重要

人工遥控采集机器人演示数据成本高、耗时长,这项工作展示了如何借助语言能力强的AI模型大幅降低这一成本。它为用更少的人力教机器人学习新任务提供了一条实用路径。

EXIMO: VLM Guided Exploration of VLA Policies figure 4
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.

本文术语

  • VLA(视觉-语言-动作模型) · 接收摄像头图像和语言指令、输出机器人动作的大型模型
  • VLM(视觉语言模型) · 能同时理解图像和文本的AI模型,这里扮演监督者角色
  • 行为克隆(Behaviour Cloning) · 通过模仿人工遥控记录的数据来训练模型的方法
  • 强化学习(RL) · 智能体通过试错并根据成功/失败信号不断改进的学习方式
  • 残差策略(Residual Policy) · 在现有模型输出动作基础上叠加微小修正值的辅助策略
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.

论文原文摘要（英文）

How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision language model (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.

作者 · Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Bhavya Sukhija et al., arXiv:2608.19891, CC BY 4.0