매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

EXIMO: VLM Guided Exploration of VLA Policies

arXiv:2608.198912026-08-21

로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다

로봇 조작 정책(VLA)은 사람이 조종해서 만든 데이터로 학습하는데, 새로운 작업을 배우려면 다시 수백 시간의 사람 조작 데이터가 필요해서 비효율적이다. EXIMO는 언어 이해에 강한 비전-언어 모델(VLM)이 어려운 작업을 잘게 쪼개 로봇에게 지시하며 데이터를 모으고, 그 데이터로 로봇 모델을 다시 학습시킨 뒤 온라인 강화학습으로 마무리하는 3단계 방법이다. 22가지 조작 과제 실험에서 기존 방식보다 적은 데이터로 더 높은 성공률을 보였다.

무엇을 했나

  1. 구글의 Gemini Robotics On-Device(GROD)라는 30억 파라미터 로봇 정책을 기본 모델로 사용하고, Gemini라는 VLM이 감독관 역할을 맡아 긴 작업을 짧은 지시로 쪼개 로봇에게 실시간으로 명령한다
  2. VLM이 지시한 대로 로봇이 실행해 성공한 시도만 모아 데이터셋을 만들고, 이 데이터로 원래 VLA 모델을 다시 지도학습(파인튜닝)시켜 VLM 없이도 스스로 잘하게 만든다
  3. 마지막으로 로봇 행동에 미세한 보정값을 더하는 잔차 정책을 강화학습으로 학습시켜 성능을 한 번 더 끌어올린다
  4. 22개 조작 과제 실험에서 VLM 지시를 받은 로봇이 그냥 로봇보다 성공률이 높았고, 그 데이터로 다시 학습시킨 로봇은 VLM 지시를 받는 로봇보다도 더 잘했다
  5. 강화학습까지 마친 EXIMO는 기존 로봇을 훨씬 많은 데이터로 강화학습시킨 경우보다도 더 높은 성공률과 빠른 성공 시간을 보였다
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Table 1: Manipulation tasks used in our evaluation and their natural-language goals. The task identifiers (T2–T23) correspond to the per-task labels in Fig. 2 and Fig. 4; T1 in those plots denotes the average over all tasks. Reasoning variants replace the explicit object name with a semantic description that the agent must ground to the correct object, while the left/right caddy tasks additionally require spatial understanding.
IDTaskNatural-language goal
T2BowlGlassOnRackput the bowl and glass on the rack
T3BananaInBowl-Reasoning0put the item that a monkey can eat into the bowl
T4MugOnPlateput the mug on the plate
T5MugOnPlate-Reasoning0put the object you pour coffee in on the plate
T6MugOnPlate-Reasoning1put the object with a handle on top of the flat object
T7PenInContainerput the pen into the white container
T8PenInContainer-Reasoning0put the object you use to write into the white container
T9PenInContainer-Reasoning1put the thinnest object into the white container
T10CanOpenerInCaddy-Left-Reasoning0place the can opener in the left compartment of the caddy
T11CanOpenerInCaddy-Right-Reasoning0place the can opener in the right compartment of the caddy
T12MagnifierCanOpenerInCaddyput the magnifier and can opener in the caddy
T13MagnifierInCaddy-Left-Reasoning0place the magnifier in the left compartment of the caddy
T14MagnifierInCaddy-Right-Reasoning0place the magnifier in the right compartment of the caddy
T15ScissorsInCaddy-Left-Reasoning0place the scissors in the left compartment of the caddy
T16ScissorsInCaddy-Right-Reasoning0place the scissors in the right compartment of the caddy
T17ScissorsMagnifierInCaddyput the scissors and magnifier in the caddy
T18ScissorsScrewdriverInCaddyput the scissors and screwdriver in the caddy
T19ScrewdriverInCaddy-Left-Reasoning0place the screwdriver in the left compartment of the caddy
T20ScrewdriverInCaddy-Right-Reasoning0place the screwdriver in the right compartment of the caddy
T21ScrewdriverMagnifierInCaddyput the screwdriver and magnifier in the caddy
T22PlateBowlOnRackput the plate and bowl on the rack
T23PlateGlassOnRackput the plate and glass on the rack
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.

왜 중요한가

사람이 로봇을 직접 조종해 데이터를 모으는 작업은 비용이 크고 시간이 오래 걸리는데, 이 방법은 말 잘하는 AI의 지식을 빌려 그 부담을 크게 줄인다. 새로운 로봇 작업을 훨씬 적은 인력과 시간으로 가르칠 수 있는 실용적인 방향을 보여준다.

EXIMO: VLM Guided Exploration of VLA Policies figure 4
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.

이 논문의 용어

  • VLA(비전-언어-행동 모델) · 카메라로 본 장면과 언어 명령을 입력받아 로봇의 움직임(행동)을 출력하는 대형 모델
  • VLM(비전-언어 모델) · 이미지와 텍스트를 함께 이해하고 답할 수 있는 AI 모델. 여기서는 감독관 역할
  • 행동 복제(Behaviour Cloning) · 사람이 조종한 데이터를 그대로 따라 하도록 모델을 학습시키는 방식
  • 강화학습(RL) · 로봇이 스스로 시행착오를 겪으며 성공/실패 신호로 점점 개선하는 학습 방식
  • 잔차 정책(Residual Policy) · 기존 로봇 모델의 행동에 작은 보정값을 더해 성능을 개선하는 추가 학습 모델
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.

논문 원문 초록 (영문)

How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision language model (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.

저자 · Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Bhavya Sukhija et al., arXiv:2608.19891, CC BY 4.0