One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

EXIMO: VLM Guided Exploration of VLA Policies

arXiv:2608.198912026-08-21

Teaching a robot new chores without human teleoperation, by letting a chatty AI supervise it

Robot manipulation policies (VLA models) are usually trained by cloning hours of human-controlled demonstrations, which is expensive to redo for every new task. EXIMO instead uses a vision-language model (VLM) to break hard, long tasks into simple steps the robot already knows, collects successful attempts, retrains the robot policy on that data, and finally polishes it with online reinforcement learning. Across 22 manipulation tasks, this three-step pipeline beat existing methods on both data efficiency and final success rate.

What they did

  1. The base robot policy is Gemini Robotics On-Device (GROD), a 3-billion-parameter VLA, while Gemini acts as the VLM 'orchestrator' that watches the scene and issues short, natural-language instructions the robot can already execute
  2. Only successful rollouts guided by the VLM are kept and used to fine-tune the original VLA via supervised learning, so the robot internalizes the new skill and no longer needs the VLM's help at test time
  3. A final stage trains a small residual policy with off-policy reinforcement learning (MPO algorithm) to add corrective tweaks to the VLA's actions
  4. Across 22 tasks, VLM-guided exploration raised success rates over the plain robot policy, and fine-tuning on that collected data outperformed even keeping the VLM active during evaluation
  5. After the full pipeline including RL, the fine-tuned policy converged to higher success rates and faster completion than the base policy given far more RL training steps to compensate
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Table 1: Manipulation tasks used in our evaluation and their natural-language goals. The task identifiers (T2–T23) correspond to the per-task labels in Fig. 2 and Fig. 4; T1 in those plots denotes the average over all tasks. Reasoning variants replace the explicit object name with a semantic description that the agent must ground to the correct object, while the left/right caddy tasks additionally require spatial understanding.
IDTaskNatural-language goal
T2BowlGlassOnRackput the bowl and glass on the rack
T3BananaInBowl-Reasoning0put the item that a monkey can eat into the bowl
T4MugOnPlateput the mug on the plate
T5MugOnPlate-Reasoning0put the object you pour coffee in on the plate
T6MugOnPlate-Reasoning1put the object with a handle on top of the flat object
T7PenInContainerput the pen into the white container
T8PenInContainer-Reasoning0put the object you use to write into the white container
T9PenInContainer-Reasoning1put the thinnest object into the white container
T10CanOpenerInCaddy-Left-Reasoning0place the can opener in the left compartment of the caddy
T11CanOpenerInCaddy-Right-Reasoning0place the can opener in the right compartment of the caddy
T12MagnifierCanOpenerInCaddyput the magnifier and can opener in the caddy
T13MagnifierInCaddy-Left-Reasoning0place the magnifier in the left compartment of the caddy
T14MagnifierInCaddy-Right-Reasoning0place the magnifier in the right compartment of the caddy
T15ScissorsInCaddy-Left-Reasoning0place the scissors in the left compartment of the caddy
T16ScissorsInCaddy-Right-Reasoning0place the scissors in the right compartment of the caddy
T17ScissorsMagnifierInCaddyput the scissors and magnifier in the caddy
T18ScissorsScrewdriverInCaddyput the scissors and screwdriver in the caddy
T19ScrewdriverInCaddy-Left-Reasoning0place the screwdriver in the left compartment of the caddy
T20ScrewdriverInCaddy-Right-Reasoning0place the screwdriver in the right compartment of the caddy
T21ScrewdriverMagnifierInCaddyput the screwdriver and magnifier in the caddy
T22PlateBowlOnRackput the plate and bowl on the rack
T23PlateGlassOnRackput the plate and glass on the rack
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.

Why it matters

Collecting robot demonstrations by human teleoperation is costly and slow, and this approach shows how to borrow knowledge from language-savvy AI models to cut that cost dramatically. It points to a practical way of teaching robots new tasks with far less human labor.

EXIMO: VLM Guided Exploration of VLA Policies figure 4
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.

Terms in this paper

  • VLA (vision-language-action model) · A large model that takes in camera images and language instructions and outputs robot actions
  • VLM (vision-language model) · An AI model that understands both images and text; here it plays the supervisor role
  • Behaviour cloning · Training a model to imitate recorded human-controlled demonstrations
  • Reinforcement learning (RL) · Training an agent by trial and error using success/failure signals from the environment
  • Residual policy · A small extra policy that adds correction tweaks on top of an existing model's actions
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.

Original abstract (English)

How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision language model (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.

Authors · Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Bhavya Sukhija et al., arXiv:2608.19891, CC BY 4.0