每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

arXiv:2608.196842026-08-21

让机器人从混杂着成功、失败与噪声的旧数据中只学到有用的动作

当机器人用预先收集的数据进行预训练时,如果数据里成功动作、失败尝试和噪声混在一起,旧方法往往无法区分,学出的技能库会变得杂乱无章。QDOS通过估计每段轨迹的价值高低来加权学习,从而提取出既多样又高质量的技能。在机械臂操作和多足行走等任务中,QDOS比此前最强的方法SUPE更快找到目标,最终表现也更好。

他们做了什么

  1. 传统方法如轨迹VAE对数据集中所有片段一视同仁地学习,导致好的动作、失败的尝试和噪声全部混入同一个技能空间,变得杂乱
  2. QDOS用IQL方法估计每段轨迹的优势值(相对于平均水平的好坏程度),并用这个优势值对技能提取和多样性目标同时加权
  3. 该方法把同一份离线数据重复利用两次:一次用于预训练底层技能,另一次通过伪标注填充在线学习的经验回放池,让高层策略从一开始就能利用先验知识
  4. 在复杂的antsoccer-arena推球任务中,QDOS的归一化得分达到0.80,远超SUPE的0.27;在kitchen-mixed厨房操作任务中QDOS拿到满分4.00,而SUPE只有3.40
  5. 在多个任务中,QDOS找到目标所需的训练步数也明显少于SUPE,说明探索效率更高
Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining: The agent learns low-level skills (πθ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse: The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.
Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining: The agent learns low-level skills (πθ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse: The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.
Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.
Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.
TABLE I: Normalized evaluation returns comparing BC, IQL, SUPE, and QDOS across various domains. Results for SUPE and QDOS are averaged over 3 seeds.
EnvironmentBC[11][26]IQL[11][26]SUPEQDOS (α=0.2)QDOS (α=0.3)
antmaze-large41±764±1083.0±21.086.0±6.0¯96.0±6.0
antsoccer-arena5±150±227.0±31.073.0±31.0¯80.0±10.0
humanoidmaze-medium8±227±297.0±6.0¯97.0±6.0¯100.0±0.0
kitchen-mixed51.551.085±13.25¯100.00±0.0083.25±28.75
kitchen-partial38.046.383.25±14.5085.75±24.7582.50±30.25
scene-task15±151±4100.0±0.0100.0±0.0100.0±0.0
Fig. 3: Learning curves across environments. QDOS is shown in red (α=0.2) and blue (α=0.3); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.
Fig. 3: Learning curves across environments. QDOS is shown in red (α=0.2) and blue (α=0.3); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.
Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.
Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.
TABLE II: Training steps to reach the goal for the first time (Mean ± Std).
EnvironmentMethodSteps
antmaze-largeSUPE13334±5774
QDOS (α=0.2)𝟏𝟏𝟔𝟔𝟖±𝟗𝟒𝟔𝟓
humanoidmazeSUPE23334±6292
QDOS (α=0.3)𝟏𝟗𝟏𝟔𝟖±𝟖𝟎𝟑𝟔
sceneSUPE5834±1443
QDOS (α=0.2)𝟓𝟎𝟎𝟏±𝟎
Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight αloss leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.
Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight αloss leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.
Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen, antmaze, and antsoccer. Orange points represent QDOS (α=0.3 for antsoccer and antmaze, α=0.2 for kitchen), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.
Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen, antmaze, and antsoccer. Orange points represent QDOS (α=0.3 for antsoccer and antmaze, α=0.2 for kitchen), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.
TABLE III: Hyperparameters for VAE training.
Parameter NameValue
Batch size256
OptimizerAdam
Learning rate3×10−4
GRU Hidden Size256 (antmaze, kitchen)
512 (antsoccer, humanoidmaze, scene)
GRU Layers2 (antmaze, kitchen)
3 (antsoccer, humanoidmaze, scene)
KL Coefficient (OPENβ)0.1 (antmaze, humanoid, kitchen)
0.2 (scene)
Latent Dimension (z)8
Segment Length (H)20
Fig. 7: QDOS skill trajectory visualization for kitchen-mixed. Red crosses mark selected timesteps (t=0,7,14,22) with corresponding scene snapshots.
Fig. 7: QDOS skill trajectory visualization for kitchen-mixed. Red crosses mark selected timesteps (t=0,7,14,22) with corresponding scene snapshots.

为什么重要

现实世界收集的机器人数据往往并不干净,而是成功、失败与噪声混杂在一起,能可靠地从中筛选出有用技能的方法让离线到在线的学习更实用。这有助于在减少昂贵的真实环境试错的同时,依然提升最终任务表现。

本文术语

  • 离线强化学习 · 完全依靠预先收集好的数据训练策略,不与环境实时交互
  • 变分自编码器(VAE) · 一种无监督模型,把数据压缩成潜在表示后再尝试还原
  • 优势值(advantage) · 衡量某个动作或轨迹片段相对平均水平好坏程度的指标
  • IQL(隐式Q学习) · 一种无需查询数据集之外动作即可稳定估计价值函数的离线强化学习方法
  • 伪标注(pseudo-labeling) · 用已训练好的编码器给离线数据片段标注推断出的技能类别和奖励估计

论文原文摘要（英文）

Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.

作者 · Tanachai Anakewat, Takayuki Osa, Tatsuya Harada

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Tanachai Anakewat et al., arXiv:2608.19684, arxiv-nonexclusive