One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

arXiv:2608.196842026-08-21

Teaching robots to pick out only the useful moves from messy, mixed-quality recorded data

When robots are pretrained on pre-collected datasets that mix successful, failed, and noisy behaviors, older methods fail to tell them apart, ending up with a cluttered library of skills. QDOS estimates how valuable each short segment of behavior was and uses that to weight learning, extracting skills that are both diverse and high quality. Across manipulation and locomotion tasks, this let QDOS reach goals faster and achieve higher final performance than the strongest prior method, SUPE.

What they did

  1. Standard approaches like trajectory VAEs treat every behavior segment in the dataset equally, so good moves, failed attempts, and noise all get baked into a tangled skill space
  2. QDOS estimates each segment's advantage (how much better it was than average) using a method called IQL, then weights both skill extraction and diversity objectives by this advantage
  3. It reuses the same offline dataset twice: once to pretrain the low-level skills, and again to fill the online replay buffer through pseudo-labeling, giving the high-level policy a head start
  4. In the complex antsoccer-arena task, QDOS reached a normalized return of 0.80, far above SUPE's 0.27, and in kitchen-mixed it hit a perfect 4.00 versus SUPE's 3.40
  5. QDOS also found the goal in far fewer training steps than SUPE across tasks, showing it explores more efficiently
Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining: The agent learns low-level skills (πθ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse: The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.
Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining: The agent learns low-level skills (πθ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse: The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.
Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.
Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.
TABLE I: Normalized evaluation returns comparing BC, IQL, SUPE, and QDOS across various domains. Results for SUPE and QDOS are averaged over 3 seeds.
EnvironmentBC[11][26]IQL[11][26]SUPEQDOS (α=0.2)QDOS (α=0.3)
antmaze-large41±764±1083.0±21.086.0±6.0¯96.0±6.0
antsoccer-arena5±150±227.0±31.073.0±31.0¯80.0±10.0
humanoidmaze-medium8±227±297.0±6.0¯97.0±6.0¯100.0±0.0
kitchen-mixed51.551.085±13.25¯100.00±0.0083.25±28.75
kitchen-partial38.046.383.25±14.5085.75±24.7582.50±30.25
scene-task15±151±4100.0±0.0100.0±0.0100.0±0.0
Fig. 3: Learning curves across environments. QDOS is shown in red (α=0.2) and blue (α=0.3); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.
Fig. 3: Learning curves across environments. QDOS is shown in red (α=0.2) and blue (α=0.3); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.
Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.
Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.
TABLE II: Training steps to reach the goal for the first time (Mean ± Std).
EnvironmentMethodSteps
antmaze-largeSUPE13334±5774
QDOS (α=0.2)𝟏𝟏𝟔𝟔𝟖±𝟗𝟒𝟔𝟓
humanoidmazeSUPE23334±6292
QDOS (α=0.3)𝟏𝟗𝟏𝟔𝟖±𝟖𝟎𝟑𝟔
sceneSUPE5834±1443
QDOS (α=0.2)𝟓𝟎𝟎𝟏±𝟎
Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight αloss leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.
Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight αloss leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.
Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen, antmaze, and antsoccer. Orange points represent QDOS (α=0.3 for antsoccer and antmaze, α=0.2 for kitchen), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.
Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen, antmaze, and antsoccer. Orange points represent QDOS (α=0.3 for antsoccer and antmaze, α=0.2 for kitchen), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.
TABLE III: Hyperparameters for VAE training.
Parameter NameValue
Batch size256
OptimizerAdam
Learning rate3×10−4
GRU Hidden Size256 (antmaze, kitchen)
512 (antsoccer, humanoidmaze, scene)
GRU Layers2 (antmaze, kitchen)
3 (antsoccer, humanoidmaze, scene)
KL Coefficient (OPENβ)0.1 (antmaze, humanoid, kitchen)
0.2 (scene)
Latent Dimension (z)8
Segment Length (H)20
Fig. 7: QDOS skill trajectory visualization for kitchen-mixed. Red crosses mark selected timesteps (t=0,7,14,22) with corresponding scene snapshots.
Fig. 7: QDOS skill trajectory visualization for kitchen-mixed. Red crosses mark selected timesteps (t=0,7,14,22) with corresponding scene snapshots.

Why it matters

Real-world robot datasets are rarely clean, they typically mix successes, failures, and noise, so a method that reliably filters this out makes offline-to-online learning more practical. This can reduce costly real-world trial and error while still boosting final task performance.

Terms in this paper

  • Offline RL · Learning a policy purely from a fixed, pre-collected dataset without interacting with the environment
  • VAE (Variational Autoencoder) · An unsupervised model that compresses data into a latent representation and reconstructs it back
  • Advantage · A measure of how much better an action or behavior segment performed compared to average
  • IQL (Implicit Q-Learning) · An offline RL technique that estimates value functions without querying actions outside the dataset
  • Pseudo-labeling · Using a trained encoder to assign inferred skill labels and reward estimates to offline data segments

Original abstract (English)

Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.

Authors · Tanachai Anakewat, Takayuki Osa, Tatsuya Harada

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Tanachai Anakewat et al., arXiv:2608.19684, arxiv-nonexclusive