매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

arXiv:2608.196842026-08-21

로봇이 뒤죽박죽 실패 기록에서도 쓸모있는 동작만 골라 배우는 법

로봇을 미리 모아둔 데이터로 훈련시킬 때, 그 데이터에 성공과 실패, 잡음이 섞여 있으면 기존 방법은 이를 구분하지 못하고 뒤죽박죽인 동작 사전을 만든다. QDOS는 각 행동 구간이 얼마나 가치 있었는지를 추정해 가중치를 매기고, 그 가중치를 반영해 다양하면서도 우수한 동작들만 골라 배운다. 그 결과 로봇 팔 조작과 다리 보행 과제 모두에서 기존 최고 성능 방법보다 목표 도달 속도와 최종 성과가 크게 향상됐다.

무엇을 했나

  1. 기존 방식(트라젝토리 VAE 등)은 데이터에 담긴 성공, 실패, 잡음을 구분하지 않고 똑같이 학습해 동작 사전이 지저분해지는 문제가 있었다
  2. QDOS는 IQL이라는 방법으로 각 행동 구간의 가치(어드밴티지)를 추정하고, 가치가 높은 구간에 더 큰 가중치를 줘서 동작을 추출하면서 동시에 다양성도 유지하도록 설계했다
  3. 미리 모은 데이터를 동작 학습에 한 번, 그리고 실시간 학습용 경험 버퍼를 채우는 데 또 한 번, 총 두 번 재사용하는 전략을 함께 적용했다
  4. 복잡한 축구공 밀기 과제에서 정규화 점수 0.80으로 기존 최고 방법인 SUPE의 0.27을 크게 앞섰고, 주방 조작 과제에서는 만점인 4.00을 달성해 SUPE의 3.40을 넘어섰다
  5. 목표를 처음 찾는 데 걸리는 학습 스텝 수도 SUPE보다 눈에 띄게 줄어, 탐색 속도가 빨라졌음을 확인했다
Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining: The agent learns low-level skills (πθ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse: The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.
Fig. 1: Overview of QDOS Framework. (1) Offline Pretraining: The agent learns low-level skills (πθ) from a mixed-quality dataset containing good moves, failed attempts, and noise. We employ an Advantage-Weighted Quality-Diversity objective to filter out sub-optimal data (failed attempts and noise) while extracting diverse, high-quality skills. (2) Online RL with Dual Dataset Reuse: The same offline data is reused to populate the high-level replay buffer via pseudo-labeling, enabling the high-level policy to utilize prior knowledge for efficient online exploration.
Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.
Fig. 2: The Mixed-Quality Dilemma: Standard VAEs encode all behaviors equally, entangling noise with useful skills.
TABLE I: Normalized evaluation returns comparing BC, IQL, SUPE, and QDOS across various domains. Results for SUPE and QDOS are averaged over 3 seeds.
EnvironmentBC[11][26]IQL[11][26]SUPEQDOS (α=0.2)QDOS (α=0.3)
antmaze-large41±764±1083.0±21.086.0±6.0¯96.0±6.0
antsoccer-arena5±150±227.0±31.073.0±31.0¯80.0±10.0
humanoidmaze-medium8±227±297.0±6.0¯97.0±6.0¯100.0±0.0
kitchen-mixed51.551.085±13.25¯100.00±0.0083.25±28.75
kitchen-partial38.046.383.25±14.5085.75±24.7582.50±30.25
scene-task15±151±4100.0±0.0100.0±0.0100.0±0.0
Fig. 3: Learning curves across environments. QDOS is shown in red (α=0.2) and blue (α=0.3); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.
Fig. 3: Learning curves across environments. QDOS is shown in red (α=0.2) and blue (α=0.3); Baseline (SUPE), ExPLORe, Trajectory Skills, HILP Online, and Diff BC JSRL are shown in black, green, purple, brown, and orange, respectively.
Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.
Fig. 4: Training steps to first goal finding. QDOS (Red) finds the goal significantly faster than baselines, indicating more efficient exploration.
TABLE II: Training steps to reach the goal for the first time (Mean ± Std).
EnvironmentMethodSteps
antmaze-largeSUPE13334±5774
QDOS (α=0.2)𝟏𝟏𝟔𝟔𝟖±𝟗𝟒𝟔𝟓
humanoidmazeSUPE23334±6292
QDOS (α=0.3)𝟏𝟗𝟏𝟔𝟖±𝟖𝟎𝟑𝟔
sceneSUPE5834±1443
QDOS (α=0.2)𝟓𝟎𝟎𝟏±𝟎
Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight αloss leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.
Fig. 5: Experimental results on the cube datasets. Reducing the diversity weight αloss leads to performance closer to the SUPE baseline, indicating that explicit diversity enforcement is less critical for high-quality datasets.
Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen, antmaze, and antsoccer. Orange points represent QDOS (α=0.3 for antsoccer and antmaze, α=0.2 for kitchen), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.
Fig. 6: t-SNE visualization (top) and skill usage timeline (bottom) across kitchen, antmaze, and antsoccer. Orange points represent QDOS (α=0.3 for antsoccer and antmaze, α=0.2 for kitchen), and Blue points represent SUPE. The t-SNE plots indicate that QDOS tends to utilize a broader region of the latent space, suggesting the capture of more diverse behaviors compared to the more restricted latent usage of SUPE.
TABLE III: Hyperparameters for VAE training.
Parameter NameValue
Batch size256
OptimizerAdam
Learning rate3×10−4
GRU Hidden Size256 (antmaze, kitchen)
512 (antsoccer, humanoidmaze, scene)
GRU Layers2 (antmaze, kitchen)
3 (antsoccer, humanoidmaze, scene)
KL Coefficient (OPENβ)0.1 (antmaze, humanoid, kitchen)
0.2 (scene)
Latent Dimension (z)8
Segment Length (H)20
Fig. 7: QDOS skill trajectory visualization for kitchen-mixed. Red crosses mark selected timesteps (t=0,7,14,22) with corresponding scene snapshots.
Fig. 7: QDOS skill trajectory visualization for kitchen-mixed. Red crosses mark selected timesteps (t=0,7,14,22) with corresponding scene snapshots.

왜 중요한가

실제 로봇 데이터는 성공한 시도만 있는 것이 아니라 실패와 잡음이 뒤섞여 있는 경우가 대부분이라, 이런 데이터에서도 안정적으로 쓸모 있는 동작을 뽑아내는 방법은 로봇 학습의 실용성을 크게 높인다. 이는 값비싼 실제 환경 시행착오를 줄이면서도 성능을 끌어올릴 수 있는 실마리가 된다.

이 논문의 용어

  • 오프라인 강화학습 · 환경과 직접 상호작용하지 않고 미리 모아둔 데이터만으로 정책을 학습하는 방법
  • VAE(변분 오토인코더) · 데이터를 압축된 잠재 표현으로 인코딩했다가 다시 복원하도록 학습하는 비지도 학습 모델
  • 어드밴티지(advantage) · 특정 행동이 평균적인 경우보다 얼마나 더 좋은 결과를 내는지를 나타내는 값
  • IQL(Implicit Q-Learning) · 데이터 범위를 벗어난 행동을 querying하지 않고도 가치 함수를 안정적으로 추정하는 오프라인 강화학습 기법
  • 의사 라벨링(pseudo-labeling) · 미리 학습된 인코더로 오프라인 데이터 구간에 추정된 동작 라벨과 보상값을 붙이는 과정

논문 원문 초록 (영문)

Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.

저자 · Tanachai Anakewat, Takayuki Osa, Tatsuya Harada

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Tanachai Anakewat et al., arXiv:2608.19684, arxiv-nonexclusive