SafeBranch: Branch-Pair Safety Alignment for Embodied Agents
arXiv:2608.197292026-08-21
로봇 AI가 '왜 안전한지'를 몰라도 되는 사고 정지 문제를, 위험했던 순간을 되감아 두 갈래 선택지로 가르쳐 해결하다
집안일을 시키면 로봇 몸을 가진 AI 에이전트가 임무는 완수하면서도 가스불을 안 끄거나 젖은 손으로 콘센트를 만지는 등 안전 규칙을 어기는 경우가 많다. 연구팀은 에이전트가 스스로 실수한 순간으로 시뮬레이션을 되감아, 같은 순간에 '안전한 선택'과 '위험한 선택'을 나란히 비교하도록 학습시키는 SafeBranch라는 방법을 만들었다. 그 결과 처음 보는 물건이 등장하는 환경에서 학습 전보다 안전하게 임무를 성공하는 비율이 약 10배로 늘었다.
무엇을 했나
문제 정의: 안전한 행동만 따라하게 시키면 로봇이 맥락 없이 안전 동작을 반복하고, 안전한 궤적과 위험한 궤적을 통째로 비교시키면 신호가 여러 단계에 흩어져 로봇이 안전을 지키려고 아예 임무를 회피해버린다.
방법: 로봇이 위험한 행동을 한 시뮬레이션을 사고가 난 바로 그 순간(안전 결정 지점)으로 되돌린 뒤, GPT-4o 같은 평가자가 무엇이 잘못됐는지 알려주고 로봇 스스로 안전한 대안 행동을 다시 생성하게 한다. 이후 평가자의 힌트는 지우고, 같은 순간에 대한 '원래의 위험한 행동'과 '스스로 만든 안전한 행동'만 짝지어 학습 데이터로 만든다.
학습: 이렇게 만든 짝(branch pair) 데이터로 BranchPO라는 학습 방식을 적용하면, 배포 시점에는 별도의 감시 장치나 평가자 없이 로봇 혼자 안전하게 행동한다.
결과: IS-Bench에서 안전하게 임무를 완수하는 비율이 학습 전 0.031에서 0.281로, 낯선 물건이 등장하는 환경에서는 0.048에서 0.469로 상승했고, 다른 시뮬레이터인 SafetyALFRED에서도 위험 감지 정확도가 0.274에서 0.438로 올랐다.
데이터 만드는 과정도 효율적이어서, 같은 시간에 기존 방식보다 약 5.2배 많은 쓸만한 학습 짝을 만들어냈고, 사람 검수로도 품질이 단계마다 좋아짐을 확인했다.
Figure 1: Branch-pair supervision for interactive safety. At a safety-critical step, structuring supervision as two branches that share the same context but differ in the actor’s action makes the safety-determining choice explicit. This branch-pair form isolates the step-level safety signal, allowing the actor to learn where and how to take the safe branch.
Table 1: Qualitative comparison of supervision signals on a hygiene task. Task: put a peach from the fridge onto a soiled plate. Safety requirement: wipe the plate before placing the peach and close the fridge after retrieval. SafeBranch trains with BranchPO on branch pairs, contrasting safe and unsafe actions at the same decision point. Orange marks stalled continuations; red marks unsafe continuations; green marks safety-relevant actions.
Training signal
Trajectory
Result
Imitation Supervision
open fridge → wipe plate → place peach on plate → close fridge → place peach on plate → place peach on plate → … (no Done)
! STALL Performs safe actions, but collapses into an action loop.
Trajectory-level Preference
open fridge → place peach on soiled plate → Done
× UNSAFE Reaches the goal through an unsafe shortcut.
Branch-pair Preference
open fridge → wipe plate → place peach on plate → close fridge → Done
√ SAFE Chooses the local safe action and completes the task.
Figure 2: Overview of SafeBranch. SafeBranch constructs same-state branch pairs by rolling unsafe rollouts back to the safety-critical anchor step and eliciting a repaired output with critic feedback. The feedback is removed before training, and BranchPO aligns the actor on these branch pairs for critic-free deployment.
Table 2: Main results on IS-Bench and the two OOD benchmarks constructed from it (ObjectShift injects distractors; TaskShift substitutes target objects). The first block runs the untrained actor with optional inference-time safety machinery; the second block trains the actor and deploys it critic-free. For each column, bold marks the best and underline the second-best.
In-Distribution
OOD-ObjectShift
OOD-TaskShift
Method
SR
SSR
SRec
SR
SSR
SRec
SR
SSR
SRec
Inference-time / untrained actor
Baseline
0.656
0.031
0.273
0.899
0.051
0.243
0.803
0.048
0.295
Self-Verification
0.607
0.071
0.333
0.693
0.053
0.244
0.651
0.082
0.295
Lookahead
0.219
0.000
0.094
0.244
0.061
0.054
0.133
0.044
0.067
Actor-trained / critic-free deployment
SFT-only
0.594
0.219
0.422
0.714
0.347
0.390
0.755
0.434
0.689
Trajectory DPO
0.656
0.000
0.256
0.613
0.118
0.245
0.748
0.075
0.253
+ success-matched
0.594
0.000
0.333
0.796
0.097
0.309
0.774
0.063
0.237
BranchPO (ours)
0.594
0.281
0.467
0.819
0.355
0.589
0.694
0.469
0.795
Figure 3: Analysis of branch pair construction. (a) Data generation efficiency. Under the same DFS rollout budget, natural best-of-N sampling and critic-guided rollback are evaluated with a fixed GPT-4o judge for usable same-anchor branch pairs; SafeBranch produces such pairs 5.2× faster. (b) Pair quality through filtering. Starting from 753 raw pairs, the SafeBranch pipeline applies judge filtering and cross-temperature deduplication to retain reliable final branch pairs, with human verification showing higher usable-pair rates across stages. (c) Downstream effect of filtering. Training BranchPO on each filtering stage shows that SR, SSR, and SRec improve most after the final filtering stage, indicating that pair quality rather than raw pair count carries the downstream safety signal.
Table 3: Cross-simulator transfer on SafetyALFRED under its native five-category taxonomy. Unsanitary† is pre-solved by the base VLM (∼35%); Fall/Trip Hazard yields no training pairs under the SafeBranch recipe. For each column, bold marks the best.
Method
Appliance Misuse
Property Damage
Unsanitary†
Spoilage
Fall/Trip Hazard
All
Baseline
0.048
0.034
0.630
0.079
0.000
0.274
Self-Verification
0.089
0.159
0.663
0.053
0.000
0.323
Lookahead
0.024
0.028
0.683
0.053
0.000
0.287
BranchPO (ours)
0.202
0.428
0.711
0.079
0.078
0.438
Box 17: Per-risk-type cumulative bar plot, showing for each hazard category how many safety conditions SafeBranch satisfies vs. violates. Three panels stacked horizontally (ID, OOD, All).
Table 4: Pair construction across preference-learning variants. Shared h: y+ and y− share the same anchor. Task success: both branches complete the task.
Method
Pair type
Shared h
Task success
SFT
Imitation
—
—
Trajectory DPO
Contrast
×
×
+ success-matched
Contrast
×
✓
BranchPO (ours)
Contrast
✓
✓
Table 12: SafeBranch data-construction funnel from the 753 extracted preference pairs (608 rollout episodes, 4 temperatures; BeforeBDDL + TermSafety triggers). “Kept” is the fraction kept from the previous stage.
Stage
Count
Kept
Extracted preference pairs
753
—
→ Judge-kept (quality)
675
89.6%
→ Final (after dedup, train pool)
475
70.4%
Table 13: Removing critic feedback (Ours) vs. retaining it (+FB) in the training prompt, evaluated critic-free on the ID split. The two data variants are byte-identical apart from the critic guidance block. SafeBranch denotes the staged SFT→BranchPO pipeline used in the body (Sec. 4); BranchPO-only drops the SFT warm-up.
Method
Variant
SR
SSR
SRec
SFT-only
Ours
0.594
0.219
0.422
+FB
0.714
0.000
0.270
BranchPO-only
Ours
0.656
0.250
0.488
+FB
0.600
0.133
0.409
SafeBranch
Ours
0.594
0.281
0.467
+FB
0.690
0.138
0.425
Table 14: Runtime full-critic (GPT-4o, one critic call per decision step) vs. SafeBranch / BranchPO (critic-free) across IS-Bench and our controlled OOD extensions. The cost column reports the upper-bound number of additional GPT-4o critic calls under a 30-step budget: (32+147+138)×30=9,510. SafeBranch adds no test-time critic calls.
ID
OOD-ObjectShift
OOD-TaskShift
Test-time cost
Method
SR
SSR
SRec
SR
SSR
SRec
SR
SSR
SRec
Extra GPT-4o calls
Full-critic (GPT-4o)
0.656
0.406
0.680
0.762
0.524
0.742
0.723
0.616
0.793
≤9,510
SafeBranch (ours)
0.594
0.281
0.467
0.819
0.355
0.589
0.694
0.469
0.795
0
Table 15: Per-checkpoint training dynamics on the development split (32 tasks, actor-only). Best checkpoint per method (by SSR) in bold; these are the checkpoints used in the main results.
Method
Step
SR
SSR
SFT-only
10
0.656
0.031
20
0.688
0.125
30
0.594
0.219
60
0.094
0.094
90
0.281
0.156
BranchPO-only
30
0.688
0.031
50
0.656
0.188
70
0.594
0.156
90
0.656
0.250
140
0.594
0.188
210
0.594
0.188
SafeBranch
30
0.594
0.281
(SFT→BranchPO)
50
0.581
0.161
70
0.552
0.138
90
0.633
0.100
110
0.517
0.138
130
0.433
0.100
150
0.310
0.034
Table 16: SafeBranch per-risk-type safety recall on ID and OOD splits. Safe is the number of satisfied safety conditions, and Total is the number of required safety conditions. SRec is computed as Safe / Total. The All columns aggregate ID and OOD. SafeBranch checkpoint = SFT→BranchPO at step 30 (see Table 15).
ID
OOD
All
Risk type
SRec
Safe
Total
SRec
Safe
Total
SRec
Safe
Total
Collision/Tripping Hazard
0.429
9
21
0.560
190
339
0.552
199
360
Fire Hazard
0.167
1
6
0.233
7
30
0.222
8
36
Food Contamination
0.143
1
7
0.341
14
41
0.312
15
48
Chemical Hazard
0.750
6
8
0.610
86
141
0.617
92
149
Water Spill Damage
1.000
1
1
0.000
0
12
0.077
1
13
Falling Object Hazard
0.400
2
5
0.784
29
37
0.738
31
42
Electrical Shock
0.000
0
2
0.367
11
30
0.344
11
32
All
0.400
20
50
0.535
337
630
0.525
357
680
왜 중요한가
가정용 로봇이나 AI 에이전트가 실제 환경에서 작업할 때, 임무 성공만 보고 배포하면 안전사고로 이어질 수 있다는 우려를 이 연구는 구체적인 학습 데이터 설계로 줄여준다. 매 단계 외부 감시 장치를 붙이는 대신 로봇 자체를 안전하게 훈련시키므로, 배포 비용과 반응 속도 면에서도 실용적인 대안이 될 수 있다.
이 논문의 용어
VLM 기반 에이전트 · 카메라로 본 장면과 언어 지시를 함께 이해해 행동을 결정하는 AI 로봇 시스템
안전 결정 지점(safety-critical step) · 로봇의 선택 하나로 이후 결과가 안전해지거나 위험해지는 갈림길이 되는 순간
branch pair(가지 짝) · 같은 순간, 같은 상황에서 안전한 행동과 위험한 행동만 다르게 짝지어 놓은 학습 데이터
환경 되감기(rollback) · 시뮬레이션을 사고가 나기 전 특정 순간으로 되돌리는 기능
BranchPO · 안전한 행동을 더 선호하도록 만드는, 짝 데이터 전용으로 설계된 학습 방식
critic-free 배포 · 실제 사용 시에는 별도 감시·평가 장치 없이 로봇 혼자 판단해 행동하는 방식
논문 원문 초록 (영문)
Vision-language-model-based embodied agents can complete instructed tasks but often violate safety constraints in the process, a problem recently framed as interactive safety. Training such agents to act safely is difficult, since safety and task success are distinct objectives, and safety arises only at a small number of safety-critical steps within a trajectory. Standard supervision is insufficient: imitating safe trajectories teaches behavior without explaining why it is safe, and contrasting arbitrary safe and unsafe trajectories mixes the safety signal with unrelated differences. We propose SafeBranch, a framework that aligns an embodied actor on safety through branch pairs constructed from the actor's own unsafe rollouts via environment rollback. SafeBranch rolls each unsafe rollout back to the safety-critical step that caused the violation, queries the actor for a safe alternative, and pairs the original action with the alternative so that the two branches differ only at that step. The trained actor acts safely at deployment with no critic in the loop. On IS-Bench, SafetyALFRED, and out-of-distribution variants with unseen tasks and objects, it handles safety reliably without sacrificing task success, achieving roughly ten times more safe successes than the untrained baseline on the unseen-object variant.
