타오라이브 라이브방송 AI 아바타가 진행 규칙이 계속 바뀌어도 느려지거나 헛소리하지 않게 훈련하는 법
arXiv:2608.157632026-08-21
Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report
타오라이브 라이브방송 AI 아바타가 진행 규칙이 계속 바뀌어도 느려지거나 헛소리하지 않게 훈련하는 법
라이브 커머스 AI 아바타는 상품 질문에 실시간으로 답하면서도 운영팀이 수시로 바꾸는 판매 전략을 즉시 반영해야 한다. 이 논문은 모델 가중치를 그대로 두고 스킬·프롬프트·도구·후크만 바꿔서 시스템을 진화시키는 Harness 구조와, 그 변화에도 작은 모델이 흔들리지 않도록 훈련하는 HAT(Harness-Aware Training) 기법을 제안한다. 실제 타오바오 라이브 서비스에 적용해 지연시간 기준을 지키면서 응답 품질도 크게 개선했다.
METAL LAB 해설 도표
METAL LAB이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.
무엇을 했나
큰 모델은 낯선 설정도 스스로 잘 이해하지만 응답이 너무 느리고, 작은 모델은 빠르지만 고정된 설정에만 맞춰 훈련되면 설정이 바뀔 때 제대로 대응하지 못하는 딜레마를 다룬다.
Harness라는 이름의 실행 환경(스킬, 도구 목록, 프롬프트 구조, 검증 후크)을 모델 가중치와 분리해 두고, 이 환경만 바꿔서 서비스를 개선하는 방식을 'Harness Evolution'이라 부른다.
훈련용 데이터에서 스킬 이름, 도구 이름, 프롬프트 구조, 후크 동작을 의도적으로 다양하게 바꿔주는 HSA(Harness-State Augmentation) 기법으로 모델이 특정 이름이나 형식을 외우는 대신 실제 기능을 이해하게 만든다.
훈련은 HSA를 적용한 지도학습(HSA-SFT), 일반 능력을 되살리는 온폴리시 증류(General OPD), 시뮬레이터에서 강화학습으로 다양한 환경에 적응시키는 HSA-RL 세 단계로 진행된다.
실제 라이브방송 QA에서 94.8점(기본 모델 80.3점, 최강 범용 LLM 93.0점 대비), 설정이 바뀐 Harness-Variant QA에서 94.6점(기본 75.4점)을 기록했고, 단일 GPU에서 P50 3.4초, P95 8.1초의 지연시간을 달성했다.
Figure 1: Application flow and the two independent update paths. Harness Evolution edits agent runtime modules while keeping model weights fixed; Harness-Aware Training (HAT) provides a well-performing, low-latency, and robust model. Both paths meet inside the same production runtime.
Table 1: Approximate intent mix in the available internal traffic summary.
Scenario
Proportion
Product Q&A
∼46%
Casual chat and engagement
∼19%
Clarification follow-up
∼16%
After-sales handling
∼7%
Discount and promotion inquiry
∼4%
Presentation-order adjustment
∼2%
FAQ-based reply
∼2%
Silent refusal of irrelevant content
<1%
Figure 2: Harness Evolution with a fixed DeepSeek-V4-flash on dev-set (n=482). The process is non-monotonic: Evolution 2 is the selected development checkpoint, while later long-tail edits introduce regressions.
Table 2: Evaluation sets and sources. The sets cover real live-stream reply quality, in-family Harness variation, synthetic tool and prompt robustness, general instruction following, and actual deployment latency.
Set
Size
Source
Evidentiary role
𝒯1
978
Real live-room
Primary industrial quality (live-stream reply).
𝒯2
978
Real live-room, aug. Harness
In-family robustness to new Harness contexts.
𝒯3
2023
Synthetic avatar scenarios
In-domain transfer under broader tools/prompts.
𝒯4
541
Public IFEval
General instruction following (official evaluator).
𝒞judge
482
Real live-room, human-labeled
Judge calibration, not a policy test set. Also dev-set for harness evolving.
𝒟perf
110
Deployment replay
End-to-end deployment latency.
Figure 3: Simplified Harness Agent runtime control flow. Prompt rules and dynamically loaded Skills assemble the model context; lifecycle Hooks validate inference, tool use, and stopping. A failed stop gate starts another round, while a passed gate produces the final response and structured trajectory. Per-call fallback and trajectory-level recovery protect different failure scopes.
Table 4: Training-pipeline ablation. Each row adds one stage to the previous checkpoint; ↑/↓ denotes a descriptive score change from the row above and does not include retraining variance.
Configuration
𝒯1
𝒯2
Tool
Prompt
IFE-P
IFE-I
Non-augmented pathway (single-run trajectory)
Base
80.3
75.4
69.5
72.8
81.5
87.7
+SFT
89.5↑9.2
88.2↑12.8
82.0↑12.5
68.2↓4.6
73.8↓7.7
82.4↓5.3
+General OPD
89.5
89.1↑0.9
84.3↑2.3
72.6↑4.4
82.3↑8.5
87.9↑5.5
+RL
95.1↑5.6
94.4↑5.3
83.7↓0.6
66.7↓5.9
82.7↑0.4
87.8↓0.1
Augmented pathway (single-run trajectory)
+Aug-SFT
90.0↑9.7
90.2↑14.8
85.2↑15.7
77.3↑4.5
81.7↑0.2
87.4↓0.3
+General OPD
90.9↑0.9
90.9↑0.7
85.6↑0.4
77.1↓0.2
81.9↑0.2
88.0↑0.6
+RL
94.1↑3.2
93.7↑2.8
84.3↓1.3
78.7↑1.6
83.0↑1.1
88.2↑0.2
+AUG Env (Ours)
94.8↑0.7
94.6↑0.9
84.0↓0.3
77.6↓1.1
83.5↑0.5
88.7↑0.5
Figure 4: Overview of HAT. (a) HSA constructs diversified Harness states; HSA-SFT followed by General OPD produces the shared initialization. (b) Controlled comparison of Agentic RL and Agentic OPD under the original Harness and HSA environments. Every variant starts from the HSA-SFT + General OPD result, and each OPD variant receives its paired same-environment RL teacher. The selected Agentic RL policy under HSA is HSA-RL.
Table 5: Single-run comparison of Agentic optimization variants. All configurations share the same Aug-SFT + General OPD initialization; ⋆ marks the deployment choice rather than statistical superiority.
Method
Environment
𝒯1
IFE-P
PR-AVG
Agentic RL
Original
94.1
83.0
78.7
Agentic RL
Augmented
94.8⋆
83.5⋆
77.6
Agentic OPD
Original
94.0
82.8
73.9
Agentic OPD
Augmented
93.9
81.7
75.5
Paired contrasts
Δ(RL−OPD)
Original
+0.1
+0.2
+4.8
Δ(RL−OPD)
Augmented
+0.9
+1.8
+2.1
Δ(Aug−Orig)
RL
+0.7
+0.5
−1.1
Δ(Aug−Orig)
OPD
−0.1
−1.1
+1.6
Figure 5: Checkpoint trajectories across four training configurations formed by crossing HSA at the SFT and RL stages, evaluated on Accuracy, Effectiveness, Skill Selection, and Tool Rationality. In these trajectories, HSA-SFT is associated with higher reply-quality rewards, while HSA-RL is associated with stronger agentic-behavior rewards, especially Skill Selection and Tool Rationality; combining HSA-SFT and HSA-RL gives the most balanced late-stage reward profile.
Table 6: Single-run isolation of data augmentation at SFT and RL stages. The reported differences are descriptive and do not include retraining variance.
SFT Stage
RL Stage
𝒯1 AVG
IFE-P
ΔIFE
Standard
Standard RL
95.1
71.3
–
Standard
Augmented RL
94.9
72.3
+1.0
Augmented
Standard RL
95.4
80.7
+9.4
Augmented
Augmented RL
93.3
81.3
+10.0
Augmented SFT: +9.4 IFE-P; augmented RL: +1.0.
Figure 6: Single-run CoT-length diagnostics for the selected HSA-RL trajectory. Faint lines show recorded step values and bold lines show 25-step trailing means. Panel (a) separates tool-call and final-reply CoT lengths and marks the no-penalty (L≤100), linear (100<L<200), and saturated (L≥200) regions. Panel (b) shows the logged score q(L)∈[0,1] and the equivalent raw reward contribution −0.1q(L). The figure is a training diagnostic, not a causal ablation of latency or quality.
Table 7: Controlled low-concurrency deployment replay. Latencies are seconds; Decode is the arithmetic mean of client-observed completion tokens/s over model calls. It is length-sensitive and is not an intrinsic cross-checkpoint kernel-speed measure. TTFT is the first call’s client-observed P95 and includes network and queueing. Each row contains 100 measured Agent requests after 10 warm-up cases. API rows are operational references, not same-hardware model comparisons.
Configuration
MTP
C
Wall P50
Wall P95
TTFT P95
Decode
≤15s
Success
Vendor API routes
DeepSeek-V4-flash
–
1
11.210
21.191
2.067
103.44
71%
99%
DeepSeek-V4-flash
–
2
11.976
26.278
1.291
97.01
71%
98%
DeepSeek-V4-pro
–
1
14.312
28.882
1.312
53.53
61%
99%
DeepSeek-V4-pro
–
2
13.752
23.393
1.248
55.04
60%
99%
Qwen3.6-35B-A3B · one H20
Base
Off
1
4.101
10.172
0.508
195.42
100%
100%
Base
Off
2
4.560
11.504
0.623
138.81
97%
100%
Base
On
1
3.648
9.805
0.502
215.95
99%
100%
Base
On
2
4.806
10.251
0.698
165.74
97%
100%
Task-trained checkpoint (Ours) · one H20
Ours
Off
1
4.176
8.98
0.499
160.30
100%
100%
Ours
Off
2
4.936
10.074
0.692
129.22
100%
100%
Ours
On
1
3.407
8.114
0.553
271.40
100%
100%
Ours
On
2
4.479
9.047
0.754
196.15
100%
100%
Figure 7: Judge alignment to a human-labeled calibration cohort (n=482). Accuracy and Effectiveness agreement improve through evidence-tool, rubric, and voting revisions; the cohort evaluates the scoring instrument rather than the policy.
Table 8: Harness-Aware Benchmark held-out test results before and after Harness edits. Counts are total detected errors; error reduction is computed relative to the before-edit count.
Model
Before
After
Error reduction
Naive SFT
149
122
18.1%
Ours
238
115
51.7%
Figure 8: Single-run standalone MTP-loss trajectories for a task-trained policy with a transplanted and subsequently post-trained NextN head, and for the base model with its native factory-trained head. Faint lines show step-level loss; bold lines show a 25-step trailing mean. The runs have different lengths and provide adaptation diagnostics only: they do not compare convergence speed, training stability, sample efficiency, or compute efficiency.
Table 9: Human blind-test preference distribution on 100 paired real live-streaming requests.
Preference
Count
Share
Harness better
35
35.0%
Tie
64
64.0%
ReAct better
1
1.0%
Table 10: Attribution of the 35 Harness-preferred examples. Categories are assigned from annotator rationales after the blind decision.
Attribution category
Count
Share
More accurate input understanding
12
34.3%
More reliable output
8
22.9%
More appropriate scenario behavior
8
22.9%
Higher response quality
5
14.3%
More appropriate tool use
2
5.7%
Table 11: Preliminary seven-day online A/B point estimates in Taobao Live’s production digital-avatar business. Metrics are relative changes of the Harness system against the ReAct control.
Metric
Harness vs. ReAct
Cumulative confirmed-receipt GMV per bucketed user
+5.54%
Completed orders per bucketed user
+0.80%
Table 12: Working inventory of built-in and MCP-discovered runtime interfaces.
Retrieve campaign conditions and buyer-side resources from the external marketing service.
MCP calculation
calculate_optimal_promotion, calculate_promotion
Compute eligible promotional combinations and resulting prices.
Table 13: Working Skill inventory and routing responsibilities.
Reply Skill
Response responsibility
Strategy Skill
Tool-chain responsibility
item_qa
Product attributes, specifications, comparison, price, and recommendation.
tool_strategy_benefit
Entitlement, price, and promotion retrieval.
chillchat
Non-product conversation and engagement.
tool_strategy_compare
Multi-product comparison and retrieval.
aftersale
Returns, exchanges, complaints, and other after-sales requests.
tool_strategy_multi
Aggregation and orchestration for multiple comments.
clarification
Follow-up questions when available context is insufficient.
tool_strategy_switch
Presentation-order adjustment.
faq_reply
Responses grounded in configured FAQ entries.
general_discount
Room-level promotion retrieval and response composition.
refusal
Declines for inappropriate or unsupported requests.
change_order
Confirmation of presentation-order changes.
thanks
Responses to gratitude and positive feedback.
greet
Welcome messages for viewers entering the room.
Table 14: Fixed-policy Harness Evolution record. Values come from six direct evaluation summaries on the same 482-item dev-set. “Selected” is an engineering early-stop decision.
Stage
Harness changes
Acc.
Eff.
Engineering diagnosis
ReAct
Conventional reasoning–action loop without the modular Harness.
80.33
84.58
System baseline.
Harness base
Initial manually maintained Skills and modular runtime.
82.40
87.16
Establishes the editable Harness.
Evolution 1
Add refusal tool and stop-loop Hook; rewrite refusal, chat, and after-sales Skills; add four global constraints.
92.13
84.16
Accuracy rises sharply; internal diagnosis attributes Effectiveness loss to over-triggered refusal.
Evolution 2
Add seven whitelist exclusions before refusal; map explanation triggers and require the relevant tool call.
92.55
92.75
Selected; restores Effectiveness while retaining Accuracy.
Long-tail rules interact and regress both metrics.
Evolution 4
Relax length, tool-trigger, parameter, transaction-intent, and attribution restrictions.
91.51
89.96
Partial rollback does not recover Effectiveness.
Table 15: Independent paired-bootstrap stability check. Differences are reported in percentage points; intervals are 95% percentile bootstrap intervals over evaluation items.
Contrast / dimension
Δ [95% CI]
Pboot(Δ≤0)
IFEval prompt-level accuracy
Naive SFT − Base
−6.47[−10.72,−2.22]
0.9992
Ours − Base
+2.03[−1.11,+5.18]
0.1134
Ours − Naive SFT
+8.50[+4.44,+12.38]
0.0000
Prompt Robustness
Ours − Naive SFT
+8.46[+5.27,+11.66]
<0.0001
Naive SFT − Base
−3.22[−6.59,+0.21]
0.9661
𝒯1 Live-Stream QA: Ours − Base
Accuracy
+6.95[+4.60,+9.41]
<0.0001
Effectiveness
+30.88[+27.76,+33.90]
<0.0001
AVG
+18.92[+17.00,+20.81]
<0.0001
Table 16: Same-checkpoint 𝒯1 quality check with MTP Off and On (n=978). Both rows use the same Final Evaluation Judge. AVG is the arithmetic mean of Accuracy and Effectiveness.
Serving mode
Accuracy
Effectiveness
AVG
MTP Off
95.8
93.8
94.8
MTP On
96.22
94.22
95.22
Table 17: MTP trade-off across concurrency. Arrows show Off → On. Decode TPS is client-observed mean per call; Wall and TTFT are P95 seconds. C=1, 2, and 8 are single runs. C=4 values are means of three run-level statistics, not pooled percentiles.
Checkpoint
C
Decode TPS
Speedup
Wall P95 (s)
TTFT P95 (s)
≤15s
Qwen3.6 Base
1
195.42→215.95
1.11×
10.172→9.805
0.508→0.502
100→99%
2
138.81→165.74
1.19×
11.504→10.251
0.623→0.698
97→97%
4
105.04→113.25
1.08×
13.408→16.266
0.643→1.088
97.3→92.3%
8
70.28→70.19
1.00×
17.848→26.136
0.953→1.519
90→79%
Ours
1
160.30→271.40
1.69×
8.98→8.114
0.499→0.553
100→100%
2
129.22→196.15
1.52×
10.074→9.047
0.692→0.754
100→100%
4
99.65→126.64
1.27×
10.869→12.460
0.692→1.349
99.3→98.0%
8
59.01→70.22
1.19×
18.342→20.298
0.900→2.574
91→75%
Table 18: C=1 workload shape. Values are means over successful complete-Agent requests or their constituent model calls.
Route / checkpoint
MTP
Calls/req
Input/call
Output/call
DeepSeek-V4-flash API
–
4.04
6,052
146
DeepSeek-V4-pro API
–
3.67
5,537
119
Qwen3.6 Base
Off
3.23
5,622
115
Qwen3.6 Base
On
3.08
5,571
109
Ours
Off
3.04
5,679
106
Ours
On
3.12
5,786
105
Table 19: MTP serving configuration used for the reported quality check and deployment measurements.
Setting
Engine argument
Value
Speculative algorithm
--speculative-algo
NEXTN
Speculative steps
--speculative-num-steps
3
EAGLE top-k
--speculative-eagle-topk
1
Draft tokens
--speculative-num-draft-tokens
4
Single-token acceptance threshold
--speculative-accept-threshold-single
0.5
Accumulated acceptance threshold
--speculative-accept-threshold-acc
0.7
Table 20: SGLang log-sample diagnostics for MTP-On serving. Triples are mean/P50/P95; Queue is P95/max.
Checkpoint
Running req
Queue
Accept len
Accept rate
Qwen3.6 Base
2.03/1/8
0/0
2.94/2.92/3.58
.735/.730/.890
Ours
2.03/1/7
0/0
3.12/3.12/3.62
.780/.780/.910
실제로 확인된 결과
실시간 라이브방송 QA에서 HAT 모델은 94.8점을 기록해 기본 모델 80.3점과 가장 강한 범용 LLM 93.0점을 넘어섰다.
Harness 설정이 바뀐 상황을 테스트한 Harness-Variant QA에서 HAT는 94.6점으로 기본 모델 75.4점보다 크게 높았다.
고정된 Harness로만 지도학습한 모델(Fixed-Harness SFT)은 일반 지시 이행 능력 지표인 IFEval이 기본 모델보다 7.7점 떨어졌지만, HAT는 이런 퇴화 없이 83.5점을 유지했다.
단일 NVIDIA H20 GPU에서 최적화를 적용한 상태로 P50 지연시간 3.4초, P95 지연시간 8.1초를 달성했다.
타오바오 라이브의 실제 디지털 아바타 서비스에 배포한 온라인 A/B 테스트에서 거래액(GMV)과 상품 페이지 조회수가 긍정적으로 나타났다.
어디에 쓸 수 있나
운영 규칙이나 판매 전략이 자주 바뀌는 라이브 커머스, 고객상담 등에서 모델 재훈련 없이 시스템 동작을 조정해야 하는 서비스
여러 도구와 스킬을 동적으로 로드하는 에이전트형 챗봇에서 도구 이름이나 프롬프트 형식이 바뀔 때 성능 저하를 줄이려는 시스템 설계
지연시간 제약이 엄격한 실시간 응답 서비스에서 소형 모델의 견고성을 높이려는 훈련 파이프라인 설계
한계와 남은 검증
논문에 보고된 성능은 타오바오 라이브의 특정 상품·라이브방송 도메인과 Qwen3.6-35B-A3B 규모 모델, 단일 H20 GPU 환경에 한정된 결과로 다른 도메인이나 모델 크기에 그대로 적용될지는 검증되지 않았다.
지연시간 측정치는 소규모 동시성(C=1~2) 조건에서 얻은 것이며, C=4~8 같은 높은 동시성에서는 MTP 가속 효과가 줄어드는 것으로 나타나 실제 대규모 트래픽에서의 성능은 별도 검증이 필요하다.
MTP 드래프트 헤드 이식 실험은 단일 실행(run)에 기반한 진단적 관찰로, 여러 시드에 걸친 수렴 속도나 학습 안정성을 비교하는 근거로는 사용할 수 없다고 저자들이 명시했다.
온라인 A/B 테스트는 7일간의 예비 결과로 표현되어 있어, 장기적이고 안정적인 프로덕션 효과인지는 추가 확인이 필요하다.
왜 중요한가
라이브 커머스처럼 빠른 응답과 잦은 정책 변경이 동시에 필요한 서비스에서는 큰 모델의 정확성과 작은 모델의 속도를 둘 다 챙기기 어려운데, 이 연구는 실행 환경과 모델을 분리하고 훈련 데이터를 다양화하는 방식으로 그 절충점을 실제 서비스에서 검증했다는 점이 중요하다.
이 논문의 용어
Harness · 모델 가중치와 분리되어 독립적으로 업데이트 가능한 스킬·프롬프트·도구·후크로 이루어진 실행 환경 설정
HSA (Harness-State Augmentation) · 스킬 이름, 도구 정의, 프롬프트 구조, 후크 동작을 의도적으로 다양하게 바꿔서 훈련 데이터에 넣는 기법
HAT (Harness-Aware Training) · HSA-SFT, General OPD, HSA-RL 세 단계로 구성된, 환경 변화에 강한 소형 모델을 만드는 훈련 파이프라인
GRPO/GDPO · 여러 응답 후보를 그룹으로 비교해 강화학습 보상을 계산하는 최적화 기법들
MTP(멀티토큰 예측) 드래프트 헤드 · 다음 토큰을 미리 여러 개 예측해 추론 속도를 높이는 보조 모듈
![Figure 6: Single-run CoT-length diagnostics for the selected HSA-RL trajectory. Faint lines show recorded step values and bold lines show 25-step trailing means. Panel (a) separates tool-call and final-reply CoT lengths and marks the no-penalty (L≤100), linear (100<L<200), and saturated (L≥200) regions. Panel (b) shows the logged score q(L)∈[0,1] and the equivalent raw reward contribution −0.1q(L). The figure is a training diagnostic, not a causal ablation of latency or quality.](https://media.metallab.ai/papers/2608.15763/f5.svg)
