Figure 1: Application flow and the two independent update paths. Harness Evolution edits agent runtime modules while keeping model weights fixed; Harness-Aware Training (HAT) provides a well-performing, low-latency, and robust model. Both paths meet inside the same production runtime.
Table 1: Approximate intent mix in the available internal traffic summary.
Scenario
Proportion
Product Q&A
∼46%
Casual chat and engagement
∼19%
Clarification follow-up
∼16%
After-sales handling
∼7%
Discount and promotion inquiry
∼4%
Presentation-order adjustment
∼2%
FAQ-based reply
∼2%
Silent refusal of irrelevant content
<1%
Figure 2: Harness Evolution with a fixed DeepSeek-V4-flash on dev-set (n=482). The process is non-monotonic: Evolution 2 is the selected development checkpoint, while later long-tail edits introduce regressions.
Table 2: Evaluation sets and sources. The sets cover real live-stream reply quality, in-family Harness variation, synthetic tool and prompt robustness, general instruction following, and actual deployment latency.
Set
Size
Source
Evidentiary role
𝒯1
978
Real live-room
Primary industrial quality (live-stream reply).
𝒯2
978
Real live-room, aug. Harness
In-family robustness to new Harness contexts.
𝒯3
2023
Synthetic avatar scenarios
In-domain transfer under broader tools/prompts.
𝒯4
541
Public IFEval
General instruction following (official evaluator).
𝒞judge
482
Real live-room, human-labeled
Judge calibration, not a policy test set. Also dev-set for harness evolving.
𝒟perf
110
Deployment replay
End-to-end deployment latency.
Figure 3: Simplified Harness Agent runtime control flow. Prompt rules and dynamically loaded Skills assemble the model context; lifecycle Hooks validate inference, tool use, and stopping. A failed stop gate starts another round, while a passed gate produces the final response and structured trajectory. Per-call fallback and trajectory-level recovery protect different failure scopes.
Table 4: Training-pipeline ablation. Each row adds one stage to the previous checkpoint; ↑/↓ denotes a descriptive score change from the row above and does not include retraining variance.
Configuration
𝒯1
𝒯2
Tool
Prompt
IFE-P
IFE-I
Non-augmented pathway (single-run trajectory)
Base
80.3
75.4
69.5
72.8
81.5
87.7
+SFT
89.5↑9.2
88.2↑12.8
82.0↑12.5
68.2↓4.6
73.8↓7.7
82.4↓5.3
+General OPD
89.5
89.1↑0.9
84.3↑2.3
72.6↑4.4
82.3↑8.5
87.9↑5.5
+RL
95.1↑5.6
94.4↑5.3
83.7↓0.6
66.7↓5.9
82.7↑0.4
87.8↓0.1
Augmented pathway (single-run trajectory)
+Aug-SFT
90.0↑9.7
90.2↑14.8
85.2↑15.7
77.3↑4.5
81.7↑0.2
87.4↓0.3
+General OPD
90.9↑0.9
90.9↑0.7
85.6↑0.4
77.1↓0.2
81.9↑0.2
88.0↑0.6
+RL
94.1↑3.2
93.7↑2.8
84.3↓1.3
78.7↑1.6
83.0↑1.1
88.2↑0.2
+AUG Env (Ours)
94.8↑0.7
94.6↑0.9
84.0↓0.3
77.6↓1.1
83.5↑0.5
88.7↑0.5
Figure 4: Overview of HAT. (a) HSA constructs diversified Harness states; HSA-SFT followed by General OPD produces the shared initialization. (b) Controlled comparison of Agentic RL and Agentic OPD under the original Harness and HSA environments. Every variant starts from the HSA-SFT + General OPD result, and each OPD variant receives its paired same-environment RL teacher. The selected Agentic RL policy under HSA is HSA-RL.
Table 5: Single-run comparison of Agentic optimization variants. All configurations share the same Aug-SFT + General OPD initialization; ⋆ marks the deployment choice rather than statistical superiority.
Method
Environment
𝒯1
IFE-P
PR-AVG
Agentic RL
Original
94.1
83.0
78.7
Agentic RL
Augmented
94.8⋆
83.5⋆
77.6
Agentic OPD
Original
94.0
82.8
73.9
Agentic OPD
Augmented
93.9
81.7
75.5
Paired contrasts
Δ(RL−OPD)
Original
+0.1
+0.2
+4.8
Δ(RL−OPD)
Augmented
+0.9
+1.8
+2.1
Δ(Aug−Orig)
RL
+0.7
+0.5
−1.1
Δ(Aug−Orig)
OPD
−0.1
−1.1
+1.6
Figure 5: Checkpoint trajectories across four training configurations formed by crossing HSA at the SFT and RL stages, evaluated on Accuracy, Effectiveness, Skill Selection, and Tool Rationality. In these trajectories, HSA-SFT is associated with higher reply-quality rewards, while HSA-RL is associated with stronger agentic-behavior rewards, especially Skill Selection and Tool Rationality; combining HSA-SFT and HSA-RL gives the most balanced late-stage reward profile.
Table 6: Single-run isolation of data augmentation at SFT and RL stages. The reported differences are descriptive and do not include retraining variance.
SFT Stage
RL Stage
𝒯1 AVG
IFE-P
ΔIFE
Standard
Standard RL
95.1
71.3
–
Standard
Augmented RL
94.9
72.3
+1.0
Augmented
Standard RL
95.4
80.7
+9.4
Augmented
Augmented RL
93.3
81.3
+10.0
Augmented SFT: +9.4 IFE-P; augmented RL: +1.0.
Figure 6: Single-run CoT-length diagnostics for the selected HSA-RL trajectory. Faint lines show recorded step values and bold lines show 25-step trailing means. Panel (a) separates tool-call and final-reply CoT lengths and marks the no-penalty (L≤100), linear (100<L<200), and saturated (L≥200) regions. Panel (b) shows the logged score q(L)∈[0,1] and the equivalent raw reward contribution −0.1q(L). The figure is a training diagnostic, not a causal ablation of latency or quality.
Table 7: Controlled low-concurrency deployment replay. Latencies are seconds; Decode is the arithmetic mean of client-observed completion tokens/s over model calls. It is length-sensitive and is not an intrinsic cross-checkpoint kernel-speed measure. TTFT is the first call’s client-observed P95 and includes network and queueing. Each row contains 100 measured Agent requests after 10 warm-up cases. API rows are operational references, not same-hardware model comparisons.
Configuration
MTP
C
Wall P50
Wall P95
TTFT P95
Decode
≤15s
Success
Vendor API routes
DeepSeek-V4-flash
–
1
11.210
21.191
2.067
103.44
71%
99%
DeepSeek-V4-flash
–
2
11.976
26.278
1.291
97.01
71%
98%
DeepSeek-V4-pro
–
1
14.312
28.882
1.312
53.53
61%
99%
DeepSeek-V4-pro
–
2
13.752
23.393
1.248
55.04
60%
99%
Qwen3.6-35B-A3B · one H20
Base
Off
1
4.101
10.172
0.508
195.42
100%
100%
Base
Off
2
4.560
11.504
0.623
138.81
97%
100%
Base
On
1
3.648
9.805
0.502
215.95
99%
100%
Base
On
2
4.806
10.251
0.698
165.74
97%
100%
Task-trained checkpoint (Ours) · one H20
Ours
Off
1
4.176
8.98
0.499
160.30
100%
100%
Ours
Off
2
4.936
10.074
0.692
129.22
100%
100%
Ours
On
1
3.407
8.114
0.553
271.40
100%
100%
Ours
On
2
4.479
9.047
0.754
196.15
100%
100%
Figure 7: Judge alignment to a human-labeled calibration cohort (n=482). Accuracy and Effectiveness agreement improve through evidence-tool, rubric, and voting revisions; the cohort evaluates the scoring instrument rather than the policy.
Table 8: Harness-Aware Benchmark held-out test results before and after Harness edits. Counts are total detected errors; error reduction is computed relative to the before-edit count.
Model
Before
After
Error reduction
Naive SFT
149
122
18.1%
Ours
238
115
51.7%
Figure 8: Single-run standalone MTP-loss trajectories for a task-trained policy with a transplanted and subsequently post-trained NextN head, and for the base model with its native factory-trained head. Faint lines show step-level loss; bold lines show a 25-step trailing mean. The runs have different lengths and provide adaptation diagnostics only: they do not compare convergence speed, training stability, sample efficiency, or compute efficiency.
Table 9: Human blind-test preference distribution on 100 paired real live-streaming requests.
Preference
Count
Share
Harness better
35
35.0%
Tie
64
64.0%
ReAct better
1
1.0%
Table 10: Attribution of the 35 Harness-preferred examples. Categories are assigned from annotator rationales after the blind decision.
Attribution category
Count
Share
More accurate input understanding
12
34.3%
More reliable output
8
22.9%
More appropriate scenario behavior
8
22.9%
Higher response quality
5
14.3%
More appropriate tool use
2
5.7%
Table 11: Preliminary seven-day online A/B point estimates in Taobao Live’s production digital-avatar business. Metrics are relative changes of the Harness system against the ReAct control.
Metric
Harness vs. ReAct
Cumulative confirmed-receipt GMV per bucketed user
+5.54%
Completed orders per bucketed user
+0.80%
Table 12: Working inventory of built-in and MCP-discovered runtime interfaces.
Retrieve campaign conditions and buyer-side resources from the external marketing service.
MCP calculation
calculate_optimal_promotion, calculate_promotion
Compute eligible promotional combinations and resulting prices.
Table 13: Working Skill inventory and routing responsibilities.
Reply Skill
Response responsibility
Strategy Skill
Tool-chain responsibility
item_qa
Product attributes, specifications, comparison, price, and recommendation.
tool_strategy_benefit
Entitlement, price, and promotion retrieval.
chillchat
Non-product conversation and engagement.
tool_strategy_compare
Multi-product comparison and retrieval.
aftersale
Returns, exchanges, complaints, and other after-sales requests.
tool_strategy_multi
Aggregation and orchestration for multiple comments.
clarification
Follow-up questions when available context is insufficient.
tool_strategy_switch
Presentation-order adjustment.
faq_reply
Responses grounded in configured FAQ entries.
general_discount
Room-level promotion retrieval and response composition.
refusal
Declines for inappropriate or unsupported requests.
change_order
Confirmation of presentation-order changes.
thanks
Responses to gratitude and positive feedback.
greet
Welcome messages for viewers entering the room.
Table 14: Fixed-policy Harness Evolution record. Values come from six direct evaluation summaries on the same 482-item dev-set. “Selected” is an engineering early-stop decision.
Stage
Harness changes
Acc.
Eff.
Engineering diagnosis
ReAct
Conventional reasoning–action loop without the modular Harness.
80.33
84.58
System baseline.
Harness base
Initial manually maintained Skills and modular runtime.
82.40
87.16
Establishes the editable Harness.
Evolution 1
Add refusal tool and stop-loop Hook; rewrite refusal, chat, and after-sales Skills; add four global constraints.
92.13
84.16
Accuracy rises sharply; internal diagnosis attributes Effectiveness loss to over-triggered refusal.
Evolution 2
Add seven whitelist exclusions before refusal; map explanation triggers and require the relevant tool call.
92.55
92.75
Selected; restores Effectiveness while retaining Accuracy.
Long-tail rules interact and regress both metrics.
Evolution 4
Relax length, tool-trigger, parameter, transaction-intent, and attribution restrictions.
91.51
89.96
Partial rollback does not recover Effectiveness.
Table 15: Independent paired-bootstrap stability check. Differences are reported in percentage points; intervals are 95% percentile bootstrap intervals over evaluation items.
Contrast / dimension
Δ [95% CI]
Pboot(Δ≤0)
IFEval prompt-level accuracy
Naive SFT − Base
−6.47[−10.72,−2.22]
0.9992
Ours − Base
+2.03[−1.11,+5.18]
0.1134
Ours − Naive SFT
+8.50[+4.44,+12.38]
0.0000
Prompt Robustness
Ours − Naive SFT
+8.46[+5.27,+11.66]
<0.0001
Naive SFT − Base
−3.22[−6.59,+0.21]
0.9661
𝒯1 Live-Stream QA: Ours − Base
Accuracy
+6.95[+4.60,+9.41]
<0.0001
Effectiveness
+30.88[+27.76,+33.90]
<0.0001
AVG
+18.92[+17.00,+20.81]
<0.0001
Table 16: Same-checkpoint 𝒯1 quality check with MTP Off and On (n=978). Both rows use the same Final Evaluation Judge. AVG is the arithmetic mean of Accuracy and Effectiveness.
Serving mode
Accuracy
Effectiveness
AVG
MTP Off
95.8
93.8
94.8
MTP On
96.22
94.22
95.22
Table 17: MTP trade-off across concurrency. Arrows show Off → On. Decode TPS is client-observed mean per call; Wall and TTFT are P95 seconds. C=1, 2, and 8 are single runs. C=4 values are means of three run-level statistics, not pooled percentiles.
Checkpoint
C
Decode TPS
Speedup
Wall P95 (s)
TTFT P95 (s)
≤15s
Qwen3.6 Base
1
195.42→215.95
1.11×
10.172→9.805
0.508→0.502
100→99%
2
138.81→165.74
1.19×
11.504→10.251
0.623→0.698
97→97%
4
105.04→113.25
1.08×
13.408→16.266
0.643→1.088
97.3→92.3%
8
70.28→70.19
1.00×
17.848→26.136
0.953→1.519
90→79%
Ours
1
160.30→271.40
1.69×
8.98→8.114
0.499→0.553
100→100%
2
129.22→196.15
1.52×
10.074→9.047
0.692→0.754
100→100%
4
99.65→126.64
1.27×
10.869→12.460
0.692→1.349
99.3→98.0%
8
59.01→70.22
1.19×
18.342→20.298
0.900→2.574
91→75%
Table 18: C=1 workload shape. Values are means over successful complete-Agent requests or their constituent model calls.
Route / checkpoint
MTP
Calls/req
Input/call
Output/call
DeepSeek-V4-flash API
–
4.04
6,052
146
DeepSeek-V4-pro API
–
3.67
5,537
119
Qwen3.6 Base
Off
3.23
5,622
115
Qwen3.6 Base
On
3.08
5,571
109
Ours
Off
3.04
5,679
106
Ours
On
3.12
5,786
105
Table 19: MTP serving configuration used for the reported quality check and deployment measurements.
Setting
Engine argument
Value
Speculative algorithm
--speculative-algo
NEXTN
Speculative steps
--speculative-num-steps
3
EAGLE top-k
--speculative-eagle-topk
1
Draft tokens
--speculative-num-draft-tokens
4
Single-token acceptance threshold
--speculative-accept-threshold-single
0.5
Accumulated acceptance threshold
--speculative-accept-threshold-acc
0.7
Table 20: SGLang log-sample diagnostics for MTP-On serving. Triples are mean/P50/P95; Queue is P95/max.
AI-powered digital avatar streamers must answer product questions, engage viewers, and execute marketing strategies in real time, demanding low latency, frequent strategy updates, and accurate yet effective responses. Evolvable Harnesses, whose Skills, Hooks, prompts, and tools can be updated independently of model weights, enable rapid iteration but expose a trade-off: large models adapt zero-shot yet are too slow, whereas compact models meet latency targets but overfit to fixed Harness configurations. We propose Harness-Aware Training (HAT), which trains compact models to adapt to changing Harnesses. Its key component, Harness-State Augmentation (HSA), applies task-preserving transformations to Skill identifiers and content, tool schemas, prompt structures, and Hook functions. Training proceeds in three stages: HSA-SFT learns reasoning and tool use from strong-model trajectories across diverse environments; General On-Policy Distillation restores generalization lost during SFT; and HSA-RL improves robustness to changing Harnesses through reinforcement learning in augmented environments. Across four evaluation sets, HAT achieves 94.8 on Live-Stream QA (base: 80.3; strongest general LLM: 93.0) and 94.6 on Harness-Variant QA (base: 75.4). Unlike Fixed-Harness SFT, which lowers IFEval by 7.7 points from the base model, HAT avoids this regression and reaches 83.5. On one NVIDIA H20 GPU, the optimized system delivers P50 and P95 latencies of 3.4 s and 8.1 s. Deployed in Taobao Live's digital-avatar service, it also yields positive online A/B test results for GMV and item-page views.
![Figure 6: Single-run CoT-length diagnostics for the selected HSA-RL trajectory. Faint lines show recorded step values and bold lines show 25-step trailing means. Panel (a) separates tool-call and final-reply CoT lengths and marks the no-penalty (L≤100), linear (100<L<200), and saturated (L≥200) regions. Panel (b) shows the logged score q(L)∈[0,1] and the equivalent raw reward contribution −0.1q(L). The figure is a training diagnostic, not a causal ablation of latency or quality.](https://media.metallab.ai/papers/2608.15763/f5.svg)
