SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
训练AI智能体时,只跑一次任务、不用额外评判模型也能学得更好
SAPO用同一个语言模型,在一次生成过程中同时输出决定行动的策略和评估行动好坏的价值判断。以往的方法要么需要对同一个任务反复尝试多次再比较结果,要么需要单独训练一个评判模型,而SAPO利用文本从左到右生成的自然顺序,在一次尝试里就同时得到这两者。在ALFWorld和WebShop这两个多步骤任务环境中,SAPO的成功率平均比PPO高15.1个百分点,比GRPO高12.1个百分点,同时每轮训练耗时比PPO减少33.2%。
他们做了什么
- 现有的分组比较方法(如GRPO)需要对同一任务反复采样多次结果再比较,当各次结果奖励接近时学习信号会消失,而且采样次数和性能之间存在昂贵的权衡
- SAPO在同一个语言模型生成文本的不同位置读取信息:动作生成之前总结当前状态,动作生成之后评估这个动作,全部在一次因果生成序列中完成
- 通过从轨迹末尾向前逐步回溯计算每一步的贡献(结合λ回报和批量归一化),无需多次同步采样也能稳定估计每一轮的价值
- 在Qwen2.5-1.5B和7B模型上,ALFWorld和WebShop测试显示,SAPO成功率平均比PPO高15.1个百分点,比GRPO高12.1个百分点,同时省去了单独评判模型的内存开销,每轮训练时间比PPO减少33.2%
| Type | Method | ALFWorld | WebShop | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Pick | Look | Clean | Heat | Cool | Pick2 | All | Score | Succ. | ||
| Closed-Source Model | ||||||||||
| Prompting | GPT-4o | 75.3 | 60.8 | 31.2 | 56.7 | 21.6 | 49.8 | 48.0 | 31.8 | 23.7 |
| Prompting | Gemini-2.5-Pro | 92.8 | 63.3 | 62.1 | 69.0 | 26.6 | 58.7 | 60.3 | 42.5 | 35.9 |
| Qwen2.5-1.5B-Instruct | ||||||||||
| Prompting | ReAct | 17.4 | 20.5 | 15.7 | 6.2 | 7.7 | 2.0 | 12.8 | 40.1 | 11.3 |
| Prompting | Reflexion | 35.3 | 22.2 | 21.7 | 13.6 | 19.4 | 3.7 | 21.8 | 55.8 | 21.9 |
| RL Training | RLOO | 88.3±3.0 | 52.8±8.6 | 71.0±5.9 | 62.8±8.7 | 66.4±5.5 | 56.9±4.7 | 69.7±2.5 | 73.9±5.6 | 52.1±6.7 |
| RL Training | EMPG | 85.5 | 33.5 | 78.9 | 76.2 | 74.7 | 89.1 | 73.7 | 80.4 | 60.8 |
| RL Training | GiGPOw/ std | 94.4±5.9 | 67.5±4.6 | 94.8±3.8 | 94.4±7.8 | 79.8±4.7 | 76.4±5.4 | 86.7±1.7 | 83.1±1.6 | 65.0±3.2 |
| RL Training | GiGPOw/o std | 96.0±1.4 | 76.5±3.9 | 91.8±5.5 | 91.3±6.3 | 71.7±8.4 | 79.5±7.7 | 86.1±4.7 | 83.5±1.8 | 67.4±4.5 |
| RL Training | PPO (with critic) | 64.8±3.5 | 40.5±6.9 | 57.1±4.9 | 60.6±6.6 | 46.4±4.0 | 47.4±1.9 | 54.4±3.1 | 73.8±3.0 | 51.5±2.9 |
| RL Training | GRPO | 85.3±1.5 | 53.7±8.0 | 84.5±6.8 | 78.2±7.9 | 59.7±5.0 | 53.5±5.6 | 72.8±3.6 | 75.8±3.5 | 56.8±3.8 |
| RL Training | SAPO | 92.0±2.9 | 76.9±6.3 | 100.0±0.0 | 100.0±0.0 | 82.8±4.7 | 82.4±5.0 | 90.1±2.3 | 82.21.4 | 63.71.6 |
| Qwen2.5-7B-Instruct | ||||||||||
| Prompting | ReAct | 48.5 | 35.4 | 34.3 | 13.2 | 18.2 | 17.6 | 31.2 | 46.2 | 19.5 |
| Prompting | Reflexion | 62.0 | 41.6 | 44.9 | 30.9 | 36.3 | 23.8 | 42.7 | 58.1 | 28.8 |
| RL Training | RLOO | 87.6±4.3 | 78.2±8.3 | 87.3±5.8 | 81.3±7.6 | 71.9±5.2 | 48.9±8.4 | 75.5±4.6 | 80.3±3.2 | 65.7±4.0 |
| RL Training | EMPG | 92.9 | 75.2 | 74.8 | 86.3 | 73.7 | 65.3 | 78.5 | 81.0 | 69.3 |
| RL Training | GiGPOw/ std | 97.7±1.6 | 82.7±7.9 | 98.8±1.6 | 83.7±7.2 | 89.3±8.2 | 79.2±6.6 | 90.8±1.3 | 84.4±2.9 | 72.8±3.2 |
| RL Training | GiGPOw/o std | 91.8±5.4 | 88.6±6.3 | 95.9±3.2 | 90.2±2.6 | 86.5±5.5 | 85.2±7.5 | 90.2±2.3 | 86.2±2.6 | 75.2±3.8 |
| RL Training | PPO (with critic) | 92.3±4.0 | 64.0±8.4 | 92.5±2.4 | 89.5±7.0 | 80.3±2.0 | 68.8±8.3 | 80.4±2.7 | 81.4±3.1 | 68.7±5.1 |
| RL Training | GRPO | 90.8±5.1 | 66.1±6.7 | 89.3±5.4 | 74.7±6.9 | 72.5±5.4 | 64.7±7.3 | 77.6±5.2 | 79.3±2.8 | 66.1±3.7 |
| RL Training | SAPO | 99.0±1.4 | 82.3±2.1 | 100.0±0.0 | 97.9±4.7 | 79.7±3.9 | 91.7±1.6 | 94.0±1.7 | 88.6±1.8 | 82.4±2.0 |
为什么重要
这意味着训练需要多步骤与环境交互的AI智能体时,可以在降低内存和计算成本的同时反而提升性能。对计算资源有限、又想训练长周期交互式智能体的研究者和企业来说,这能带来实际的成本节省。
本文术语
- 策略(policy) · 决定在特定情境下采取什么行动的AI规则
- 价值函数(value function) · 预测某个状态或行动未来会带来多好结果的函数
- 评判模型(critic) · 与策略模型分开设置、专门用来评估行动好坏的辅助模型
- PPO · 一种限制策略每次更新幅度以保持训练稳定的经典强化学习方法
- GRPO · 通过比较同一任务多次尝试的结果来训练、无需单独评判模型的方法
论文原文摘要（英文）
Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO.
在 arXiv 阅读最新论文
- LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment在正式微调前先偷看几步训练的梯度,让LoRA的初始化更聪明
- Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI助手在该向你提问的时候,却更愿意自己去核实事实
- Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correction文本信息缺失或损坏时,这个AI不靠一次性猜测,而是反复修正猜测结果,从而更准确地判断情绪
- Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems要测试访谈式对话系统需要大量不同性格的虚拟用户,这项研究用大语言模型自动生成这些虚拟用户人设
- Rethinking Patch Based Multivariate Time Series Forecasting with Semantic Structured Partitioning别再机械切分时间序列,按语义把它切成有意义的块
- Reliable Financial Named Entity Recognition under Domain ShiftAI在正式文件里学到的自信,一到推特上就变得不可信
- Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis让AI分析脑影像数据时,把“为什么这个结论可信”也一并记录下来
- GenMatch: An End-to-End Generative Matching Framework for Micro-View Order-Dispatching in Ride-Hailing滴滴把打车派单从预测-计算-匹配三段式流程改成一次生成完成,线上效果提升明显
METAL LAB 最新报道
图片来源： Dayang Liang et al., arXiv:2608.19842, CC BY 4.0