SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

arXiv:2608.198422026-08-21

A way to train AI agents better using just one attempt per task, with no separate evaluator model needed

SAPO uses a single language model to produce both the policy (which decides actions) and the value estimate (which judges how good those actions are) from one pass. Previous methods needed either multiple attempts at the same task to compare rewards, or a separate evaluator model, but SAPO exploits the natural left-to-right order of text generation to get both from a single rollout. On the ALFWorld and WebShop multi-turn task benchmarks, it beat standard PPO and GRPO by an average of 15.1 and 12.1 percentage points respectively, while cutting per-iteration runtime by 33.2% compared with PPO.

What they did

Existing group-comparison methods like GRPO need multiple attempts at the same task to compare rewards, and when rewards are nearly identical across attempts the learning signal can vanish; this also forces a costly tradeoff between how many attempts to sample and how well the model performs
SAPO reads a policy decision and a value judgment from different points within the same sequence of generated text: before an action it estimates the state, and after generating the action it evaluates that action, all within one causal language model
It computes each turn's contribution by working backward through the trajectory using lambda-returns combined with batch-level normalization, giving stable credit assignment without needing multiple synchronized rollouts
Tested with Qwen2.5-1.5B and 7B models on ALFWorld and WebShop, SAPO improved success rates by a mean of 15.1 percentage points over PPO and 12.1 over GRPO, removed the memory overhead of a separate critic model, and reduced per-iteration runtime by 33.2% versus PPO

Table 1: Evaluation results on ALFWorld and WebShop. For each RL training method, we report the mean and standard deviation over three random seeds. The ALFWorld contains six categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). Most entries in this table are reported by Feng et al. [4]. Notably, the baseline GiGPOw/o std replaces the group-relative normalization std with one.
Type	Method	ALFWorld	WebShop
Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-Source Model
Prompting	GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Prompting	Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Qwen2.5-1.5B-Instruct
Prompting	ReAct	17.4	20.5	15.7	6.2	7.7	2.0	12.8	40.1	11.3
Prompting	Reflexion	35.3	22.2	21.7	13.6	19.4	3.7	21.8	55.8	21.9
RL Training	RLOO	88.3±3.0	52.8±8.6	71.0±5.9	62.8±8.7	66.4±5.5	56.9±4.7	69.7±2.5	73.9±5.6	52.1±6.7
RL Training	EMPG	85.5	33.5	78.9	76.2	74.7	89.1	73.7	80.4	60.8
RL Training	GiGPOw/ std	94.4±5.9	67.5±4.6	94.8±3.8	94.4±7.8	79.8±4.7	76.4±5.4	86.7±1.7	83.1±1.6	65.0±3.2
RL Training	GiGPOw/o std	96.0±1.4	76.5±3.9	91.8±5.5	91.3±6.3	71.7±8.4	79.5±7.7	86.1±4.7	83.5±1.8	67.4±4.5
RL Training	PPO (with critic)	64.8±3.5	40.5±6.9	57.1±4.9	60.6±6.6	46.4±4.0	47.4±1.9	54.4±3.1	73.8±3.0	51.5±2.9
RL Training	GRPO	85.3±1.5	53.7±8.0	84.5±6.8	78.2±7.9	59.7±5.0	53.5±5.6	72.8±3.6	75.8±3.5	56.8±3.8
RL Training	SAPO	92.0±2.9	76.9±6.3	100.0±0.0	100.0±0.0	82.8±4.7	82.4±5.0	90.1±2.3	82.21.4	63.71.6
Qwen2.5-7B-Instruct
Prompting	ReAct	48.5	35.4	34.3	13.2	18.2	17.6	31.2	46.2	19.5
Prompting	Reflexion	62.0	41.6	44.9	30.9	36.3	23.8	42.7	58.1	28.8
RL Training	RLOO	87.6±4.3	78.2±8.3	87.3±5.8	81.3±7.6	71.9±5.2	48.9±8.4	75.5±4.6	80.3±3.2	65.7±4.0
RL Training	EMPG	92.9	75.2	74.8	86.3	73.7	65.3	78.5	81.0	69.3
RL Training	GiGPOw/ std	97.7±1.6	82.7±7.9	98.8±1.6	83.7±7.2	89.3±8.2	79.2±6.6	90.8±1.3	84.4±2.9	72.8±3.2
RL Training	GiGPOw/o std	91.8±5.4	88.6±6.3	95.9±3.2	90.2±2.6	86.5±5.5	85.2±7.5	90.2±2.3	86.2±2.6	75.2±3.8
RL Training	PPO (with critic)	92.3±4.0	64.0±8.4	92.5±2.4	89.5±7.0	80.3±2.0	68.8±8.3	80.4±2.7	81.4±3.1	68.7±5.1
RL Training	GRPO	90.8±5.1	66.1±6.7	89.3±5.4	74.7±6.9	72.5±5.4	64.7±7.3	77.6±5.2	79.3±2.8	66.1±3.7
RL Training	SAPO	99.0±1.4	82.3±2.1	100.0±0.0	97.9±4.7	79.7±3.9	91.7±1.6	94.0±1.7	88.6±1.8	82.4±2.0

Figure 2: Per-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2.5-1.5B. The vertical axis is shown on a logarithmic scale to accommodate the large variation in module runtimes. N/A indicates that SAPO does not require the corresponding value-model or critic-update module. Overall, SAPO reduces the measured per-iteration runtime from 451.2 s to 301.4 s, corresponding to a 33.2% reduction compared with PPO.

Why it matters

This matters because training AI agents that act over many steps in interactive environments is usually expensive in both memory and compute, and SAPO shows a way to cut those costs while actually improving performance. That is directly useful for researchers and companies with limited computing budgets who want to build long-horizon interactive agents.

Terms in this paper

policy · the part of an AI that decides which action to take given a situation
value function · a function that predicts how good a state or action will turn out to be in the long run
critic model · a separate model used alongside the policy to judge how good its actions are
PPO · a widely used reinforcement learning method that limits how much the policy can change in each update
GRPO · a method that compares rewards across multiple attempts at the same task instead of using a separate critic model

Original abstract (English)

Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO.

Authors · Dayang Liang, Lang Feng, Bo An, Yunlong Liu

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Dayang Liang et al., arXiv:2608.19842, CC BY 4.0