每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Chain-of-Experience for Continual LLM Improvement

arXiv:2608.180272026-08-17

大模型也能靠反复试错变强:边做题边接收反馈,准确率和成本同步改善

这篇论文提出了Chain-of-Experience(CoE)方法,让大语言模型在回答同一道题时不断利用之前尝试留下的反馈重新作答,而不是一问一答就结束。研究在GPT-5、Gemini-2.5 Pro、Claude-4.5 Sonnet等8个模型上,针对数学、编程、知识类任务测试后发现,CoE比不带反馈的基线方法表现更稳定,整体准确率提升5.6%,同时API成本降低19%。即便只用模型自己给自己的反馈(没有标准答案),提升幅度也很可观,叠加多种反馈效果更好。

他们做了什么

  1. 以往对大模型的评测大多是一问一答,没有考察模型能否通过反复交互和反馈不断变强,这篇论文正是研究这一点
  2. CoE让模型在生成下一个答案时,参考此前所有尝试及其对应反馈。研究设计了四种反馈:完全没有反馈、代码执行结果反馈、由另一个模型充当评委给出的反馈、以及直接告知对错的正确性反馈
  3. 实验覆盖GPT-5、o3、Gemini-2.5 Pro、Claude-4.5 Sonnet等8个最新推理模型,测试集包括AIME 2025、OmniMath(数学)、LiveCodeBench V6、LiveBench Code(编程)、GPQA Diamond(知识)等六个基准
  4. 仅靠模型自我评判的反馈,就比常见的少样本上下文学习(ICL)、Dynamic CheatSheet、Agentic Context Engineering等方法高出7%到9%;如果用最强的正确性反馈,平均准确率能冲到79.3%,而对比方法大多只有62%到67%左右
  5. 基础能力越强的模型,从反馈中获益越多(平均皮尔逊相关系数为0.5);即使故意给模型持续错误或误导性的反馈,模型的表现依然相对稳健,不会大幅崩溃
Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.
Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.
Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.
Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.
Table 1: Dual feedback and principled experience selection results using Claude 4.5 Sonnet. Dual feedback combines model feedback with correctness (math) or executor (code) signals. Memory-based methods (DC, SimpleMem) are applied within-task with no cross-task leakage. Acc: best accuracy (%) over 20 iterations; Best R: iteration achieving best performance.
AIME 2025LiveBench (Code)OmniMath
SettingAccBest RAccBest RAccBest R
Dual (Model + Corr/Exec)76.7R1981.2R1573.5R17
Correctness / Executor70.0R1378.1R1574.5R17
Binary Executor––71.9R13––
Model60.0R657.8R1750.5R9
Model + DC50.0R851.6R1546.0R10
Model + SimpleMem56.7R654.7R1749.5R12
Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.
Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.
Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.
Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.
Table 2: The best performance over 20 iterations under constant correct” or incorrect” feedback (e.g., “the answer is correct”). Selective majority voting (SelMV) helps LLMs maintain performance. Results are averaged over 3 runs with best scores emphasized.
FeedbackAIME 2025GPQA Diamond
GPT-5 minio4 miniGPT-5 minio4 mini
Self93.391.179.978.8
SelMV Self91.188.980.479.5
All Correct90.073.379.375.8
SelMV Correct93.373.379.376.3
Incorrect91.783.379.372.7
SelMV Incorrect89.786.782.877.8
Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.
Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.
Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.
Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.
Table 3: Average performance comparison (%) across different LLMs on different datasets. For baselines, ICL, ACE, DC stands for few-shot in-context learning, agentic context engineering, and dynamic cheatsheet, respectively.
MethodAIME 2025LiveCodeBench (V6)LiveBench (Code)OmniMathGPQA DiamondEvaLearn
ICL71.83%62.50%65.46%53.12%78.45%40.99%
ACE71.98%66.94%69.38%50.33%76.58%42.54%
DC73.33%63.59%68.58%48.64%79.56%42.68%
w/o Feedback77.78%72.57%60.16%65.17%80.02%44.91%
Reasoning-high69.05%70.63%55.46%61.81%76.21%39.58%
Reasoning-low60.48%61.03%55.38%50.60%72.92%29.34%
Binary-Executor–72.90%71.65%–––
Self82.22%75.69%69.94%67.52%81.03%51.73%
Correctness/Executor89.05%74.50%75.78%79.61%99.52%57.05%
Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.
Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.
Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.
Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.
Table 4: Cohen’s Kappa agreement between GPT-5 judge and human annotators across four improvement categories on 100 sampled trajectories. The overall κ of 0.768 indicates substantial agreement [landis1977measurement].
CategoryAgreement (%)Cohen’s κ
Feedback Fidelity84.00.81
Self Reflection72.00.71
Specification Recall80.00.78
Random68.00.63
Overall76.00.768
Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.
Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.
Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.
Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.
Table 5: Token complexity vs. accuracy across methods. CEF: Correctness/Executor Feedback; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet. Token counts are aggregated across all iterations.
DatasetMethodTokensAcc (%)
AIME 2025CEF108,73484.6
SF108,23183.8
NF106,82574.1
DC11,23374.7
OmniMathCEF176,41274.2
SF175,80672.1
NF173,94466.8
DC16,90463.9
LiveCodeBench (V6)CEF224,11872.6
SF223,44171.2
NF221,55068.0
DC20,77166.4
Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.
Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.
Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).
Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).

为什么重要

这说明不需要重新训练模型,只要让大模型在回答过程中不断利用反馈进行自我修正,就能同时提升准确率并降低推理成本,这对开发多轮对话、编程助手或智能体类产品的团队很有实用价值。研究还给出了不同任务类型该用哪种反馈更划算的具体依据。

本文术语

  • Chain-of-Experience(CoE) · 让模型在多次尝试同一任务的过程中不断积累此前的答案与反馈,并据此改进下一次作答的测试时方法
  • self feedback · 由模型自己(或另一个模型)对答案做出评判所产生的反馈,不依赖标准答案
  • correctness feedback · 直接告知答案对错的反馈信号
  • 上下文学习(ICL) · 在提示中放入几个已解决的示例,让模型参考后再回答新问题
  • Dynamic CheatSheet / Agentic Context Engineering · 此前提出的方法,把从已解决问题中提炼出的策略存入外部记忆库供后续复用

论文原文摘要（英文）

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

作者 · Haoqin Tu

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Haoqin Tu et al., arXiv:2608.18027, CC BY 4.0