One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Chain-of-Experience for Continual LLM Improvement

arXiv:2608.180272026-08-17

LLMs can learn from their own mistakes mid-task, boosting accuracy while cutting API costs

This paper introduces Chain-of-Experience (CoE), a way for LLMs to keep reattempting a problem while accumulating feedback from previous tries, instead of answering once and stopping. Tested on 8 models including GPT-5, Gemini-2.5 Pro, and Claude-4.5 Sonnet across math, coding, and knowledge tasks, CoE consistently beat feedback-free baselines, delivering a 5.6% overall accuracy gain with 19% lower API cost. Even feedback generated by the model itself, without any ground-truth signal, produced large gains, and combining feedback types helped further.

What they did

  1. Standard LLM evaluation treats every question as a one-shot event, ignoring whether models can actually get better by interacting with feedback over multiple tries -- this paper studies that ability directly
  2. CoE lets a model condition its next answer on the full history of prior attempts and the feedback each attempt received. Four feedback types were tested: none, execution feedback (code test results), model feedback (a judge model's critique), and correctness feedback (right/wrong signal)
  3. Experiments covered 8 state-of-the-art reasoning models (GPT-5, o3, Gemini-2.5 Pro, Claude-4.5 Sonnet, among others) on six benchmarks: AIME 2025, OmniMath (math), LiveCodeBench V6, LiveBench Code (coding), and GPQA Diamond (knowledge)
  4. Self feedback alone outperformed established test-time methods like few-shot in-context learning, Dynamic CheatSheet, and Agentic Context Engineering by 7-9%, while the strongest feedback setting pushed average accuracy to 79.3% versus roughly 62-67% for those baselines
  5. Models with stronger baseline (zero-shot) ability tended to improve more from feedback (average Pearson correlation of 0.5), and models stayed fairly robust even when fed deliberately wrong or misleading feedback
Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.
Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.
Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.
Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.
Table 1: Dual feedback and principled experience selection results using Claude 4.5 Sonnet. Dual feedback combines model feedback with correctness (math) or executor (code) signals. Memory-based methods (DC, SimpleMem) are applied within-task with no cross-task leakage. Acc: best accuracy (%) over 20 iterations; Best R: iteration achieving best performance.
AIME 2025LiveBench (Code)OmniMath
SettingAccBest RAccBest RAccBest R
Dual (Model + Corr/Exec)76.7R1981.2R1573.5R17
Correctness / Executor70.0R1378.1R1574.5R17
Binary Executor––71.9R13––
Model60.0R657.8R1750.5R9
Model + DC50.0R851.6R1546.0R10
Model + SimpleMem56.7R654.7R1749.5R12
Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.
Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.
Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.
Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.
Table 2: The best performance over 20 iterations under constant correct” or incorrect” feedback (e.g., “the answer is correct”). Selective majority voting (SelMV) helps LLMs maintain performance. Results are averaged over 3 runs with best scores emphasized.
FeedbackAIME 2025GPQA Diamond
GPT-5 minio4 miniGPT-5 minio4 mini
Self93.391.179.978.8
SelMV Self91.188.980.479.5
All Correct90.073.379.375.8
SelMV Correct93.373.379.376.3
Incorrect91.783.379.372.7
SelMV Incorrect89.786.782.877.8
Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.
Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.
Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.
Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.
Table 3: Average performance comparison (%) across different LLMs on different datasets. For baselines, ICL, ACE, DC stands for few-shot in-context learning, agentic context engineering, and dynamic cheatsheet, respectively.
MethodAIME 2025LiveCodeBench (V6)LiveBench (Code)OmniMathGPQA DiamondEvaLearn
ICL71.83%62.50%65.46%53.12%78.45%40.99%
ACE71.98%66.94%69.38%50.33%76.58%42.54%
DC73.33%63.59%68.58%48.64%79.56%42.68%
w/o Feedback77.78%72.57%60.16%65.17%80.02%44.91%
Reasoning-high69.05%70.63%55.46%61.81%76.21%39.58%
Reasoning-low60.48%61.03%55.38%50.60%72.92%29.34%
Binary-Executor–72.90%71.65%–––
Self82.22%75.69%69.94%67.52%81.03%51.73%
Correctness/Executor89.05%74.50%75.78%79.61%99.52%57.05%
Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.
Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.
Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.
Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.
Table 4: Cohen’s Kappa agreement between GPT-5 judge and human annotators across four improvement categories on 100 sampled trajectories. The overall κ of 0.768 indicates substantial agreement [landis1977measurement].
CategoryAgreement (%)Cohen’s κ
Feedback Fidelity84.00.81
Self Reflection72.00.71
Specification Recall80.00.78
Random68.00.63
Overall76.00.768
Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.
Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.
Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.
Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.
Table 5: Token complexity vs. accuracy across methods. CEF: Correctness/Executor Feedback; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet. Token counts are aggregated across all iterations.
DatasetMethodTokensAcc (%)
AIME 2025CEF108,73484.6
SF108,23183.8
NF106,82574.1
DC11,23374.7
OmniMathCEF176,41274.2
SF175,80672.1
NF173,94466.8
DC16,90463.9
LiveCodeBench (V6)CEF224,11872.6
SF223,44171.2
NF221,55068.0
DC20,77166.4
Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.
Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.
Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).
Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).

Why it matters

It shows that instead of retraining a model, developers can get meaningful accuracy gains and lower inference costs simply by letting an LLM iterate on its own attempts with feedback -- relevant for anyone building multi-turn coding assistants, tutoring tools, or agents. It also gives practical guidance on which feedback type to invest in depending on task type and budget.

Terms in this paper

  • Chain-of-Experience (CoE) · A test-time method where a model repeatedly retries a task while accumulating its past attempts and the feedback each attempt received
  • self feedback · Feedback generated by a model judging its own (or another model's) answer, without access to the ground-truth answer
  • correctness feedback · A feedback signal that directly tells the model whether its answer was right or wrong
  • in-context learning (ICL) · A prompting method that shows the model a few solved examples before asking it to answer a new question
  • Dynamic CheatSheet / Agentic Context Engineering · Prior methods that build an external memory of reusable strategies distilled from previously solved problems

Original abstract (English)

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

Authors · Haoqin Tu

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Haoqin Tu et al., arXiv:2608.18027, CC BY 4.0