매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

Chain-of-Experience for Continual LLM Improvement

arXiv:2608.180272026-08-17

LLM도 풀다 보면 실수에서 배운다: 반복 피드백으로 정답률과 비용을 동시에 개선하는 방법

이 연구는 LLM이 시험 시점(inference-time)에 자신의 이전 시도와 피드백을 계속 누적하며 문제를 다시 풀게 하는 Chain-of-Experience(CoE) 방식을 제안한다. GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet 등 8개 모델을 수학·코딩·지식 문제에서 테스트한 결과, 피드백 없이 반복만 하는 것보다 일관되게 성능이 좋았고 전체적으로 5.6% 성능 향상과 19% 비용 절감을 동시에 얻었다. 모델 자체 판단만으로 주는 피드백(self feedback)만으로도 상당한 개선이 있었고, 여러 종류 피드백을 합치면 더 좋아졌다.

무엇을 했나

  1. 기존 LLM 평가는 한 번 질문하고 한 번 답하면 끝나는 방식이라, 모델이 반복 상호작용을 통해 스스로 나아지는 능력을 측정하지 못한다는 문제의식에서 출발했다
  2. CoE는 모델이 이전 시도(행동)와 그에 대한 피드백을 계속 쌓아가며 다음 답을 만들도록 설계했다. 피드백은 네 종류: 아예 없음, 코드 실행 결과(execution), 다른 모델이 채점해주는 판단(model feedback), 정답 여부를 알려주는 correctness feedback
  3. GPT-5, o3, Gemini-2.5 Pro, Claude-4.5 Sonnet 등 8개 최신 모델을 AIME 2025, OmniMath(수학), LiveCodeBench·LiveBench Code(코딩), GPQA Diamond(지식) 등 6개 벤치마크에서 테스트했다
  4. 자체 판단 피드백만 써도 기존의 문맥 학습(ICL)이나 외부 메모리를 쌓는 기법(Dynamic CheatSheet, Agentic Context Engineering)보다 7~9% 더 좋은 성적을 냈고, 정답 여부를 알려주는 강한 피드백을 쓰면 최대 79.3%까지 올라갔다(다른 방법들은 62~67% 수준)
  5. 기본 실력이 좋은 모델일수록 피드백을 받았을 때 더 크게 향상되는 상관관계(평균 피어슨 상관계수 0.5)가 나타났고, 일부러 거짓 피드백(항상 '틀렸다'고 알려주기)을 줘도 모델들이 크게 흔들리지 않고 회복하는 강건함을 보였다
Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.
Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incorporating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1.
Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.
Figure 2: An overview of our studies iterative improvement loop for LMs. The model ℳ learns by repeatedly interacting with the environment ℰ (e.g., model simulation or coding environment). In each cycle, ℳ generates actions conditioned on the input and accumulated experience, then receives feedback from ℰ to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.
Table 1: Dual feedback and principled experience selection results using Claude 4.5 Sonnet. Dual feedback combines model feedback with correctness (math) or executor (code) signals. Memory-based methods (DC, SimpleMem) are applied within-task with no cross-task leakage. Acc: best accuracy (%) over 20 iterations; Best R: iteration achieving best performance.
AIME 2025LiveBench (Code)OmniMath
SettingAccBest RAccBest RAccBest R
Dual (Model + Corr/Exec)76.7R1981.2R1573.5R17
Correctness / Executor70.0R1378.1R1574.5R17
Binary Executor––71.9R13––
Model60.0R657.8R1750.5R9
Model + DC50.0R851.6R1546.0R10
Model + SimpleMem56.7R654.7R1749.5R12
Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.
Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.
Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.
Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H.
Table 2: The best performance over 20 iterations under constant correct” or incorrect” feedback (e.g., “the answer is correct”). Selective majority voting (SelMV) helps LLMs maintain performance. Results are averaged over 3 runs with best scores emphasized.
FeedbackAIME 2025GPQA Diamond
GPT-5 minio4 miniGPT-5 minio4 mini
Self93.391.179.978.8
SelMV Self91.188.980.479.5
All Correct90.073.379.375.8
SelMV Correct93.373.379.376.3
Incorrect91.783.379.372.7
SelMV Incorrect89.786.782.877.8
Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.
Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs.
Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.
Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.
Table 3: Average performance comparison (%) across different LLMs on different datasets. For baselines, ICL, ACE, DC stands for few-shot in-context learning, agentic context engineering, and dynamic cheatsheet, respectively.
MethodAIME 2025LiveCodeBench (V6)LiveBench (Code)OmniMathGPQA DiamondEvaLearn
ICL71.83%62.50%65.46%53.12%78.45%40.99%
ACE71.98%66.94%69.38%50.33%76.58%42.54%
DC73.33%63.59%68.58%48.64%79.56%42.68%
w/o Feedback77.78%72.57%60.16%65.17%80.02%44.91%
Reasoning-high69.05%70.63%55.46%61.81%76.21%39.58%
Reasoning-low60.48%61.03%55.38%50.60%72.92%29.34%
Binary-Executor–72.90%71.65%–––
Self82.22%75.69%69.94%67.52%81.03%51.73%
Correctness/Executor89.05%74.50%75.78%79.61%99.52%57.05%
Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.
Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leveraging the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.
Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.
Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green.
Table 4: Cohen’s Kappa agreement between GPT-5 judge and human annotators across four improvement categories on 100 sampled trajectories. The overall κ of 0.768 indicates substantial agreement [landis1977measurement].
CategoryAgreement (%)Cohen’s κ
Feedback Fidelity84.00.81
Self Reflection72.00.71
Specification Recall80.00.78
Random68.00.63
Overall76.00.768
Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.
Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.
Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.
Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge.
Table 5: Token complexity vs. accuracy across methods. CEF: Correctness/Executor Feedback; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet. Token counts are aggregated across all iterations.
DatasetMethodTokensAcc (%)
AIME 2025CEF108,73484.6
SF108,23183.8
NF106,82574.1
DC11,23374.7
OmniMathCEF176,41274.2
SF175,80672.1
NF173,94466.8
DC16,90463.9
LiveCodeBench (V6)CEF224,11872.6
SF223,44171.2
NF221,55068.0
DC20,77166.4
Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.
Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.
Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).
Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback (i.e., self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).

왜 중요한가

AI 서비스에 LLM을 쓸 때 매번 새로 학습시키지 않고도, 답변을 주고받는 과정에서 스스로 개선되게 만들 수 있다는 뜻이라 실제 비용과 성능 두 마리 토끼를 잡는 데 도움이 된다. 특히 코딩 도구나 챗봇처럼 사용자와 여러 번 주고받는 서비스를 만드는 개발자에게 어떤 피드백을 언제 주는 것이 효율적인지 구체적인 근거를 제공한다.

이 논문의 용어

  • Chain-of-Experience(CoE) · 모델이 이전 시도와 피드백을 계속 누적하며 반복적으로 답을 개선해 나가는 시험 시점 학습 방식
  • self feedback · 정답 여부를 알려주는 외부 정보 없이, 모델 스스로(또는 같은 모델)가 자신의 답을 평가해서 주는 피드백
  • correctness feedback · 답이 맞았는지 틀렸는지를 정확히 알려주는 정답 기준 피드백
  • in-context learning(ICL) · 정답 예시 몇 개를 프롬프트에 함께 넣어 모델이 참고하게 하는 학습 방법
  • Dynamic CheatSheet / Agentic Context Engineering · 이전에 푼 문제에서 얻은 전략을 요약해 외부 메모리에 저장해두고 재사용하는 기존 기법들

논문 원문 초록 (영문)

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

저자 · Haoqin Tu

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Haoqin Tu et al., arXiv:2608.18027, CC BY 4.0