工作日早上 7 点读 AI，周日早上 8 点读周报订阅邮件

METAL LAB

Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning

arXiv:2607.292112026-07-30

研究揭示大模型遇到超出能力的难题时会编造看似合理却错误的推理,并训练模型学会说"我解不出来"

大语言模型面对超出自身能力的问题时往往不会认输,而是不断生成表面合理但实际错误的推理过程。研究团队将这一现象命名为"无效推理",分析了其成因,并提出了CaRL强化学习方法,让模型学会在拒绝回答与编造答案之间优先选择拒绝。将CaRL应用于Qwen3-8B和Qwen3-14B后,无效推理比例大幅下降,同时任务表现基本保持不变。

METAL LAB 解读图

CaRL降低无效推理的流程

证据状态已报告实测结果

  1. 问题诊断在分级Countdown谜题上测试多个模型,发现它们在超出能力的问题上不会拒绝,反而编造看似合理却错误的"无效推理"
  2. 能力校准的奖励设计重新设计奖励机制,正确答案给最高分、拒绝回答给中间分、错误答案给最低分,使拒绝比编造错误答案更有利
  3. 事后拒绝增强将失败推理轨迹的结尾改写成拒绝表述,增加稀缺的拒绝训练样本
  4. GRPO强化学习训练结合上述奖励与增强数据,用GRPO算法训练Qwen3-8B和Qwen3-14B
  5. 结果验证无效推理率在8B模型从65.5%降至7.0%,14B模型从78.6%降至1.0%,同时通用任务准确率差异保持在2%以内
这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 研究团队构建了按难度分级的Countdown任务(一种用给定数字通过加减乘除凑出目标值的算术谜题),测试了Qwen3-8B、Qwen3-32B、gpt-oss-120b、Qwen3-235B-A22B和DeepSeek-V3.2等多个模型。
  2. 所有模型都表现出"普遍性能力越界":即使错误率随难度升高超过60%,拒绝回答的比例却几乎没有增加;即便明确提示模型在无法解决时要承认,最难等级下仍有超过80%的尝试是无效推理。
  3. 失败模式分析发现,表面逻辑连贯但暗藏细微错误的"貌似合理的推理"是最主要的失败类型,占比57%到68%,且随难度升高占比进一步上升。
  4. 为解决这一问题,研究提出结合两个机制的CaRL:能力校准的奖励设计,让拒绝回答的奖励介于正确答案和错误答案之间;以及事后拒绝增强,将失败的推理轨迹回溯改写成拒绝样本,弥补拒绝训练数据稀缺的问题。
  5. 将CaRL应用到Qwen3-8B和Qwen3-14B后,无效推理比例分别从65.5%降到7.0%、从78.6%降到1.0%,可靠性指标(正确记1分、拒绝记0.5分、错误记0分)分别提升0.13和0.16。
Figure 1: Illustration of futile reasoning generated by DeepSeek-R1 Guo et al. 2025. When faced with problems beyond its capability, the model generates plausible-looking but fundamentally incorrect reasoning traces instead of refusing to answer.
Figure 1: Illustration of futile reasoning generated by DeepSeek-R1 Guo et al. 2025. When faced with problems beyond its capability, the model generates plausible-looking but fundamentally incorrect reasoning traces instead of refusing to answer.
Figure 2: Universal Capability Overreach.
Figure 2: Universal Capability Overreach.
Table 1: Main Results on In-Distribution and Out-of-Distribution Tasks. Values in parentheses show changes relative to Vanilla baseline. Green indicates improvement, red indicates degradation.
MethodIn-Distribution (Countdown)Out-of-Distribution (Sudoku)
Acc ↑Reliability ↑RefusalFutile ↓Acc ↑Reliability ↑RefusalFutile ↓
Qwen3-8B
Vanilla59.670.666313.9265.5046.880.496910.6289.41
Standard RL64.08 (+4.4)0.6425 (-.02)0.3399.00 (+33.5)43.25 (-3.6)0.4744 (-.02)13.1385.00 (-4.4)
RLunk=062.71 (+3.0)0.6296 (-.04)0.5099.00 (+33.5)44.62 (-2.3)0.4850 (-.01)12.1286.00 (-3.4)
RLunk=0.563.42 (+3.8)0.6371 (-.03)0.5898.00 (+32.5)45.25 (-1.6)0.5131 (+.02)16.7578.00 (-11.4)
RFT59.13 (-0.5)0.7610 (+.09)35.2117.00 (-48.5)0.00 (-46.9)0.4763 (-.02)95.255.00†
CaRL (Ours)61.00 (+1.3)0.7915 (+.13)37.177.00 (-58.5)46.25 (-0.6)0.6156 (+.12)36.6343.02 (-46.4)
Qwen3-14B
Vanilla63.250.67197.8878.5750.750.555618.6280.46
Standard RL56.42 (-6.8)0.5750 (-.10)2.1795.03 (+16.5)43.63 (-7.1)0.4831 (-.07)13.1383.00 (+2.5)
RLunk=068.21 (+5.0)0.8050 (+.13)24.5823.00 (-55.6)48.38 (-2.4)0.5369 (-.02)14.0079.00 (-1.5)
RFT63.12 (-0.1)0.7879 (+.12)31.3315.00 (-63.6)0.00 (-50.8)0.4525 (-.10)90.5010.00†
CaRL (Ours)67.25 (+4.0)0.8348 (+.16)32.501.00 (-77.6)44.87 (-5.9)0.6262 (+.07)38.8736.00 (-44.5)
† RFT’s low futile rate on OOD is a trivial result of collapsing into near-total refusal (Ref >90%, Acc =0%).
Figure 3: Distribution of Futile Reasoning Patterns.
Figure 3: Distribution of Futile Reasoning Patterns.
Figure 4: Distribution of Capability Quadrants.
Figure 4: Distribution of Capability Quadrants.
Table 2: Futile Rate(%) and response length across difficulty levels on Qwen3-8B.
Level 4Level 6Level 8
MethodFutileLengthFutileLengthFutileLength
RLunk95.8224099.7494898.47042
RFT2.8232714.4647620.49133
CaRL2.018045.641888.16156
Figure 5: The Alignment Trade-off. Naive prompting leads to a collapse in Refusal Recall (Green) on hard tasks while simultaneously increasing Capability Loss (Pink) on solvable tasks.
Figure 5: The Alignment Trade-off. Naive prompting leads to a collapse in Refusal Recall (Green) on hard tasks while simultaneously increasing Capability Loss (Pink) on solvable tasks.
Figure 6: Reasoning Depth Distribution. Refusal behaviors show decisive termination (Peaks), whereas Over-Confidence exhibits a long-tail distribution, confirming the high computational cost of futile reasoning.
Figure 6: Reasoning Depth Distribution. Refusal behaviors show decisive termination (Peaks), whereas Over-Confidence exhibits a long-tail distribution, confirming the high computational cost of futile reasoning.
Table 3: Performance on General Tasks on Qwen3-8B.
MethodAIME 2024GPQA
Acc ↑Reliability ↑Length ↓Acc ↑Reliability ↑Length ↓
Vanilla75.400.754214,78859.850.59857,506
CaRL74.600.785412,41158.330.67685,620
Δ-0.8+3.1-16.1%-1.5+13.1-25.1%
Figure 7: Overview of CaRL. Our framework combines (1) Capability-Calibrated Reward Shaping that establishes a preference hierarchy favoring refusal over hallucination, and (2) Hindsight Refusal Augmentation that converts failed trajectories into refusal trajectories, enabling capability-aligned behavior.
Figure 7: Overview of CaRL. Our framework combines (1) Capability-Calibrated Reward Shaping that establishes a preference hierarchy favoring refusal over hallucination, and (2) Hindsight Refusal Augmentation that converts failed trajectories into refusal trajectories, enabling capability-aligned behavior.
Figure 8: Case study on the countdown task.
Figure 8: Case study on the countdown task.

研究结果

  • 所有测试模型(Qwen3-8B、Qwen3-32B、gpt-oss-120b、Qwen3-235B-A22B、DeepSeek-V3.2)都表现出"普遍性能力越界":难度上升时拒绝率几乎不变,即使加入明确提示,最难等级下仍有超过80%的尝试是无效推理。
  • 失败类型中"貌似合理的推理"占比最高,达57%至68%;"无休止生成"稳定在30%至40%;"退化性重复"从13%降到2%,说明难度越高模型越倾向于编造更精细的错误论证而非简单循环。
  • 在Qwen3-32B上,过度自信(20%)出现频率是过度谨慎(3.4%)的6倍;随着难度上升,拒绝召回率从100%降至30%,而能力损失(在可解问题上不必要拒绝)从0%升至10%。
  • 经CaRL训练后,Qwen3-8B的无效推理率从65.5%降到7.0%,可靠性提升0.13;Qwen3-14B从78.6%降到1.0%,可靠性提升0.16;在分布外任务(Sudoku)和通用基准(AIME 2024、GPQA)上准确率差异均小于2%。
  • 仅做奖励调整的RLunk变体在8B模型上无效推理率仍高达98%至99%,而基于静态数据微调的RFT基线在分布外的Sudoku任务上表现严重崩溃,说明CaRL的奖励设计与事后拒绝增强两个组件缺一不可。
Figure 9: Initial Reasoning Phase. The model systematically explores combinations (e.g., 97+66=163, 3×51=153), attempting to construct the target value 275. Early attempts show valid mathematical reasoning but fail to reach the exact target.
Figure 9: Initial Reasoning Phase. The model systematically explores combinations (e.g., 97+66=163, 3×51=153), attempting to construct the target value 275. Early attempts show valid mathematical reasoning but fail to reach the exact target.
Figure 10: Final Output After Degenerate Repetition. After 50+ failed attempts, the model outputs (97+66+51+38+37)−(3+3+3)=280 while incorrectly asserting it equals 275. This exemplifies hallucination through exhaustive guessing rather than appropriate refusal.
Figure 10: Final Output After Degenerate Repetition. After 50+ failed attempts, the model outputs (97+66+51+38+37)−(3+3+3)=280 while incorrectly asserting it equals 275. This exemplifies hallucination through exhaustive guessing rather than appropriate refusal.

可应用场景

  • 该训练思路可用于其他推理模型,使其在遇到超出能力的算法或谜题类任务时学会承认"我解不出来"。
  • 其奖励设计思想可为高可靠性应用场景中的强化学习方案提供参考,以降低模型自信给出错误答案的风险。
  • 提前中止无效推理的能力可为需要控制计算开销的系统设计提供参考。

局限与待验证事项

  • 实验仅限于不需要外部知识的纯算法推理任务(Countdown和Sudoku),尚未验证该方法在知识密集型任务(如开放域问答)中是否同样有效。
  • 作者明确表示计划将CaRL扩展到开放域问答或数学推理等领域,以验证拒绝机制是否能够泛化到所有场景。
  • 测试仅覆盖Qwen3-8B和Qwen3-14B两种规模,尚未验证在其他架构或更大规模模型上是否能取得同样效果。

为什么重要

看似合理却错误的AI答案会误导用户信任错误的推导,这在需要高可靠性的场景中是严重风险。这项研究提供了一种具体的训练方法,让模型能够识别自身能力边界并主动承认"不知道",而不是编造看似正确的错误推理。

本文术语

  • 无效推理(futile reasoning) · 模型面对超出自身能力的问题时,持续生成表面看似合理但实际毫无价值且错误的推理过程
  • CaRL(能力对齐的强化学习) · 一种强化学习框架,训练模型的行为与其真实能力边界相匹配,学会在无法解决时拒绝回答
  • 事后拒绝增强(Hindsight Refusal Augmentation) · 将失败推理轨迹的结尾改写成拒绝回答的表述,从而增加稀缺的拒绝训练样本
  • GRPO(群体相对策略优化) · 一种强化学习算法,通过比较一组采样回答的相对好坏来更新模型策略
  • Countdown任务 · 一种算术谜题,是24点游戏的变体,要求用给定数字和基本运算凑出目标数值

论文原文摘要（英文）

Large language models generate computationally expensive yet semantically void reasoning on beyond-capability tasks, creating risks where plausible-sounding but incorrect derivations mislead users. We characterize this futile reasoning phenomenon through systematic analysis, revealing universal capability overreach and systematic miscalibration between capability and behavior. The dominant failure mode is specious reasoning, which outputs look superficially valid but contain subtle errors, escalating with task difficulty. To address this, we introduce CaRL (Capability-aligned Reinforcement Learning), which aligns model behavior with capability boundaries through reward shaping that incentivizes refusal over futile reasoning and hindsight refusal augmentation that converts failures into refusal supervision. Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties, effectively achieving capability-aligned behavior without sacrificing utility. https://github.com/icip-cas/Knowing-When-to-Quit

作者 · Xinyan Guan

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Xinyan Guan et al., arXiv:2607.29211, arxiv-nonexclusive