Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning

arXiv:2607.292112026-07-30

A study diagnoses why LLMs keep spinning out plausible-but-wrong answers on problems too hard for them, and trains them to say 'I can't solve this' instead

Large language models often refuse to give up on problems beyond their ability, instead generating reasoning that looks valid but is subtly wrong. The authors name this 'futile reasoning', analyze why it happens, and propose CaRL, a reinforcement learning method that rewards honest refusal over fabricated answers. Applied to Qwen3-8B and Qwen3-14B, CaRL sharply cut futile reasoning while keeping task performance almost unchanged.

METAL LAB explanatory visual

How CaRL reduces futile reasoning

Evidence statusMeasured results reported

Diagnose the problemTested multiple models on graded Countdown puzzles and found they generate plausible-but-wrong 'futile reasoning' instead of refusing beyond-capability problems
Capability-calibrated reward shapingRedesigned the reward so correct answers score highest, refusals score in the middle, and incorrect answers score lowest, making refusal more attractive than fabricating a wrong answer
Hindsight refusal augmentationRewrote the ending of failed reasoning traces into refusal statements to generate more refusal training examples
GRPO reinforcement learningTrained Qwen3-8B and Qwen3-14B with the combined reward and augmented data using the GRPO algorithm
Measured outcomeFutile reasoning fell from 65.5% to 7.0% on 8B and from 78.6% to 1.0% on 14B, while general task accuracy stayed within 2% of baseline

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The team built a difficulty-graded Countdown task (an arithmetic puzzle where you combine given numbers with +,-,x,/ to hit a target) and tested Qwen3-8B, Qwen3-32B, gpt-oss-120b, Qwen3-235B-A22B, and DeepSeek-V3.2.
All models showed 'universal capability overreach': as error rates climbed past 60% on harder problems, refusal rates barely rose, and even explicit prompts to admit ignorance still left over 80% of attempts as futile reasoning on the hardest level.
Failure analysis found 'specious reasoning' -- outputs that look logically sound but contain subtle errors -- as the dominant failure mode at 57-68% of cases, growing more common as difficulty increased.
To fix this, the authors combined Capability-Calibrated Reward Shaping, which gives refusal a reward between correct and incorrect answers, with Hindsight Refusal Augmentation, which retroactively converts failed reasoning traces into refusal examples to solve the scarcity of refusal training data.
Applying the combined method, CaRL, to Qwen3-8B and Qwen3-14B cut futile reasoning from 65.5% to 7.0% and from 78.6% to 1.0% respectively, while reliability scores (1 point for correct, 0.5 for refusal, 0 for wrong) improved by +0.13 and +0.16.

Figure 1: Illustration of futile reasoning generated by DeepSeek-R1 Guo et al. 2025. When faced with problems beyond its capability, the model generates plausible-looking but fundamentally incorrect reasoning traces instead of refusing to answer.

Figure 2: Universal Capability Overreach.

Table 1: Main Results on In-Distribution and Out-of-Distribution Tasks. Values in parentheses show changes relative to Vanilla baseline. Green indicates improvement, red indicates degradation.
Method	In-Distribution (Countdown)	Out-of-Distribution (Sudoku)
Acc ↑	Reliability ↑	Refusal	Futile ↓	Acc ↑	Reliability ↑	Refusal	Futile ↓
Qwen3-8B
Vanilla	59.67	0.6663	13.92	65.50	46.88	0.4969	10.62	89.41
Standard RL	64.08 (+4.4)	0.6425 (-.02)	0.33	99.00 (+33.5)	43.25 (-3.6)	0.4744 (-.02)	13.13	85.00 (-4.4)
RLunk=0	62.71 (+3.0)	0.6296 (-.04)	0.50	99.00 (+33.5)	44.62 (-2.3)	0.4850 (-.01)	12.12	86.00 (-3.4)
RLunk=0.5	63.42 (+3.8)	0.6371 (-.03)	0.58	98.00 (+32.5)	45.25 (-1.6)	0.5131 (+.02)	16.75	78.00 (-11.4)
RFT	59.13 (-0.5)	0.7610 (+.09)	35.21	17.00 (-48.5)	0.00 (-46.9)	0.4763 (-.02)	95.25	5.00†
CaRL (Ours)	61.00 (+1.3)	0.7915 (+.13)	37.17	7.00 (-58.5)	46.25 (-0.6)	0.6156 (+.12)	36.63	43.02 (-46.4)
Qwen3-14B
Vanilla	63.25	0.6719	7.88	78.57	50.75	0.5556	18.62	80.46
Standard RL	56.42 (-6.8)	0.5750 (-.10)	2.17	95.03 (+16.5)	43.63 (-7.1)	0.4831 (-.07)	13.13	83.00 (+2.5)
RLunk=0	68.21 (+5.0)	0.8050 (+.13)	24.58	23.00 (-55.6)	48.38 (-2.4)	0.5369 (-.02)	14.00	79.00 (-1.5)
RFT	63.12 (-0.1)	0.7879 (+.12)	31.33	15.00 (-63.6)	0.00 (-50.8)	0.4525 (-.10)	90.50	10.00†
CaRL (Ours)	67.25 (+4.0)	0.8348 (+.16)	32.50	1.00 (-77.6)	44.87 (-5.9)	0.6262 (+.07)	38.87	36.00 (-44.5)
† RFT’s low futile rate on OOD is a trivial result of collapsing into near-total refusal (Ref >90%, Acc =0%).

Figure 3: Distribution of Futile Reasoning Patterns.

Figure 4: Distribution of Capability Quadrants.

Table 2: Futile Rate(%) and response length across difficulty levels on Qwen3-8B.
	Level 4	Level 6	Level 8
Method	Futile	Length	Futile	Length	Futile	Length
RLunk	95.8	2240	99.7	4948	98.4	7042
RFT	2.8	2327	14.4	6476	20.4	9133
CaRL	2.0	1804	5.6	4188	8.1	6156

Figure 5: The Alignment Trade-off. Naive prompting leads to a collapse in Refusal Recall (Green) on hard tasks while simultaneously increasing Capability Loss (Pink) on solvable tasks.

Figure 6: Reasoning Depth Distribution. Refusal behaviors show decisive termination (Peaks), whereas Over-Confidence exhibits a long-tail distribution, confirming the high computational cost of futile reasoning.

Table 3: Performance on General Tasks on Qwen3-8B.
Method	AIME 2024	GPQA
Acc ↑	Reliability ↑	Length ↓	Acc ↑	Reliability ↑	Length ↓
Vanilla	75.40	0.7542	14,788	59.85	0.5985	7,506
CaRL	74.60	0.7854	12,411	58.33	0.6768	5,620
Δ	-0.8	+3.1	-16.1%	-1.5	+13.1	-25.1%

Figure 7: Overview of CaRL. Our framework combines (1) Capability-Calibrated Reward Shaping that establishes a preference hierarchy favoring refusal over hallucination, and (2) Hindsight Refusal Augmentation that converts failed trajectories into refusal trajectories, enabling capability-aligned behavior.

Figure 8: Case study on the countdown task.

Findings

Every tested model (Qwen3-8B, Qwen3-32B, gpt-oss-120b, Qwen3-235B-A22B, DeepSeek-V3.2) showed 'universal capability overreach': refusal rates stayed near zero even as difficulty rose, and even with explicit prompting, over 80% of attempts on the hardest level remained futile.
Specious reasoning (plausible-looking but subtly wrong steps) was the dominant failure type at 57-68%; endless generation held steady at 30-40%; degenerate repetition dropped from 13% to 2% as difficulty rose, showing models fabricate more sophisticated justifications for harder problems.
On Qwen3-32B, over-confidence (20%) occurred 6 times more often than over-conservatism (3.4%), and as difficulty rose, refusal recall collapsed from 100% to 30% while capability loss (needless refusal on solvable problems) rose from 0% to 10%.
After CaRL training, Qwen3-8B's futile reasoning rate dropped from 65.5% to 7.0% with reliability up by +0.13, and Qwen3-14B's dropped from 78.6% to 1.0% with reliability up by +0.16, while out-of-distribution (Sudoku) results and general benchmarks (AIME 2024, GPQA) showed less than 2% accuracy loss.
Reward-shaping-only variants (RLunk) left futile reasoning at 98-99% on the 8B model, and the supervised-finetuning baseline (RFT) collapsed on the out-of-distribution Sudoku task, showing both components of CaRL (reward shaping and hindsight refusal augmentation) are needed together.

Figure 9: Initial Reasoning Phase. The model systematically explores combinations (e.g., 97+66=163, 3×51=153), attempting to construct the target value 275. Early attempts show valid mathematical reasoning but fail to reach the exact target.

Figure 10: Final Output After Degenerate Repetition. After 50+ failed attempts, the model outputs (97+66+51+38+37)−(3+3+3)=280 while incorrectly asserting it equals 275. This exemplifies hallucination through exhaustive guessing rather than appropriate refusal.

Where it can be used

The training approach could be adapted to other reasoning models to teach them to admit 'I can't solve this' on algorithmic or puzzle-style tasks beyond their ability.
The reward-shaping idea could inform reinforcement learning designs for high-reliability applications where confidently wrong answers are especially costly.
The early-abort behavior could be used to reduce unnecessary computation in systems where long futile reasoning chains waste resources.

Limits and open work

Experiments are limited to pure algorithmic reasoning tasks (Countdown and Sudoku) that require no external knowledge, so it's untested whether the approach works on knowledge-intensive tasks like open-domain QA.
The authors state they plan to extend CaRL to domains such as open-domain question answering or mathematical reasoning to check whether the refusal mechanism generalizes.
Testing was done only on Qwen3-8B and Qwen3-14B, so it is unverified whether the same gains hold for other model architectures or much larger models.

Why it matters

Plausible-looking but wrong AI answers can mislead users who trust them, which is a serious risk wherever reliability matters. This work offers a concrete training method to make models recognize their own limits and say 'I don't know' instead of fabricating convincing but false reasoning.

Terms in this paper

futile reasoning · when a model faces a problem beyond its ability and keeps generating reasoning that looks plausible but is actually useless and wrong
CaRL (Capability-aligned Reinforcement Learning) · a reinforcement learning framework that trains a model to refuse tasks it cannot solve, matching its behavior to its true ability
Hindsight Refusal Augmentation · a technique that rewrites the ending of a failed reasoning trace into a refusal statement, creating more refusal examples for training
GRPO (Group Relative Policy Optimization) · a reinforcement learning algorithm that compares a group of sampled responses and learns to favor the relatively better ones
Countdown task · an arithmetic puzzle, a variant of the Game of 24, where you must reach a target number using given numbers and basic operations

Original abstract (English)

Large language models generate computationally expensive yet semantically void reasoning on beyond-capability tasks, creating risks where plausible-sounding but incorrect derivations mislead users. We characterize this futile reasoning phenomenon through systematic analysis, revealing universal capability overreach and systematic miscalibration between capability and behavior. The dominant failure mode is specious reasoning, which outputs look superficially valid but contain subtle errors, escalating with task difficulty. To address this, we introduce CaRL (Capability-aligned Reinforcement Learning), which aligns model behavior with capability boundaries through reward shaping that incentivizes refusal over futile reasoning and hindsight refusal augmentation that converts failures into refusal supervision. Experiments demonstrate a substantial reduction in futile reasoning while preserving performance across task difficulties, effectively achieving capability-aligned behavior without sacrificing utility. https://github.com/icip-cas/Knowing-When-to-Quit

Authors · Xinyan Guan

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Xinyan Guan et al., arXiv:2607.29211, arxiv-nonexclusive