SHAPE of Chain-of-Thought in Math Reasoning

arXiv:2608.286002026-09-01

A framework that tags each step of an AI's math 'thinking' with what mathematical move it's making and what interpretation it's stuck in, revealing why some reasoning succeeds and why RL training narrows a model's strategies

SHAPE breaks down an LLM's chain-of-thought into 'semantic spaces' (how it currently frames the problem, e.g. algebraically or geometrically) and 'heuristics' (concrete moves like working backward or simplifying) borrowed from math-education research. Using an automated labeling pipeline, the authors show these heuristic-based features predict whether an answer is correct better than surface features like reasoning length, and that correct answers tend to stay within a few semantic spaces rather than wandering. They also find that reinforcement learning post-training squeezes models into a narrower set of heuristics already present in the base model, and that explicitly feeding heuristics into training (HA-Plan-GRPO) improves accuracy over a plain planning baseline.

METAL LAB explanatory visual

How SHAPE turns a chain-of-thought into measurable structure

Evidence statusMeasured results and planned work

Raw CoT trajectoryThe step-by-step text a model writes while solving a math problem
Segment into content unitsBreak the text into the smallest spans that represent one problem-solving move
Tag heuristicsLabel each unit with the mathematical actions it performs, such as working backward or introducing a representation
Track semantic spacesClassify each unit as New, Return, or Maintain relative to the model's evolving interpretation of the problem
Compute metrics & apply to trainingDerive correctness-predicting features, measure post-training's Density/Coverage narrowing, and feed heuristics back into GRPO rollout prompts

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

SHAPE analyzes chain-of-thought text using two math-education concepts: semantic spaces (the model's current mathematical interpretation of the problem, e.g. algebraic vs. geometric) and heuristics (specific actions like working backward, introducing symbols, or checking cases).
An automated pipeline (built with models like Grok-4.1-Fast and Qwen3.5-27B) segments CoT text into content units, tags each with heuristic labels, and tracks whether each unit opens a new semantic space, returns to an old one, or maintains the current one; this was validated against a hand-annotated gold set of 48 trajectories.
Logistic regression trained on SHAPE's heuristic-frequency features predicted correctness with AUROC 0.664, beating CoT length (0.504), length+reasoning-token features (0.503), self-revision markers like 'wait'/'aha' (0.618), and the episode-label baseline ThinkARM (0.618).
Comparing base vs. RL-trained (Think-RLVR) model trajectories using Density/Coverage metrics on heuristic usage showed Density above 1.0 and Coverage below 1.0 in every post-trained pair tested, meaning RL training concentrates successful trajectories into a denser but narrower slice of the base model's existing strategies rather than creating new ones; an unrelated-model control pair showed low Density (0.520) and low Coverage (0.437), confirming this pattern is specific to base-to-post-trained pairs.
Training Qwen3-1.7B-Base with GRPO on MATH problems, a version that injects explicit heuristic hints into the rollout prompt (HA-Plan-GRPO) beat a heuristic-free planning-only version (Plan-GRPO) on Avg@64 and Pass@64 on the MATH-Perturb test set, in preliminary results.

Figure 1: Illustrative example of SHAPE annotation applied to a synthetic CoT trace. The solver first sets up an algebraic formulation (Space 1), introducing a system of equations. It then abandons this approach and switches to a trial-based counting strategy (Space 2). Finally, it returns to the algebraic formulation (Space 1) and solves the system. We refer to each problem-solving interpretation as a semantic space—a distinct mathematical interpretation defined by the objects, goals, and constraints the model adopts. Within each space, individual steps are tagged as heuristics—purposeful mathematical actions such as introducing a representation (H3a) or exploring particular cases (H9a)—or as non-heuristic steps such as stating the answer (N4). The arrows mark transitions between these spaces. For the full heuristic taxonomy, see Appendix A.

Table 1: Correctness prediction under 5-fold stratified cross-validation. “Length + reasoning” includes CoT length together with reasoning features (reasoning-token count and proportion), while “Self-revision” includes self-revision features derived from lexical self-revision markers. Within SHAPE, H denotes the 11 heuristic categories (H1–H11) and N the non-heuristic category.
Feature Set	AUROC ↑	# Features
Length	0.504±0.03	1
Length + reasoning	0.503±0.03	3
Self-revision	0.618±0.03	3
ThinkARM	0.618±0.02	8
SHAPE (H)	0.653±0.02	11
SHAPE (H+N)	0.664±0.02	12

Figure 2: Overview of the automated SHAPE analysis pipeline. Given a Chain-of-Thought (CoT) trajectory, the pipeline first segments the text into content units corresponding to heuristic-bearing spans. It then assigns one or more heuristic labels to each unit using a tagging model. Finally, a semantic-space state tracking model classify the model’s mathematical interpretation change into Maintain, New, or Return.

Table 2: Descriptive statistics of SHAPE metrics by correctness. Nspaceeff: effective number of semantic spaces; Ntranseff: effective number of semantic-space transitions; transition ratio ρ: measuring the intensity of revisiting semantic space. (C), (I), and (O) denote correct, incorrect, and overall trajectories.
	Nspaceeff	Ntranseff	ρ
Model	(C)	(I)	(O)	(C)	(I)	(O)	(C)	(I)	(O)	Acc.
Open-source reasoning models with full traces
Qwen3-32B	2.08	2.28	2.16	1.38	1.53	1.44	0.51	0.50	0.51	.63
DeepSeek-R1	1.72	2.47	2.00	0.82	1.64	1.12	0.34	0.51	0.41	.63
QwQ-32B	1.74	2.74	2.11	0.86	2.20	1.35	0.32	0.60	0.42	.63
DeepSeek-R1-Distill-Qwen-32B	1.85	2.07	1.95	1.02	1.21	1.10	0.38	0.41	0.40	.57
DeepSeek-R1-Distill-Qwen-7B	1.51	2.18	1.85	0.60	1.40	1.01	0.27	0.49	0.41	.49
Deepseek-R1-Distill-Qwen-1.5B	1.41	2.04	1.81	0.53	1.27	1.00	0.27	0.49	0.41	.36
Phi-4-Reasoning	2.03	2.83	2.53	1.15	2.10	1.75	0.35	0.58	0.49	.35
Instruction-tuned models without extended reasoning
Qwen3-32B-NR	1.66	1.74	1.71	0.70	0.75	0.73	0.29	0.33	0.32	.34
Gemini-2.0-Flash	1.43	1.58	1.53	0.43	0.59	0.54	0.20	0.28	0.25	.34
Phi-4	1.14	1.51	1.41	0.14	0.51	0.41	0.07	0.23	0.19	.29
Qwen-2.5-32B	1.33	1.39	1.37	0.37	0.42	0.40	0.16	0.20	0.19	.28
GPT-4o	1.68	1.40	1.46	0.72	0.40	0.46	0.29	0.19	0.21	.19
Proprietary reasoning models with hidden traces
Gemini-2.5-Flash	1.56	1.91	1.67	0.59	0.91	0.69	0.28	0.35	0.30	.67
GPT-o3-mini	1.51	1.65	1.59	0.56	0.70	0.63	0.27	0.32	0.30	.47
GPT-o1-mini	1.36	1.58	1.50	0.36	0.61	0.52	0.19	0.29	0.25	.37

Figure 3: Projection of successful Olmo-3-{Base, Think-RLVR} trajectories onto the first principal component of heuristic frequency space. Olmo-3-Think-RLVR concentrates around the peak of the base distribution while leaving the left tail uncovered, suggesting that post-training narrows rather than reshapes the heuristic distribution.

Table 4: Density and Coverage of post-trained model trajectories relative to base model trajectories in heuristic frequency space (k=3, successful trajectories only, aggregated across all perturbation conditions). Density > 1 indicates that post-trained trajectories concentrate in the dense core of the base distribution; among such cases, lower Coverage indicates stronger mode-seeking. The cross-model baseline confirms that unrelated base models show neither high Density nor high Coverage.
Base	Post-trained	Nbase	NPT	Density	Coverage
Post-trained
Qwen3-1.7B-Base	Qwen3-1.7B-GRPO	834	886	1.220	0.871
Olmo-3-7B	Olmo-3-7B-Think-RL-Zero	1229	1307	1.250	0.707
Olmo-3-7B	Olmo-3-7B-Think-RLVR	1507	1600	1.032	0.531
Cross-model baseline (unrelated distributions)
Olmo-3-7B	Qwen3-1.7B-Base	71	66	0.520	0.437

Table 5: Preliminary results for Heuristic-Augmented GRPO on the MATH-Perturb test split. Plan-GRPO adds a planning step without heuristic information. HA-Plan-GRPO provides eleven mathematical heuristics during rollout and incorporates them into a planning-guided rollout. Bold indicates the best results.
Model	Original	Simple	Hard
Avg@64	Pass@64	Avg@64	Pass@64	Avg@64	Pass@64
Qwen3-1.7B-Base	23.54	77.40	23.10	79.13	11.84	57.39
+ Plan-GRPO	30.00	80.00	29.86	78.26	14.52	61.74
+ HA-Plan-GRPO (ours)	36.80	80.00	35.80	79.13	17.72	62.61

Table 6: Taxonomy of Problem-Solving Strategies. This table summarizes the code, strategy name, detailed description, and theoretical sources associated with each heuristic. Here, P = Pólya (1945), K = Koichu et al. (2007), F = Favier (2022), and P&K = Posamentier and Krulik (2008).
Strategy Name	Description & Sources
H1. Changing the register of semiotic representation	This strategy involves translating the problem’s representation from one semiotic register to another. It includes converting between natural language, algebraic, geometric, and visual representations to facilitate understanding or solving. Sources: Creating a model (K); Change the semiotic representation register (Changer de registre de représentation sémiotique) (F)
H2. Cognitive Reinterpretation	This involves changing the way an object or property in the problem is interpreted. It redefines the identity or attributes of an element in a way different from the initial presentation, without necessarily changing the register. Sources: Consider another way of interpreting the problem’s objects (Envisager une autre façon d’interpréter les objets du problème) (F); Adopting a different point of view (P&K)
H3. Introduce Symbolic Representation, Formalization, and Structural Augmentation, including…
H3a. Introduce Symbolic Representation and Formalization	The act of introducing new variables, labeling unknowns, or performing substitutions to make ambiguous targets operationally manageable. Sources: Notation; Setting up equations (P); Creating a model (K); Introduce names or notations (Introduire des noms ou des notations) (F)
H3b. Structural Augmentation	Constructing auxiliary objects, lemmas, or entirely new mathematical frameworks that are not present in the original problem. This is a creative addition to the problem space, such as drawing auxiliary lines, defining new functions, or shifting the problem into a new structural representation Sources: Auxiliary elements, Lemma (P); Introducing an auxiliary element (K); Introduce auxiliary elements (Introduire des éléments auxiliaires) (F)
H4. Problem Classification / Rephrase the Problem and Goal, including…
H4a. Problem Categorization / Strategic Rephrasing of Goal / Breaking into Sub-goals	Explicitly stating the problem type, identifying applicable solution methods, or reformulating the main goal in clearer mathematical terms. Sources:What is the unknown? (P); Reformuler le problème (Reformulate the problem) (F)
H4b. Filtering Constraints	Strategically identifying the most essential constraints or conditions that guide the upcoming solution approach. Sources:Separate the various parts of the condition (P); Exploring a particular datum (K)
H5. Wishful Thinking (Simplify / Reduce the Problem and Conditions)	Temporarily modifying the problem to a simpler version to gain insight, verify formulas, or explore solution strategies. Sources: If you cannot solve the proposed problem (P); Reduce the problem to a simpler one (Réduire le problème à un problème plus simple) (F)
H6. Explicit Case Analysis, Decompose into Subproblems	Logically decomposing the problem into distinct cases, non-overlapping subsets, or sub-problems that, when combined, yield the full solution. The cases should ideally be exhaustive and mutually exclusive. Sources: Decomposing and recombining (P); Décomposer le domaine du problème et travailler cas par cas (Decompose the problem domain and work case by case) (F)
H7. Arguing by contradiction	A proof strategy where the negation of the proposition is assumed to derive a contradiction, thereby proving the original statement. Sources: Reductio ad absurdum and indirect proof (P); Arguing by contradiction (K)
H8. Analogy and Presenting Related Theorems, including…
H8a. Analogy	Recalling previously solved problems, known methods, or applying a recently established logical procedure to a new target within the same problem. This involves recognizing structural similarities and transferring a strategy from one context (or one part of the equation) to another. Sources: Have you seen it before?; Do you know a related problem? (P); Activating a previous experience (K); Make a connection with a previously encountered problem (Faire le lien avec un problème déjà rencontré) (F); Solving a simpler analogous problem (P&K)
H8b. Presenting Related Theorems, Tools, or Properties	Introducing specific mathematical theorems, formulas, identities, or properties that are not provided in the problem statement but are necessary to advance the solution. Sources: Connect with a mathematical tool (theorem, property) (Faire le lien avec un outil mathématique (théorème, propriété)) (F)
H9. Experimental and Pattern Exploration, including…
H9a. Exploring particular cases or numbers	Plugging in specific values, extreme/boundary values, or limits to discover patterns, build intuition, or verify feasibility. Sources: Specialization (P); Partial Induction (K); Explore a specific piece of data (Explorer une donnée particulière) (F); Finding a Pattern (P&K)
H9b. Exploration of symmetry	Identifying and exploiting mathematical or structural symmetry to reduce the solution space or simplify computation. Sources: Symmetry (P); Exploration of Symmetry (K); Exploit symmetry properties (Exploiter les propriétés de symétrie) (F)
H10. Thinking from the end to the beginning (Working backward)	Starting from the desired conclusion (target goal) and working logical steps backward to reach the known premises or to determine what would be sufficient to prove. Sources: Working backwards (P); Thinking Backward (K); Working backward (Travailler à reculons) (F); Working Backwards (P&K)
H11. Verification and Looking Back, including…
H11a. Re-solving & Checking the Argument	Re-performing the same logical steps or calculations without a strategic change, or conducting a direct manual check of elements to verify a previous claim. Sources: Can you check the result? (P); Local Self-evaluating (K)
H11b. Deriving the Result Differently	Solving the same problem or sub-goal using a structurally different mathematical method to provide independent confirmation. Sources: Can you derive the result differently? (P); Local Self-evaluating (K)
H11c. Backtracking for self-verification	Realizing an error, finding a flaw in an assumption, or recognizing that the current approach is not working, and revising the direction. Sources: Backtracking (Retour arrière) (F)
H11d. Checking the Result / Sanity Check / Progress Review	Broadly covers any reflection on whether the solution is on the right track or checking feasibility. Sources: Can you check the result?; Test by dimension (P); Local Self-evaluating (K)
H11e. Generalization & Corollary	Extending the result to broader cases. Identifying general principles from specific solutions. Sources: Wisdom of proverbs (P); Generalization (K)
H11f. Reflect on Rigor & Wisdom	Evaluating the efficiency of the solution strategy, questioning the rigor, or meta-reflecting on definitions/rules. Sources: Why proofs? (P)

Table 8: Heuristic-tagging performance of candidate annotator models on the gold set. Weighted F1 reflects overall label agreement, while macro F1 emphasizes performance on rare heuristic classes.
Model	Weighted F1	Macro F1
Grok-4.1-Fast	76.98	65.04
GPT-5	72.25	63.55
GPT-5-mini	66.44	55.89
Gemini-3-Flash	62.83	41.79
Gemini-3-Flash-Lite	70.21	54.14
Qwen3.5-27B	70.44	61.36

Table 9: Class-wise heuristic-tagging performance of Qwen3.5-27B on the gold set. We report precision (P), recall (R), F1, Cohen’s kappa, and the number of gold and predicted instances.
Class	F1	Kappa
H1	0.5272	0.4893
H2	0.4318	0.4176
H3	0.4884	0.4383
H4	0.6114	0.5049
H5	0.6250	0.6134
H6	0.6667	0.6626
H7	0.6667	0.6664
H8	0.7201	0.6252
H9	0.5934	0.5692
H10	0.5000	0.4988
H11	0.8115	0.6383
N	0.7216	0.6702

Table 11: Hyperparameters used for GRPO training.
Parameter	Value
Base Model	Qwen/Qwen3-1.7B-Base
Training Batch Size	32
Rollouts per Prompt	4
Number of GPUs	2×B200
Optimizer	AdamW
Temperature	1.0
Top-p	1.0
Top-k	-1
Max Response Length	2048
Learning Rate	1×10−6
Warmup Steps	10
Training Steps	200

Findings

Heuristic-frequency features from SHAPE achieved AUROC 0.664±0.02 predicting answer correctness, versus 0.504±0.03 for CoT length, 0.503±0.03 for length+reasoning-token features, 0.618±0.03 for self-revision markers, and 0.618±0.02 for the ThinkARM episode-label baseline.
Reasoning models (with extended thinking) showed higher semantic-space coverage (Nspaceeff 1.81-2.53) and transition ratio (rho 0.40-0.51) than instruction-tuned models without extended reasoning (Nspaceeff 1.37-1.71; rho 0.19-0.32); incorrect trajectories generally showed higher rho than correct ones across most models.
Under hard perturbations that require a different solution approach, models showed larger changes in heuristic-frequency distribution, semantic-space count, and transition ratio than under simple perturbations that preserve the solution method; this divergence appeared even within the first 5-10 content units.
All post-trained model pairs tested (Olmo-3-7B Think-RLVR and Think-RL-Zero, Qwen3-1.7B-Base with GRPO) showed Density above 1.0 and Coverage below 1.0 relative to their base models, while an unrelated-model control pair showed low Density (0.520) and low Coverage (0.437).
In preliminary MATH-Perturb test results, both Plan-GRPO and HA-Plan-GRPO improved over the untrained base model, and HA-Plan-GRPO (which adds heuristic hints to the rollout prompt) achieved higher Avg@64 and Pass@64 than Plan-GRPO alone.

Where it can be used

Diagnosing why a math-reasoning model fails on certain problems by inspecting which heuristics and semantic-space transitions it used, rather than only checking the final answer.
Auditing whether an RL post-training run is genuinely diversifying a model's problem-solving strategies or just concentrating it around already-known solution patterns.
Designing RLVR training prompts that inject explicit heuristic vocabulary to potentially boost accuracy on math benchmarks, following the HA-Plan-GRPO recipe.
Building automated CoT-tagging pipelines for other structured-reasoning domains by adapting the segmentation-tagging-tracking pipeline design, though this specific instantiation is math-only.

Limits and open work

The framework and its heuristic taxonomy were developed and validated only on mathematical reasoning benchmarks (MATH-Perturb, Omni-MATH subset); applying it to other domains is stated as future work, not demonstrated.
The gold standard for heuristic tagging comes from just 48 annotated trajectories across four models, and semantic-space tracking has no objective gold standard at all, so it was calibrated by iterative manual review rather than measured against ground truth.
HA-Plan-GRPO results are explicitly labeled 'preliminary' and were tested on one base model (Qwen3-1.7B-Base) with one training recipe; broader model sizes, families, or RL algorithms were not tested.
Heuristic tagging performance is weaker on rare heuristic classes (e.g., H7, H10) due to small numbers of gold examples, which may make some fine-grained conclusions less stable.
Automated annotation relies on proprietary or large open-weight LLMs (Grok-4.1-Fast, Qwen3.5-27B) as taggers, introducing potential annotation noise or model-specific bias into all downstream analyses.

Why it matters

Accuracy scores alone don't reveal how a model reasons or why reinforcement learning sometimes plateaus or even narrows a model's problem-solving repertoire; SHAPE gives a concrete, theory-grounded way to see and measure that. For anyone building or evaluating math-reasoning models, it offers both a diagnostic lens on training side-effects and a concrete recipe (heuristic-augmented prompts during RL) that showed measurable gains in the paper's own tests.

Terms in this paper

Chain-of-Thought (CoT) · The step-by-step text a model produces while solving a problem, before giving the final answer
semantic space · The model's current mathematical framing of the problem, such as treating it algebraically vs. geometrically
heuristic · A specific purposeful math move, like working backward, simplifying, or checking a case
GRPO · Group Relative Policy Optimization, a reinforcement-learning training method used to fine-tune the models here
Density and Coverage · Metrics that measure how tightly a target distribution clusters inside a reference distribution (Density) and how much of the reference it reaches (Coverage)

Original abstract (English)

Large language models (LLMs) achieve strong performance on mathematical reasoning benchmarks, yet the mathematically meaningful skills underlying their reasoning remain underexplored. We introduce \texttt{SHAPE}, a framework that analyzes Chain-of-Thought (CoT) trajectories through two lenses developed in mathematics education: (1) semantic spaces: the model's evolving mathematical interpretations of a problem (e.g., algebraic, geometric), and (2) heuristics: the specific mathematical actions taken within those spaces (e.g., simplifying the problem, working backward). We first use \texttt{SHAPE} to analyze the reasoning patterns of various models. Our findings reveal that the mathematical heuristics employed by a model better explain final answer correctness than traditional CoT features. Furthermore, models are likely to reach correct solutions by concentrating their reasoning effort within a few semantic spaces rather than exploring many disparate ones -- a pattern consistent with human behavior. Next, we utilize the \texttt{SHAPE} lens to evaluate whether post-training truly enhances mathematical proficiency. We find that reinforcement learning induces mode-seeking in heuristic usage. Lastly, we post-train LLMs by promoting diverse heuristics and demonstrate its effectiveness in improving accuracy. Overall, \texttt{SHAPE} provides a theoretically-grounded diagnostic framework for decoding LLM reasoning and offers a new path toward post-training LLMs for math reasoning. The code for our model is available at https://github.com/holi-lab/SHAPE-of-CoT

Authors · Jonghyun Song, Sangjun Song, Minjae Oh, Haesung Pyun, Sungsik Lee, Yohan Jo

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Jonghyun Song et al., arXiv:2608.28600, CC BY 4.0