METAL LAB

CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions

arXiv:2608.289582026-09-01

A new 52K-example dataset teaches multimodal AI to 'draw while thinking' instead of writing everything in words

CoVA-SFT is a 51,904-sample dataset with over 222,046 step-by-step visual reasoning traces covering 17 tasks like Sudoku, chess, family-tree kinship puzzles, and geometry problems. Models fine-tuned on it learn to build and update internal visual workspaces (tables, graphs, diagrams) while solving text-only reasoning problems, rather than forcing everything into prose. The fine-tuned model more than doubles the average score of prior interleaved visual-reasoning methods on the companion CoVA-Bench test, but still falls short of strong text-only reasoning models overall.

METAL LAB explanatory visual

How CoVA-SFT builds and tests visual-reasoning traces

Evidence statusMeasured results reported

  1. 1. Rationale & QAClaude 4.5 Sonnet explains why a visual workspace helps for a task and writes the question-answer pair.
  2. 2. Agentic renderingThe model generates reasoning step by step, calling Matplotlib to render each visual workspace and feeding the image back as context.
  3. 3. Verification loopThe model checks the rendered image against the problem statement and redraws when it finds structural errors.
  4. 4. Fine-tuningQwen3-VL-8B-Thinking is trained on the resulting 51,904 trajectories with a combined text and cosine-similarity visual-token loss.
  5. 5. CoVA-Bench evaluationThe tuned model is tested on 1,700 held-out samples across 17 tasks, scoring 38.2% average versus 16.8% for the best prior interleaved baseline.
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. Built a 51,904-example training corpus (CoVA-SFT) with 222,046 multimodal reasoning steps across 5 visual layout families (Table, Graph, Layout, Game, Math) and 17 tasks, plus a 1,700-sample held-out test set (CoVA-Bench, 100 per task).
  2. Used Claude 4.5 Sonnet as an automated generator in a three-stage pipeline: it explains why a visual workspace helps and writes a Q&A pair, renders the workspace step by step with Matplotlib while feeding each rendered image back as context, then checks the image against the problem and redraws if something is inconsistent.
  3. Fine-tuned Qwen3-VL-8B-Thinking with a combined loss: normal text-generation loss plus a cosine-similarity loss that aligns predicted latent visual tokens with target visual embeddings, so the model learns to update an internal visual state alongside its text.
  4. On CoVA-Bench, the fine-tuned model averaged 38.2%, more than double the next-best interleaved-visual-reasoning baseline (MathCanvas at 16.8%), and it beat all text-only baselines specifically on Graph reasoning (62.0% vs. 57.0% for the best text-only model).
  5. On Math tasks, the fine-tuned model scored only 8.5%, far below the best text-only baseline (27.3%), since symbolic math notation is already easy for text models and visual tokens seem to add overhead rather than help.
Figure 1: Chain of Visual Abstractions. When solv- ing text-based reasoning problems, standard textual rea- soning (left) forces models to serialize naturally visual problems into prose. In contrast, models trained on our CoVA-SFT dataset (right) learn to construct and maintain (latent) visual workspaces during the reason-
Figure 1: Chain of Visual Abstractions. When solv- ing text-based reasoning problems, standard textual rea- soning (left) forces models to serialize naturally visual problems into prose. In contrast, models trained on our CoVA-SFT dataset (right) learn to construct and maintain (latent) visual workspaces during the reason-
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 2
Table 1: Comparison with representative interleaved reasoning datasets. Prior work grounds reasoning in existing images; CoVA-SFT constructs visual workspaces from scratch for text-only inputs.
DatasetTaskInputVisual Role
Zebra-CoTVision-language reasoningMultimodalGrounded visual observations from input
Math-VRMath reasoningText & multimodalMath-specific plots and diagrams
CoVA-SFT (Ours)Text-only reasoningText-onlyVisual workspaces constructed during reasoning
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 3
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 4

Findings

  • The CoVA-SFT-trained model reached an average of 38.2% on CoVA-Bench, more than double the best prior interleaved-visual-reasoning baseline, MathCanvas, at 16.8%.
  • Other interleaved baselines scored much lower: CodePlot-CoT 12.9%, Zebra-CoT 11.2%, and Thinking-with-Generated-Images (TwGI) near zero (0.9%), reportedly because rendering errors propagate uncorrected through their reasoning chains.
  • The CoVA-SFT model outperformed all evaluated text-only baselines on Graph reasoning (62.0% vs. 57.0% for the best text-only model, Qwen3-VL-Instruct).
  • The CoVA-SFT model underperformed text-only baselines on Table (47.2% vs. up to 80.3%), Layout (53.4% vs. up to 76.3%), and especially Math (8.5% vs. 27.3% for Qwen3-VL-Thinking).
  • Text-only models themselves struggled on certain domains without any visual aid: Qwen3-Think scored only 44.5% on Graph and 19.3% on Game tasks.
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 5
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 6

Where it can be used

  • Training data and benchmark for researchers building multimodal models that need to track spatial or relational state (graphs, tables, grids) during reasoning.
  • A starting point for studying when latent visual reasoning helps versus when plain text reasoning is already sufficient, informing dataset or objective design choices.
  • Evaluation suite (CoVA-Bench) for comparing new interleaved text-visual reasoning methods on a shared set of 17 tasks.
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 7
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 8

Limits and open work

  • The dataset relies on an upstream model (Claude 4.5 Sonnet) to generate and verify traces; subtle hallucinations or reasoning errors can still slip into the training data despite the self-correction loop.
  • Visual abstractions are limited to static, programmatically renderable formats (2D tables, graphs, layout grids, geometry plots); open-world spatial scenes or continuous robotic environments are not covered.
  • The fine-tuned model still trails strong text-only reasoning baselines overall, and clearly loses ground on Math tasks, indicating the approach is not yet a universal replacement for text-only CoT.
  • Training and inference require extra compute: a dual text-plus-visual loss needs careful tuning, and tasks with many sequential visual states demand longer context and more compute at inference time.
  • Results are reported on the paper's own held-out CoVA-Bench with an LLM-as-judge scoring setup (Gemini-2.5-flash), so generalization to other benchmarks or judging methods is not yet demonstrated.
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions figure 11

Why it matters

Many reasoning problems (scheduling, mazes, chess, family trees) are naturally visual, but today's AI models mostly reason by writing long blocks of text, which is clumsy for spatial or relational structure. This dataset and benchmark give researchers a concrete, large-scale way to train and test models that reason by building and updating internal pictures instead of only prose.

Terms in this paper

  • Chain-of-thought (CoT) · A prompting technique where a model writes out intermediate reasoning steps in text before giving a final answer.
  • Interleaved reasoning · Reasoning that mixes text steps with visual steps (images, diagrams) rather than using only words.
  • Latent visual tokens · Continuous internal representations standing in for an image, used inside the model's reasoning instead of a literal rendered picture.
  • Agentic generation pipeline · An automated process where an AI model itself drives multi-step data creation: reasoning, rendering, and checking its own output.
  • CoVA-Bench · The 1,700-sample held-out test set released alongside CoVA-SFT to evaluate interleaved visual reasoning across the same 17 tasks.

Figures we cannot republish

  • Figure 0
  • Figure 9
  • Figure 10
See the figures in the original paper →

Original abstract (English)

Chain-of-thought (CoT) reasoning has dramatically improved large language models (LLMs) by allowing them to decompose problems into intermediate steps. While CoT is widely effective for linguistic tasks, text-only CoT forces models to serialize visual problems into awkward prose. Although architectural solutions exist to process visual inputs, the community lacks a massive, multi-step, self-corrected dataset to teach models how to build and maintain internal visual workspaces when solving purely textual reasoning problems. To address this limitation, we introduce CoVA-SFT, a highly structured corpus of 51.9K samples containing over 222K multimodal reasoning steps across 5 distinct layout families and 17 complex tasks, and CoVA-Bench, a companion benchmark of 1,700 held-out test samples spanning the same tasks for reproducible evaluation. By providing explicit rationale formulations, agentic renderings, and verification loops, CoVA-SFT teaches multimodal language models to interleave text and visual abstractions. We validate the dataset by demonstrating that models fine-tuned on CoVA-SFT outperform all interleaved CoT baselines by more than 2x on average on CoVA-Bench, though they still fall short of strong text-only CoT baselines, highlighting open challenges for future work.

Authors · Tsung-Han Wu, Heekyung Lee, Anya Ji, Haoming Chen, Trevor Darrell, Joseph E. Gonzalez, David M. Chan

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Tsung-Han Wu et al., arXiv:2608.28958, CC BY 4.0