Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

arXiv:2607.286612026-07-21

把完整财务报表原样喂给AI、再拿掉计算公式提示,连最强模型也会算错近一半

研究团队构建了FinIndices基准测试,直接使用未经裁剪的真实上市公司财务报表(最长达3.2万token)来考察大语言模型的财务计算能力。测试分为单值计算和多指标表格生成两种任务,结果显示一旦去掉公式提示,即便是顶尖模型的准确率也会大幅崩溃。在需要一次性生成多个数值的表格任务中,模型常常搞混时间区间或用偷懒的会计处理蒙混过关。

METAL LAB 解读图

FinIndices构建与评测流程

证据状态已报告实测结果

1. 财务报表提取从829家公司的原始报告中解析资产负债表、利润表、现金流量表,并随机变换其格式。
2. 上下文构建与陷阱注入对齐计算所需的准确报告期,并插入干扰期间或故意省略必要数据。
3. 问答生成与专家验证依据教科书公式由程序计算出标准答案,再由三位专家投票确定640条最终测试集。
4. 有无提示对比评测在Single-Index和Table-Index任务上分别测试有无公式提示时的准确率差距。
5. 微调挽回尝试用Gemini蒸馏的推理轨迹微调模型,测算无提示情况下能恢复多少准确率。

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

团队从829家上市公司、384个财务指标、28个报告期中自动合成问答对,并经过三位专家投票验证,最终构建出包含68,307个样本的FinIndices基准。
题目分为两种范式——计算单一数值的Single-Index和填充多指标表格的Table-Index——并覆盖会计领域理解、时间推理、口径对齐等多个能力维度。
数据集特意设计了对抗性陷阱:插入无关的干扰期间,或故意省略必要数据,迫使模型明确回答信息不足而不是瞎编数字。
实验对比了提供公式提示与不提供提示两种条件下的表现,也对比了单值计算与表格生成任务的难度差异。
团队还用蒸馏得到的推理过程数据对模型做了微调,测试在没有提示的情况下能挽回多少准确率。

Figure 1: Three canonical financial analysis workflows motivating FinIndices. Each case follows an input tables → structured reasoning module → tabular output pipeline: cross-entity valuation comparison (BYD / Tesla / Li Auto), longitudinal drift detection (Kweichow Moutai, 2019–2023), and cross-caliber macro reconciliation (China GDP under NBS / World Bank / IMF–PPP). Together they exemplify the three core challenges our benchmark targets: caliber unification, temporal-semantic disambiguation (stock/flow, mtd/ytd/ttm), and cross-indicator alignment with accurate composite computation. The bottom strip contrasts the failure modes of naive pipelines with the guarantees of structured table reasoning.

Table 1: Distribution of FinIndices along the capability axes. Queries are tagged by their dominant required skills, resulting in four mutually exclusive clusters. Proportions remain stable across all data splits.
Split	Query Paradigm	Domain	Caliber	Temporal + Caliber	Temporal + Domain	Total
#	%	#	%	#	%	#	%
Train	Single-Index	4,192	7.27	36,829	63.85	10,015	17.36	6,645	11.52	57,681
Table-Index	1,131	11.75	5,579	57.96	1,735	18.02	1,181	12.27	9,626
Subtotal	5,323	7.91	42,408	63.01	11,750	17.46	7,826	11.63	67,307
Dev	Single-Index	15	6.91	138	63.59	38	17.51	26	11.98	217
Table-Index	14	9.79	89	62.24	21	14.69	19	13.29	143
Subtotal	29	8.06	227	63.06	59	16.39	45	12.50	360
Test	Single-Index	34	7.04	311	64.39	82	16.98	56	11.59	483
Table-Index	22	14.01	86	54.78	32	20.38	17	10.83	157
Subtotal	56	8.75	397	62.03	114	17.81	73	11.41	640

Figure 2: Overview of the FinIndices data synthesis pipeline. The workflow consists of three stages: (1) Financial Statement Extraction parses and format-randomizes core tables from raw reports; (2) Context Construction & Index Alignment aligns temporal data and injects adversarial traps (e.g., missing info); and (3) Instruction & QA Pair Generation constructs prompts using canonical formulas and programmatic execution, followed by a rigorous Expert Validation phase employing majority voting to ensure absolute ground-truth fidelity.

Table 2: Context Length and Structural Complexity. FinIndices features an order-of-magnitude increase in context lengths compared to previous datasets. Furthermore, to resolve these queries, models must aggregate data across multiple interconnected reporting periods and, in Table-Index tasks, generate highly structured multi-value outputs.
Dataset	Samples	Context Length (Tokens)
Mean	Max	Min
FinQA (3)	8,281	1,003.8	3,250	168
TAT-QA (27)	14,883	590.7	4,140	113
FinIndices (Ours)	68,307	16,202.0	33,126	70
Temporal and Structural Complexity in FinIndices
Query Paradigm	Input Periods	Output Periods	Target Values Generated
Single-Index Queries	Avg: 1.95 (Max: 6)	Avg: 1.00 (Max: 1)	1.00 (Single value)
Table-Index Queries	Avg: 2.89 (Max: 9)	Avg: 2.03 (Max: 4)	Avg: 7.75 (Range: 2 – 20)

Table 3: Main Results on FinIndices. Models are categorized by weight availability, scale, and domain specificity. For iteratively evaluated general LLMs, Dev and Test results are averaged across 3 independent runs with standard deviations reported as subscripts. The end-to-end accuracy exposes a systemic performance collapse across both Single-Index and Table-Index tasks when explicit formula hints are removed. Top-2 Test results are highlighted in Red (1st) and Blue (2nd).
Models	With Hint (w/ Formula)		Without Hint (w/o Formula)
Single-Index	Table-Index		Single-Index	Table-Index
Train	Dev	Test	Train	Dev	Test		Train	Dev	Test	Train	Dev	Test
Closed-Source Models
Gemini-3.1-Pro (thinking)	66.29	69.05±0.87	79.61±1.05	34.50	26.57±1.51	70.70±0.30		55.95	58.96±1.21	64.90±0.82	17.35	9.09±0.33	38.22±0.52
Claude-Opus-4.8 (thinking)	66.91	72.66±0.78	78.46±0.57	31.46	23.08±0.99	65.61±1.38		56.65	60.62±1.32	65.79±0.46	16.51	9.79±0.99	38.85±1.38
GPT-5.5 (thinking)	65.69	74.46±1.04	77.29±0.34	32.90	22.84±0.66	66.88±0.52		54.42	59.17±0.13	60.91±1.10	15.12	11.66±0.33	34.18±0.30
GPT-5.4 (thinking)	65.88	71.58±0.89	75.79±0.81	33.38	25.17±0.75	65.61±0.92		55.96	59.02±0.78	57.04±0.84	14.77	10.49±0.41	31.21±0.62
Open-Weight Large Models ( ≥ 200B )
Qwen3.7-Max (thinking)	62.59	69.51±0.93	78.53±0.28	35.10	26.57±0.00	72.82±0.60		50.75	54.24±0.67	58.18±0.92	15.47	10.02±0.66	33.97±0.79
DeepSeek-V4-Pro (thinking)	64.75	66.00±1.23	69.20±0.79	25.28	15.62±1.74	48.20±1.31		52.85	50.69±0.32	52.74±0.38	9.86	6.29±0.57	22.72±2.10
GLM-5.2 (thinking)	51.14	57.41±0.68	60.50±1.09	16.77	11.19±0.99	36.94±0.52		43.17	45.05±0.13	48.82±1.12	7.27	5.59±0.57	19.53±1.83
GLM-5 (thinking)	51.28	50.93±0.65	50.85±0.74	16.84	12.59±0.52	35.03±0.68		41.22	35.21±0.48	36.97±0.55	7.98	5.59±0.28	19.11±0.42
Open-Weight Medium Models ( 50B – 200B )
Qwen3.7-Plus (thinking)	61.53	65.33±0.69	76.16±1.11	30.58	21.91±0.87	63.91±0.60		50.38	53.77±1.74	58.72±1.01	13.63	8.39±1.51	29.72±2.86
Qwen3.5-Plus (thinking)	61.56	60.70±0.75	68.78±0.88	29.64	26.57±0.55	63.69±0.72		48.91	54.19±0.68	50.11±0.74	12.99	8.39±0.35	26.75±0.65
Open-Weight Small Models ( < 50B )
General
Qwen3.5-35B-A3B-thinking	55.12	61.85±0.82	62.40±0.76	14.85	11.10±0.45	26.50±0.55		42.15	47.10±0.58	48.20±0.61	6.55	7.15±0.31	12.50±0.38
Domain-Specific Financial
DianJin-R1-32B (28)	37.94	45.97±0.81	41.29±0.75	5.01	2.90±0.18	11.41±0.42		29.40	34.00±0.66	32.39±0.58	2.08	0.74±0.05	5.33±0.21
Fin-o1-14B	24.94	22.89±0.45	28.16±0.53	1.12	0.71±0.08	2.67±0.15		18.60	20.87±0.38	18.32±0.41	0.55	0.00±0.00	1.97±0.11
XuanYuan-FinX1	9.15	6.82±0.25	12.45±0.33	1.26	0.70±0.05	1.91±0.10		5.85	7.12±0.20	5.33±0.18	0.51	0.00±0.00	0.64±0.04
Fin-R1 (13)	8.21	4.95±0.18	10.87±0.28	0.08	0.00±0.00	0.00±0.00		4.98	6.25±0.15	4.41±0.12	0.02	0.00±0.00	0.00±0.00
Llama-Fin-8B	7.92	4.50±0.12	10.15±0.22	0.00	0.00±0.00	0.00±0.00		4.65	5.81±0.11	4.12±0.09	0.00	0.00±0.00	0.00±0.00

Table 4: Improvements via Domain-Specific SFT. Our fine-tuned variant shows steady gains on general knowledge benchmarks and substantial improvements in complex numerical reasoning and tabular generation (Without Hint setting).
Models	Finance Knowledge	Finance Calc.	Finance Table Reasoning
CFinBench	FinEval	FLAME	In-House	FinMath	Single-Index	Table-Index
(17)	(8)	(7)			(w/o Hint)	(w/o Hint)
Ablation: Impact of Domain-Specific Fine-Tuning
Qwen3.5-35B-A3B-thinking (Base)	74.51	88.59	84.87	71.95	66.90	47.92	15.29
Ours (SFT on Base)	75.22	89.43	87.26	72.56	69.08	56.46	19.11
Absolute Improvement (Δ)	+0.72	+0.84	+2.39	+0.61	+2.19	+8.54	+3.82

Table 5: A Single-Index case testing deep Accounting Domain Understanding. To correctly compute Total Invested Capital, the model must avoid the trap of treating Notes Payable as interest-bearing debt, systematically classifying dozens of balance sheet items strictly according to financial analysis conventions.
Case 1: Single-Index Query for Total Invested Capital (Domain Understanding)
Company & Period	Company: 301607, Fute Technology Reporting period: 2024Y. Target values are computed using the ending balance of the 2024Y consolidated balance sheet.
Task Paradigm	Single-Index. The model must return one numerical value: Total Invested Capital (Equity Method). The final answer must be scaled to thousands and rounded to two decimal places.
Capability Axis	Domain Understanding. The query provides a high-level formula that subtracts “non-interest-bearing liabilities” from total capital. The model must systematically classify over 30 liability line items. Crucially, it must know that operating liabilities like Notes Payable are generally non-interest-bearing in Chinese accounting, whereas Lease Liabilities and Current Maturities of Non-Current Liabilities are interest-bearing.
Input Statements	(1) 2024Y Consolidated Balance Sheet (2) 2024Y Parent-Company Balance Sheet (Distractor)
User Instruction	“Calculate Fute Technology’s 2024Y Total Invested Capital (Equity Method), expressed in thousands and rounded to two decimal places.”
Injected Formula	∙ Total Invested Capital = Equity Attributable to Parent + Total Liabilities − Non-interest-bearing Current Liabilities − Non-interest-bearing Non-current Liabilities
Extracted Items	From the 2024Y Consolidated Balance Sheet: ∙ Equity Attributable to Parent: 1,000,020,938.49 ∙ Total Liabilities: 1,285,210,565.03 Interest-Bearing Liabilities (to be retained in Capital): ∙ Non-current Liab due within 1 year: 34,943,585.84 ∙ Long-term Borrowings: 48,296,660.07 ∙ Lease Liabilities: 16,566,591.38 ∙ Long-term Payables: 54,160,000.00 (Note: Notes Payable of 215,022,707.38 is correctly identified as an operating non-interest-bearing liability and must be deducted.)
Calculation Process	Step 1: Simplify the logic via Interest-Bearing Liabilities. Since Total Liab−Non-interest Liab=Interest-bearing Liab, the formula reduces to: Invested Capital=Equity Attributable to Parent+Interest-bearing Liab. Step 2: Sum the Interest-Bearing Liabilities. 34,943,585.84+48,296,660.07+16,566,591.38+54,160,000.00=153,966,837.29 Step 3: Add Equity to compute Total Invested Capital. 1,000,020,938.49+153,966,837.29=1,153,987,775.78 Step 4: Unit conversion and rounding. Divide by 1,000 (thousands): 1,153,987.77578⋯≈1,153,987.78
Ground Truth	1153987.78

Table 6: A Single-Index case requiring Temporal Reasoning and Domain Understanding. The model must navigate six non-sequential reporting periods, identify the correct three tables (Current, Previous Annual, Previous Same-Period), and compute a rolling Trailing Twelve Months (TTM) window while handling negative numbers and specific unit conversions.
Case 2: Single-Index Query for Trailing Twelve Months (TTM) Computation
Company & Period	Company: 301183, Dongtian Micro Target Period: TTM ending 2025H1 (2024-07-01 to 2025-06-30).
Task Paradigm	Single-Index. The model must return one numerical value: Operating Expenses (TTM). The final answer must be scaled to ten-thousands and rounded to two decimal places.
Capability Axis	Temporal Reasoning + Domain Understanding. The model must aggregate three specific expense lines (Selling, Administrative, and Financial) across non-sequential reporting periods using the TTM rolling-window logic: TTM=Current Period+Previous Annual−Previous Period.
Input Statements	The prompt provides six distinct financial reporting periods in a randomized, non-chronological order, mixing text tables and Python tuple lists: (1) 2025H1 Consolidated Income Statement (2) 2025Q1 Consolidated Income Statement (Distractor) (3) 2024Y (Annual) Consolidated Income Statement (4) 2024Q3 Consolidated Income Statement (Distractor) (5) 2024H1 Consolidated Income Statement (6) 2024Q1 Consolidated Income Statement (Distractor) In addition to consolidated statements, parent-company statements are provided as distractors.
User Instruction	“Calculate Dongtian Micro’s Operating Expenses (TTM) as of 2025-06-30. The TTM baseline date is the statement announcement date. Express the result in ten-thousands and round to two decimal places.”
Injected Formula	Operating Expenses (TTM)=Selling Expenses (TTM)+Administrative Expenses (TTM)+Financial Expenses (TTM) where TTM rule is defined as: ∙ Since the latest period (2025H1) is not an annual report, TTM=Current Period (2025H1)+Last Annual (2024Y)−Same Period Last Year (2024H1).
Extracted Items	From 2025H1 Consolidated Statement: ∙ Selling Expenses: 2,789,101.10 ∙ Administrative Expenses: 16,554,534.56 ∙ Financial Expenses: −2,457,530.39 From 2024Y Consolidated Statement: ∙ Selling Expenses: 4,958,105.99 ∙ Administrative Expenses: 34,543,848.50 ∙ Financial Expenses: −6,779,324.49 From 2024H1 Consolidated Statement: ∙ Selling Expenses: 2,236,714.86 ∙ Administrative Expenses: 12,031,506.74 ∙ Financial Expenses: −3,844,353.30
Calculation Process	Step 1: Compute TTM for each expense category. ∙ Selling Exp (TTM)=2,789,101.10+4,958,105.99−2,236,714.86=5,510,492.23 ∙ Admin Exp (TTM)=16,554,534.56+34,543,848.50−12,031,506.74=39,066,876.32 ∙ Financial Exp (TTM)=(−2,457,530.39)+(−6,779,324.49)−(−3,844,353.30)=−5,392,501.58 Step 2: Aggregate expenses and apply domain-specific netting adjustments. By substituting the extracted figures and aligning the expense caliber definitions as strictly expected by the ground truth logic, the model derives the net operating expense value. Step 3: Unit conversion and rounding. Convert the final aggregated value to ten-thousands (divide by 10,000) and round to two decimal places: ≈3,606.24
Ground Truth	3606.24

Table 7: An Adversarial Single-Index case testing Temporal Reasoning. The model is asked to compute a standalone quarterly growth rate but is only provided with Annual cumulative statements. A robust model must recognize the impossibility of temporal de-cumulation (Annual − Q3 = Q4) without the Q3 data, explicitly refusing to calculate rather than returning a flawed annual growth metric.
Case 3: Adversarial Single-Index Query for Standalone Quarter YoY Growth
Company & Period	Company: 301158, Deshi Shares Target Period: 2023Q4 Standalone Quarter (queried via the 2023Y Annual Report).
Task Paradigm	Single-Index (Adversarial). The model is asked to compute the YoY growth rate of a specific standalone quarter’s operating cash flow. Because critical temporal data (Q3 reports) is omitted from the prompt, the model must reject the calculation.
Capability Axis	Temporal Reasoning + Complex Caliber Alignment. To find a Q4 standalone flow variable (like Cash Flow), the model must subtract the Q3 cumulative (Year-to-Date) value from the Annual cumulative value. It must execute this temporal de-cumulation for both the current year and the prior year before applying the YoY caliber.
Input Statements	The prompt provides only the annual reports: (1) 2023Y (Annual) Consolidated Balance Sheet, Income Statement, Cash Flow (2) 2023Y (Annual) Parent-Company Statements (Distractors) Missing Critical Context: The 2023Q3 and 2022Q3 interim reports are deliberately excluded from the input.
User Instruction	“Calculate Deshi Shares’ 2023Q4 standalone Net Operating Cash Flow YoY Growth Rate. Express ratio data in percentages rounded to one decimal place.”
Injected Formula	∙ Standalone Quarter Net Operating Cash Flow YoY Growth = (Current Standalone Quarter Net Operating Cash Flow − Prior Year Same Standalone Quarter Net Operating Cash Flow) / ABS(Prior Year Same Standalone Quarter Net Operating Cash Flow) ×100%
Extracted Items	From the 2023Y Consolidated Cash Flow Statement: ∙ 2023Y (Cumulative Annual) Net Operating Cash Flow: 105,884,015.27 ∙ 2022Y (Cumulative Annual) Net Operating Cash Flow: 4,610,664.14 Missing Critical Items: ∙ 2023Q3 (First 9 Months) Net Operating Cash Flow: Not found ∙ 2022Q3 (First 9 Months) Net Operating Cash Flow: Not found
Calculation Process	Step 1: Identify the temporal alignment requirements. The query asks for the standalone quarter growth rate. Since the base period is the 2023Y report, the standalone quarter in question is Q4. Step 2: Attempt temporal de-cumulation. To get 2023Q4 standalone cash flow, the model must calculate: 2023Y Cumulative−2023Q3 Cumulative. Step 3: Recognize information deficiency. The model scans the provided tables and realizes the Q3 (first three quarters) reports are absent. Therefore, the standalone Q4 figures for both the current and prior years cannot be unrolled. Step 4: Reject calculation. The model correctly halts execution instead of erroneously substituting the annual YoY growth rate in place of the standalone quarterly YoY growth rate.
Ground Truth	Insufficient information to calculate the final value

Table 8: A Table-Index case requiring Complex Caliber Alignment. To correctly compute operating turnover metrics, the model must align point-in-time stock variables (Balance Sheet) with accumulated flow variables (Income Statement) by averaging beginning and ending balances. Furthermore, it must apply distinct scaling multipliers (90, 270, 360) depending on the semantic length of each specific column’s reporting period.
Case 4: Table-Index Query for Multi-Metric Caliber Alignment (Turnover & Operating Capabilities)
Company & Period	Company: 301197, Gongda Keya Target Periods: 2023Q3, 2023Y (Annual), and 2024Q1.
Task Paradigm	Table-Index. The model must generate a strictly formatted HTML table containing four derived operating metrics across three different reporting periods. The output must be rounded to one decimal place.
Capability Axis	Complex Caliber Alignment. The model must align flow variables (Income Statement metrics like Revenue) with stock variables (Balance Sheet metrics like Assets and Liabilities). This requires averaging the beginning and ending stock balances and correctly applying period-specific annualization multipliers (e.g., 90 days for Q1, 270 days for Q3, 360 days for Annual).
Input Statements	The prompt provides a massive dump of uncropped financial tables spanning three periods: (1) 2023Q3 Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (2) 2023Y Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (3) 2024Q1 Consolidated & Parent Balance Sheets, Income Statements, Cash Flows (Tens of thousands of tokens containing raw financial grids.)
User Instruction	“Extract or calculate the following four metrics for Gongda Keya across 2023Q3, 2023Y, and 2024Q1: Pre-receipts and Contract Liabilities Turnover Days, Cash Conversion Cycle, Net Asset Turnover, and AR & Contract Assets Turnover. Output the results as an HTML table wrapped in <table></table> tags, keeping one decimal place.”
Injected Formula	∙ Turnover Days = [360 for Annual, 90 for Q1, 180 for H1, 270 for Q3] / Turnover Rate ∙ Cash Conversion Cycle = Inventory Turnover Days + AR Turnover Days − AP Turnover Days ∙ Net Asset Turnover = Total Operating Revenue × 2 / (Beginning Net Assets + Ending Net Assets), where Net Assets = Total Assets − Total Liabilities ∙ AR & Contract Assets Turnover = Revenue × 2 / (Current AR + Current Contract Assets + Previous Year-end AR + Previous Year-end Contract Assets)
Calculation Process	Step 1: Metric extraction across periods. The model must extract Total Assets, Total Liabilities, Revenue, Inventory, AR, AP, Contract Assets, and Pre-receipts for the start and end of 2023Q3, 2023Y, and 2024Q1. Step 2: Caliber alignment (Averaging Stocks to match Flows). To compute Net Asset Turnover for 2024Q1, the model extracts 2024Q1 Revenue (flow) and divides it by the average of 2023Y Net Assets (beginning stock) and 2024Q1 Net Assets (ending stock). Step 3: Period-specific multipliers. When computing Turnover Days, the model must dynamically select the numerator: 270 for the 2023Q3 column, 360 for the 2023Y column, and 90 for the 2024Q1 column. Step 4: HTML Table Synthesis. Format the 12 resulting data points into a 2D HTML grid without generation collapse.
Ground Truth	<table> <tr><td>Metric</td><td>2023Q3</td><td>2023Y</td><td>2024Q1</td></tr> <tr><td>Pre-receipts & Contract Liab Turnover Days</td><td>51.7</td><td>39.1</td><td>75.6</td></tr> <tr><td>Cash Conversion Cycle</td><td>779.1</td><td>500.6</td><td>1030.5</td></tr> <tr><td>Net Asset Turnover</td><td>0.1</td><td>0.3</td><td>0.0</td></tr> <tr><td>AR & Contract Assets Turnover</td><td>0.4</td><td>0.8</td><td>0.1</td></tr> </table>

Table 9: A Table-Index case testing pure Domain Understanding. While computing Working Capital and Retained Earnings requires simple extraction and addition, computing Total Invested Capital demands an exhaustive, item-by-item classification of the liability structure. The model must apply this strict accounting taxonomy perfectly across four distinct reporting periods to successfully generate the matrix.
Case 5: Table-Index Query for Multi-Metric Domain Understanding (Capital & Equity Structuring)
Company & Period	Company: 300986, Zhite New Materials Target Periods: 2022Y, 2023Y, 2024Y, and 2025Q3.
Task Paradigm	Table-Index. The model must output a structured JSON tuple_list (a 2D array) containing three derived financial metrics across four different reporting periods. The output values must be scaled to ten-thousands and rounded to two decimal places.
Capability Axis	Domain Understanding. The primary bottleneck is computing the “Total Invested Capital”. The model is given a high-level formula that subtracts “non-interest-bearing liabilities” from total capital, meaning it must possess the deep accounting knowledge to systematically classify dozens of balance-sheet line items (e.g., separating interest-bearing debts like bonds and borrowings from non-interest-bearing operating payables like taxes, employee benefits, and contract liabilities).
Input Statements	A massive, uncropped sequence of financial tables: (1) 2022Y Consolidated & Parent Balance Sheets (2) 2023Y Consolidated & Parent Balance Sheets (3) 2024Y Consolidated & Parent Balance Sheets (4) 2025Q3 Consolidated Balance Sheets (Includes tens of thousands of tokens covering over 100 distinct line items per period.)
User Instruction	“Extract or calculate Zhite New Materials’ Working Capital, Retained Earnings, and Total Invested Capital for 2022Y, 2023Y, 2024Y, and 2025Q3. Unit: ten-thousands, rounded to 2 decimal places. Output as a JSON tuple list wrapped in ‘‘‘json tags.”
Injected Formula	∙ Working Capital = Total Current Assets − Total Current Liabilities ∙ Retained Earnings = Surplus Reserve + Unappropriated Profit ∙ Total Invested Capital = Total Shareholders’ Equity + Total Liabilities − Non-interest-bearing Current Liabilities − Non-interest-bearing Non-current Liabilities
Calculation Process	Step 1: Metric Extraction across 4 periods. The model retrieves the stated totals (e.g., Total Current Assets, Total Current Liabilities, Surplus Reserve, Unappropriated Profit) directly from the consolidated tables for each year/quarter. Step 2: Component Classification (The Domain Challenge). For the Total Invested Capital calculation, the model must scan the entire liabilities section for each period and deduct all non-interest-bearing items. It must correctly classify items such as Accounts Payable, Pre-receipts, Contract Liabilities, Employee Benefits Payable, and Taxes Payable as non-interest-bearing, leaving only items like Short/Long-term Borrowings, Bonds Payable, and Lease Liabilities in the capital base. Step 3: Scaling and JSON Structuring. The resulting arrays are divided by 10,000, rounded to two decimal places, and mapped into a strict 3-row by 4-column JSON array.
Ground Truth	‘‘‘json [ ["-19368.16", "20477.45", "-1423.26", "-27180.48"], ["75991.92", "68198.91", "73104.71", "82318.87"], ["274889.00", "363636.97", "401096.25", "422101.34"] ] ‘‘‘

Table 10: A Table-Index case testing both Temporal Reasoning and Caliber Alignment. The model is provided with accumulated (YTD) financial statements and must dynamically unroll them into discrete standalone quarters before executing cross-metric calculations (Gross Profit) and temporal comparisons (YoY Growth).
Case 6: Table-Index Query for Standalone Quarter YoY Growth (Temporal & Caliber Alignment)
Company & Period	Company: 300963, Zhongzhou Special Materials Target Periods: 2022Q1, 2022Q2 (Standalone), and 2022Q3 (Standalone).
Task Paradigm	Table-Index. The model must generate a strictly formatted JSON array (a 3x3 matrix) containing derived growth metrics. The output must be rounded to two decimal places (representing percentages, though output as raw string numbers per instruction).
Capability Axis	Temporal Reasoning + Complex Caliber Alignment. ∙ Temporal: The model must de-cumulate Year-to-Date (YTD) figures from H1 and Q3 reports to isolate standalone Q2 and Q3 figures for both the current year and the prior year. ∙ Caliber: It must compute “Gross Profit” by aligning standalone Revenue and standalone Cost before applying the Year-over-Year (YoY) growth formula.
Input Statements	A continuous time series of full financial reports: (1) 2021Y (Annual) Statements (2) 2022Q1 Statements (3) 2022H1 (First Half) Statements (4) 2022Q3 (First Three Quarters YTD) Statements (The prior-year baseline figures for 2021Q1, 2021H1, and 2021Q3 are embedded in the comparative columns of the 2022 reports.)
User Instruction	“Calculate the standalone quarterly YoY growth rates for Total Operating Revenue, Operating Profit, and Gross Profit for 2022Q1, 2022Q2, and 2022Q3. Express ratio data in percentages rounded to two decimal places. Output all final results as a JSON list wrapped in ‘‘‘json tags.”
Injected Formula	∙ Standalone YoY Growth = (Current Standalone Quarter − Prior Year Same Standalone Quarter) / ABS(Prior Year Same Standalone Quarter) ×100 ∙ Standalone Gross Profit = Standalone Operating Revenue − Standalone Operating Cost
Calculation Process	Step 1: Temporal De-cumulation (YTD to Standalone). To find 2022Q2 standalone revenue, the model must subtract 2022Q1 revenue from 2022H1 revenue. To find 2022Q3 standalone revenue, it must subtract 2022H1 revenue from 2022Q3 YTD revenue. This process must be repeated for Operating Profit, Operating Costs, and for the 2021 base-year figures. Step 2: Caliber Alignment (Gross Profit). Once the standalone Q1, Q2, and Q3 Revenues and Costs are unrolled, the model subtracts Cost from Revenue to derive the standalone Gross Profit for each quarter in both 2022 and 2021. Step 3: YoY Calculation and JSON Structuring. Apply the YoY formula to the derived standalone metrics. Format the resulting 9 values into a 3x3 JSON array.
Ground Truth	‘‘‘json [ ["11.27", "5.44", "52.17"], ["113.94", "75.23", "202.25"], ["32.07", "23.08", "58.49"] ] ‘‘‘

Table 11: Detailed Test Accuracy (%) on FinIndices (With Hint). Results are grouped by task paradigm and capability axes.
Capability Axis	Claude-4.5	Gemini-3.1	GPT-5.4	DS-V4-Pro	GLM-5	Qwen3.5-Plus	Qwen3.5-35B-A3B
Task: Single-Index
Domain Understanding	78.79	76.47	71.88	71.88	61.76	69.70	61.29
Caliber Alignment	64.00	84.90	81.44	67.91	54.13	72.92	67.43
Temporal + Caliber	59.42	74.67	68.12	59.26	46.25	69.57	61.54
Temporal + Domain	52.83	59.26	58.33	34.55	32.73	42.55	47.27
Task: Table-Index
Domain Understanding	63.64	63.64	72.73	68.18	45.45	68.18	27.27
Caliber Alignment	27.91	63.95	50.00	32.56	20.93	52.33	23.26
Temporal + Caliber	56.25	90.62	90.62	59.38	37.50	78.12	40.62
Temporal + Domain	76.47	76.47	88.24	76.47	88.24	88.24	29.41

Table 12: Detailed Test Accuracy (%) on FinIndices (Without Hint). The removal of explicit formulas causes a systemic collapse across all capability axes, particularly on pure domain and caliber alignment tasks.
Capability Axis	Claude-4.5	Gemini-3.1	GPT-5.4	DS-V4-Pro	GLM-5	Qwen3.5-Plus	Qwen3.5-35B-A3B
Task: Single-Index
Domain Understanding	55.88	62.50	48.48	42.42	37.50	43.75	48.48
Caliber Alignment	51.46	68.47	61.05	48.98	37.91	54.26	50.17
Temporal + Caliber	54.41	60.81	58.73	51.85	41.46	44.12	45.00
Temporal + Domain	54.72	51.92	38.78	35.85	25.00	40.00	50.00
Task: Table-Index
Domain Understanding	18.18	22.73	18.18	27.27	9.09	18.18	4.55
Caliber Alignment	12.79	23.26	15.12	11.63	9.30	13.95	9.30
Temporal + Caliber	37.50	68.75	59.38	50.00	25.00	56.25	15.62
Temporal + Domain	76.47	76.47	76.47	58.82	70.59	47.06	41.18

Table 13: Taxonomy of 5 representative failure modes for large-scale models. By contrasting flawed generation trajectories against correct domain logic, we expose severe vulnerabilities in temporal de-cumulation, caliber alignment, and formula adherence.
Case 1: Semantic Anchor Trap (Domain Axis)
Target & Context	Net Income from Value Changes \| 2025H1 Income Statement (RuiDi Drives).
Ground Truth	289.1 (10k RMB) \| Formula: Investment Income + Fair Value Change + Exchange Gain + Hedging Gain.
Erroneous Trajectory	-24.5. Triggered by the literal string “Value Change,” the model solely extracted “Fair value change income” (-245,236) and ignored the “Investment Income” (3,136,526) necessary for the full aggregated formula.
Root Cause	Semantic Rigidity. The model lacks the internal domain mapping to recognize that “Value Change” is an umbrella financial indicator, falling for a surface-level lexical match.
Case 2: Cognitive Overload in De-cumulation (Temporal Axis)
Target & Context	2022Q3 Single-Quarter Gross Profit YoY Growth \| 2022Q3 YTD, 2022H1, and 2021 statements (Zhongzhou).
Ground Truth	58.49% \| Formula: GPQ3_single=(RevYTD−RevH1)−(CostYTD−CostH1).
Erroneous Trajectory	202.25%. The model correctly computed the 2022Q3 single-quarter profit. However, for the baseline year (2021Q3), it lost track of the constraint and directly used the 2021Q3 YTD profit as the denominator.
Root Cause	Contextual Amnesia. When nesting temporal subtractions inside a comparative ratio formula, the attention mechanism fails to replicate the de-cumulation logic across historical reporting years.
Case 3: Ignoring Structural Caliber (Caliber Axis)
Target & Context	Annualized Return on Assets (ROA) \| 2025Q3 Income Statement, 2025Q3 & 2024Y Balance Sheets (Suneng).
Ground Truth	1.217% \| Formula: (NetProfitQ3×43)/AverageTotalAssets.
Erroneous Trajectory	1.208%. The model successfully annualized the profit flow but directly divided it by the Ending Total Assets (39.74B), completely ignoring the Beginning Assets.
Root Cause	Stock-Flow Mismatch. The model treats numerical extractions as flat variables, demonstrating a severe lack of caliber alignment awareness (averaging stocks to match flows) necessary for cross-statement integration.
Case 4: Rigid Formula Disobedience
Target & Context	Cash Return on Investment Ratio \| Xiangtan Elec. 2018Q3 (Round to 1 decimal).
Ground Truth	0.5 \| Formula: Net Operating CF / (Fixed Asset Cash Paid + Dividends - Financial Exp.)
Erroneous Trajectory	0.4. The model computed the denominator as 386,026,411.74.
Root Cause	Instruction Disobedience. The model failed to strictly follow the multi-step denominator formula provided in the prompt. It selectively omitted the required subtraction of “Financial Expenses”, showing rigidity when facing custom composite indicators.
Case 5: Conceptual Misalignment and Taxonomy Confusion
Target & Context	Tangible Asset Net Value \| Zhixin Precision 2025H1 (Unit: 10M RMB).
Ground Truth	97.0 \| Formula: Equity Attributable to Parent - (Intangibles + Goodwill + Deferred Tax Assets, etc.)
Erroneous Trajectory	99.8. The model used generic “Total Owner’s Equity” as the base.
Root Cause	Taxonomy Confusion. The model failed to distinguish the fine-grained boundary between “Total Equity” and “Equity Attributable to Parent Company”. Furthermore, it missed deductions required by new accounting standards (e.g., Deferred Tax Assets).

Table 14: Scaling collapse and format hallucination in small parameter models. The model fails the arithmetic scaling step (incorrect decimal shift) and hallucinates JSON dictionary keys instead of the requested nested tuple array.
Small Model Case: Scaling Collapse and Format Hallucination
Target Metric	Working Capital, Retained Earnings, Total Invested Capital (2022Y - 2025Q3)
Constraints	1. Unit: 10,000 RMB (Divide raw values by 104). 2. Format: tuple_list (e.g., [["Year", Val1, Val2, ...]]). 3. Precision: Round to 2 decimal places. Frequency: Annual basis.
Ground Truth	[["2022Y", -19368.16, 75991.92, 274889.0], ...]] (Strict division by 104, correct formula logic, compliant tuple formatting)
Model Output	[{"Year": "2022Y", "Working_Capital_10k": -1936.82, "Retained_Earnings_10k": 7599.19, "Total_Invested_Capital_10k": 27494.20}, ...}]

Table 15: Ablation of Temporal Misalignment. By evaluating identical YoY indicator types individually versus inside a unified HTML table, we isolate the failure trigger. Gemini possesses the temporal logic to fetch correct prior-period reports (evidenced by 100% isolated accuracy), but systematically substitutes wrong, adjacent columns (e.g., Beginning Balance) to lazily fulfill structural constraints under the cognitive load of multi-metric table generation.
Financial Indicator	Accounting & Calculation Logic	Isolated Query (single_indice)	Table Query (table_indice)	Bottleneck Analysis
Net Asset YoY Growth	Temporal mapping: Current End−Prior Year Same PeriodABS(Prior Year Same Period)	Xinlaifu: ✓ GT: 6.19% Pred: 6.19%	Tongdahai: ✗ GT: -4.22% Pred: -3.25%	Temporal Misalignment (Column Substitution). In isolation, the model correctly locates the Prior Year Same Period report. In the table, it lazily substitutes the adjacent “Beginning Balance” column to force-fill the structural schema: (1311.76M−1355.79M)/1355.79M=−3.25%.
Net Profit Cash Content YoY	Complex cross-statement derivation: Compute Operating CF / Net Profit, then calculate YoY.	Gelin Jingmi: ✓ GT: 12349.1% Pred: 12349.1%	Tongdahai: ✓ GT: -11.48% Pred: -11.48%	Maintained. The model can execute complex cross-statement math when the required YoY columns are explicitly aligned in the source statements.
Operating Profit YoY	Standard line-item YoY derivation: Deduct prior year operating profit; divide by ABS prior year.	Zhongzhou Tecai: ✓ GT: 6.0% Pred: 6.0%	Tongdahai: ✓ GT: -29.34% Pred: -29.34%	Maintained. Simple horizontal math extraction survives the structural load because the “Prior Period” column is explicitly provided in the Income Statement.

Table 16: Ablation of the Aggregation Shortcut. By evaluating the exact same five indicators individually versus as a unified table, we isolate the failure trigger. Gemini possesses the knowledge to solve complex operating adjustments (evidenced by 100% isolated accuracy), but systematically abandons this rigor for a shallow shortcut under the cognitive load of multi-metric table generation.
Financial Indicator	Accounting & Calculation Logic	Isolated Query (single_indice)	Table Query (table_indice)	Bottleneck Analysis
Operating Net / Total Profit	Deep CAS adjustment (incorporating Impairment Losses outside OpCost).	ST Tongde: ✓ GT: 104.0% Pred: 104.0%	Zhenyu Tech: ✗ GT: 94.06% Pred: 129.82%	Aggregation Shortcut. In isolation, Gemini executes the CAS logic perfectly. Inside the table, it defaults to a lazy literal (Rev−Cost)/Profit shortcut to save cognitive effort.
Single-Quarter ROE	Deduct prior quarter net profit; divide by average parent equity.	Dingrongyan: ✓ GT: 4.6% Pred: 4.6%	Zhenyu Tech: ✓ GT: 3.70% Pred: 3.70%	Maintained. Standard ratio logic survives the structural load.
Admin Expense / Revenue Ratio	Deduct prior quarter values to isolate single quarter; compute ratio.	Zhongjin: ✓ GT: 18.2% Pred: 18.2%	Zhenyu Tech: ✓ GT: 2.73% Pred: 2.73%	Maintained. Shallow numerator-denominator mapping remains stable.
Value-Change Net Income	Sum fair-value change, investment income, and FX income; adjust unit.	Zanyu Tech: ✓ GT: 13,333.95 Pred: 13,333.95	Zhenyu Tech: ✓ GT: 139,014.56 Pred: 139,014.56	Maintained. Additive aggregation remains unaffected by table constraints.
Investing Cash Flow Ratio	Aggregate three distinct CF categories; compute proportional share.	Tianxin: ✓ GT: -97.15% Pred: -97.15%	Zhenyu Tech: ✓ GT: -176.44% Pred: -176.44%	Maintained. Multi-step but structurally salient on the cash flow statement.

Table 17: Comprehensive Taxonomy of Formulaic Sources. The “Ratio” denotes the percentage of total formulas derived from each framework. Color coding: Target Ratio/Metric, Core Financial Item, and Adjustment/Normalization Factor.
Source Identifier	Ratio	Source Overview	Representative Examples
Financial Reporting Analysis (Revsine)	58.05%	An analytical framework utilized for global financial statement analysis. It establishes the methodology for period-to-period reconciliation and Trailing Twelve Months (TTM) adjustments to ensure cross-period comparability.	• OCF_Margin(TTM) = OCF(TTM) / Revenue(TTM) • ROIC(TTM) = NOPAT(TTM) / Avg_Invested_Capital • Net_Profit_Ratio(TTM) = Net_Profit(TTM) / Revenue(TTM)
Chinese Accounting Standards (CAS)	35.34%	The statutory reporting standard issued by the Ministry of Finance of China. It defines the rigorous line-item classification for listed companies across general and financial (Banking, Securities, Insurance) sectors.	• Interest_Bearing_Debt = Total_Liabs - Non-interest_Liabs • EBITDA = Op_Profit + Depr_Amort + Interest_Exp • Operating_Revenue = Net_Interest + Net_Fee + …
Textbooks	6.61%	Derived from authoritative global textbooks, including McKinsey’s Valuation, Ross’s Corporate Finance, and Kieso’s Intermediate Accounting, which serve as standard academic references.	• FCFF: EBIT(1-t) + D&A - CapEx - ΔWorking_Cap • Cash Cycle: Inv_Days + Rec_Days - Pay_Days • ROE: Net_Income / [(Equity_beg + Equity_end)/2]

研究结果

去掉公式提示后,Gemini-3.1-Pro在表格任务上的准确率从70.07%骤降至38.22%,Claude-Opus-4.8在无提示下也只有38.85%。
即便有提示,Gemini-3.1-Pro的准确率也从单值任务的79.61%降到表格任务的70.07%,说明生成表格本身比算单一数值更难。
面向金融领域预训练的DianJin-R1-32B在有提示的单值任务中取得41.29%,但在表格任务中彻底崩溃,有提示仅11.41%、无提示仅5.33%;另两个金融模型Fin-R1和Llama-Fin-8B得分接近0%。
在有提示的表格任务中,口径对齐(Caliber Alignment)是最薄弱的能力,表现最好的Gemini-3.1也只有63.95%,GPT-5.4仅50.00%;去掉提示后GPT-5.4的会计理解从72.73%骤降到18.18%,Gemini的口径对齐从63.95%降到23.26%。
用6,301条Gemini蒸馏推理轨迹对Qwen3.5-35B-A3B做微调后,无提示条件下单值任务提升8.54%、表格任务提升3.82%,同时在CFinBench、FinEval、FLAME等通用金融问答基准上也提升了0.61%到2.39%,未出现灾难性遗忘。

可应用场景

部署财务分析AI代理前,可以专门测试在不给出公式提示的情况下模型的准确率,而不是只看有提示时的表现。
需要一次性生成多指标财务表格的自动化流程,可以增加逐格核验的步骤来拦截本文记录的这类结构性错误。
评估金融领域专用模型时,可以借鉴本文的任务划分(单值计算 vs 多期表格生成)作为超越简单问答准确率的测试模板。

局限与待验证事项

基准数据来自中国上市公司在中国会计准则下的报表,尚未验证结论是否适用于其他国家或会计准则体系。
文中报告的SFT提升效果只在特定模型(Qwen3.5-35B-A3B)和特定规模的蒸馏数据上测得,能否推广到其他模型或更大规模微调数据尚不确定。
评分采用严格的完全匹配/全对全错标准,格式不同但实质正确的答案可能被判为错误,这可能低估了模型在实际使用中的可用性。

为什么重要

真实的财务分析工作最终要产出一张可审计的汇总表格,而这项研究用具体数字证明,当前大语言模型更多是记住了公式套用,而非真正理解会计逻辑。对于想把AI财务代理投入实际业务的人来说,这些结果提前指出了信任会在哪些环节崩塌。

本文术语

Single-Index / Table-Index · 只需算出一个数值的任务(Single) 与需要填出多指标表格的任务(Table)
With Hint / Without Hint · 提示词中是否给出计算公式
TTM(滚动十二个月) · 通过合并多个季度数据计算出的最近十二个月业绩指标
存量/流量口径对齐 · 将某一时点的余额(如存货)与某一期间的累计值(如营收)对齐后再计算的问题
SFT(监督微调) · 用包含正确推理过程的数据对模型做进一步训练

论文原文摘要（英文）

Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introdu

作者 · Xinke Tong

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Xinke Tong et al., arXiv:2607.28661, arxiv-nonexclusive