One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems

arXiv:2608.185342026-08-20

AI often names the right cause of a financial mismatch without ever finding the proof for it

A new benchmark called FinRCA-Bench tests AI systems that diagnose why company transactions fail to reconcile across invoices, payments, and bank records. Swapping only the retrieval method under the same language model pushed accuracy from 2.05% to 72.44%, showing that finding the right records mattered far more than the model's reasoning. Even correct answers were frequently unsupported: strict evidence-contract accuracy was just 5.72%.

What they did

  1. The benchmark uses 2,250 synthetic accounts-payable-to-bank reconciliation cases across 14 operational tables, including 1,500 cases with injected failures across 15 cause categories and 750 legitimate or tricky 'hard negative' cases.
  2. Instead of retrieving records by text similarity (Dense RAG), the study introduces Typed Provenance Graph Retrieval (TPGR), which follows only real, persisted transaction relationships between records, and compares both under the identical downstream reasoning model.
  3. Changing only the retrieval method raised macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%, while structured baselines (rule-based system: 84.97%, classical machine learning: 95.44%) already outperformed the LLM-based system.
  4. Of 437 cases where the model got the right answer, 311 (71.17%) did so without meeting the required evidence contract, and only 5.72% passed the strictest evidence check; 95 of the remaining errors were traced to retrieval failure versus only 15 to genuine reasoning failure.

Why it matters

For anyone deploying AI in financial operations, a correct answer alone doesn't mean the system actually found the records that justify it, which matters when auditors or finance teams need to trust and trace a diagnosis. This work gives concrete numbers showing that retrieval quality, not model intelligence, is often the real bottleneck.

Terms in this paper

  • RAG (retrieval-augmented generation) · a method where an AI first fetches relevant records before generating an answer
  • TPGR (Typed Provenance Graph Retrieval) · a retrieval method that follows only predefined, real transaction links between records instead of using similarity scoring
  • hard negative · a case that looks suspicious but is actually legitimate, included to test whether the system overreacts
  • macro recall · the average, across all cases, of how completely the needed records were retrieved
  • provenance · the traceable chain showing where a record came from and how it genuinely connects to others

Original abstract (English)

Large language models are increasingly used to support financial operations, but their apparent reasoning performance can depend on whether they receive the right evidence. In financial reconciliation, the evidence needed for diagnosis is distributed across invoices, purchase orders, approvals, allocations, payments, ledger entries, and bank activity, linked by transactional relationships rather than textual similarity. End-to-end accuracy can ther

Authors · Pratik Ghawate

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB