Reliable Financial Named Entity Recognition under Domain Shift

arXiv:2608.195582026-08-21

An AI's confidence trained on formal filings turns unreliable once it reads tweets

Researchers took named-entity recognition AI trained on SEC filings and tested it on financial news and general social media, finding that not just accuracy but the AI's own confidence scores broke down under this shift. Whole-sentence probability, the best confidence signal in-domain, became untrustworthy out of domain, while entity-span probability and self-consistency (agreement across repeated generations) held up better. Still, on the extreme social-media shift, no confidence signal could reliably identify a safe subset of predictions to automate.

What they did

Trained a BERT tagger and small language models (Qwen2.5 0.5B/1.5B fine-tuned with LoRA) on SEC filings, then tested them on financial news and general-topic Twitter data across three seeds with bootstrap confidence intervals
Compared five inference-time confidence signals: whole-output probability, token probability, entity-span probability, entity-type probability, and self-consistency across repeated samples
Whole-output probability was the strongest error detector in-domain but degraded sharply under domain shift; entity-span probability and self-consistency were more robust, with self-consistency also better calibrated without any post-hoc fixing
Letting the AI automatically answer only the top 40% most-confident sentences cut in-domain sentence error from 34.3% to under 2%, still helped on financial news, but found no usefully large safe subset on the extreme social-media data
Proposes a staged deployment approach: detect severe domain shift first, then apply confidence-based automation only within domains the system can reliably handle

Fig. 1: Reliability diagrams for self-consistency confidence (0.5B, pooled seeds). Near-diagonal in-domain; on both shifted tiers the top-confidence bin inverts.

TABLE I: Dataset statistics after harmonization and filtering.
Split	Domain	Sent.	Ent.	PER	ORG	LOC
Train	FIN (filings) [27]	1014	980	648	175	157
Valid	FIN (filings) [27]	150	177	97	68	12
Test	FIN (filings) [27]	299	295	201	56	38
Test	FiNER-ORD (news) [28]	300	322	78	151	93
Test	TweetNER7 (tweets) [30]	300	619	381	135	103

Fig. 2: Sentence-level risk–coverage (Qwen2.5-0.5B; line = 3-seed mean, band = seed range; lower is better). Abstention sharply reduces risk in-domain and on news, but cannot rescue the far-shift tier.

TABLE II: Main results (%; mean±std over 3 seeds; Qwen2.5-1.5B is a single-seed scale check).
Model	Domain	P	R	F1	Halluc.%	ECE
BERT-base	FIN	66.7±2.3	74.2±1.4	70.3±1.5	–	0.102±0.016
BERT-base	FiNER-ORD	39.1±0.3	37.6±2.3	38.3±1.1	–	0.071±0.017
BERT-base	TweetNER7	35.2±5.2	18.4±1.7	24.1±2.5	–	0.074±0.007
Qwen-0.5B	FIN	40.5±0.4	34.7±2.1	37.4±1.3	3.6±2.4	0.362±0.009
Qwen-0.5B	FiNER-ORD	30.7±1.3	18.0±4.4	22.6±3.8	8.2±0.7	0.407±0.005
Qwen-0.5B	TweetNER7	30.8±0.7	23.7±4.2	26.7±2.8	5.6±1.1	0.370±0.007
Qwen-1.5B	FIN	47.7	44.7	46.2	4.3	0.322
Qwen-1.5B	FiNER-ORD	56.3	48.4	52.1	3.6	0.227
Qwen-1.5B	TweetNER7	44.9	50.9	47.7	6.8	0.358

TABLE III: Error-detection AUROC and ECE per confidence signal (Qwen2.5-0.5B; mean±std over 3 seeds), by test domain.
Signal	FIN	FiNER-ORD	TweetNER7
	AUROC	ECE	AUROC	ECE	AUROC	ECE
Sequence prob.	0.839±0.014	0.512±0.005	0.661±0.015	0.593±0.011	0.626±0.019	0.586±0.006
Token prob.	0.823±0.012	0.531±0.004	0.666±0.019	0.615±0.013	0.605±0.018	0.611±0.006
Span prob.	0.801±0.004	0.362±0.009	0.690±0.030	0.407±0.005	0.721±0.011	0.370±0.007
Type prob.	0.492±0.019	0.506±0.004	0.536±0.022	0.533±0.019	0.511±0.010	0.573±0.002
Self-consistency	0.690±0.019	0.116±0.029	0.671±0.048	0.120±0.015	0.682±0.017	0.100±0.008

TABLE IV: Scale check: error-detection AUROC by signal and domain for Qwen2.5-0.5B (3-seed mean) vs. Qwen2.5-1.5B (single seed).
Signal	0.5B (3 seeds)	1.5B (1 seed)
	FIN	FiNER	Tweet	FIN	FiNER	Tweet
Sequence prob.	0.839	0.661	0.626	0.764	0.720	0.637
Span prob.	0.801	0.690	0.721	0.819	0.729	0.638
Self-consistency	0.690	0.671	0.682	0.735	0.744	0.663

TABLE V: Entity-level selective precision (%) at 100/80/60% coverage (Qwen2.5-0.5B; mean±std over 3 seeds).
Domain	Signal	P@100%	P@80%	P@60%
FIN	Seq. prob.	40.5±0.4	49.0±0.8	58.4±0.6
FIN	Span prob.	40.5±0.4	49.2±0.3	56.4±0.6
FIN	Self-consistency	40.5±0.4	47.7±0.4	52.8±1.0
FiNER	Seq. prob.	30.7±1.3	34.9±3.0	40.1±1.3
FiNER	Span prob.	30.7±1.3	35.3±1.7	40.8±2.6
FiNER	Self-consist.	30.7±1.3	34.9±3.0	41.4±2.3
Tweet	Seq. prob.	30.8±0.7	34.2±1.0	36.8±2.9
Tweet	Span prob.	30.8±0.7	36.6±1.1	42.7±1.1
Tweet	Self-consist.	30.8±0.7	35.5±1.7	40.5±1.5

TABLE VI: TweetNER7 filtering sensitivity, main (drop-sentence) vs. relaxed (ignore-entity) protocol. Encoder: 3-seed mean; Qwen: seed 42.
	Filtered	Relaxed
Sentences dropped (of 2807)	2179	0
Evaluated (cap 300)	300	300
Gold PER/ORG/LOC	619	443
Ignored entities	0	488
Encoder F1	24.1±2.5	19.5±1.0
Encoder MSP AUROC	0.615±0.054	0.650±0.047
Qwen-0.5B F1	29.7	29.1
Qwen-0.5B span AUROC	0.725	0.715

Why it matters

In finance, where extraction errors feed into compliance and risk pipelines, accuracy scores alone can't tell you when it's safe to trust an AI's output automatically. This study shows that an AI's self-reported confidence can itself fail silently under domain shift, and offers concrete guidance on which confidence signals to trust and when.

Terms in this paper

Named Entity Recognition (NER) · an AI task that finds and labels names of people, organizations, and locations in text
distribution shift · when the style or topic of real-world input text differs from the data the AI was trained on
selective prediction · letting an AI withhold an answer when unsure and send that case to a human reviewer instead
self-consistency · a confidence measure based on how often an AI gives the same answer across multiple generated outputs for the same input
LoRA · a lightweight fine-tuning method that adjusts only a small set of added parameters instead of retraining the whole model
AUROC / ECE · AUROC measures how well a confidence score separates correct from incorrect predictions; ECE measures how well the stated confidence matches actual accuracy

Original abstract (English)

Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, while standard F1 scores do not indicate which predictions remain safe to automate when the input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition. We evaluate a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models using five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under distribution shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.

Authors · Zihao Zheng, Baichuan Li, Junyi Yao, Jiayu Long

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Zihao Zheng et al., arXiv:2608.19558, arxiv-nonexclusive