Reliable Financial Named Entity Recognition under Domain Shift

arXiv:2608.195582026-08-21

AI在正式文件里学到的自信,一到推特上就变得不可信

研究者用美国SEC财务文件训练的命名实体识别AI,拿去测试财经新闻和普通社交媒体文本,发现不仅准确率下降,连AI自己给出的置信度也开始失灵。在训练领域内表现最好的整句概率置信度,一旦领域变化就变得不可靠,而只看实体片段的概率以及多次生成结果的一致性(自洽性)则更加稳健。但在极端的社交媒体场景下,没有任何置信度指标能可靠地找出一批可以放心自动处理的预测。

他们做了什么

用SEC财务文件训练BERT标注器和经LoRA微调的小型语言模型(Qwen2.5 0.5B/1.5B),再拿到财经新闻和普通话题推特数据上测试,重复三个随机种子并计算自助法置信区间
比较了五种推理时置信度信号:整句输出概率、词元概率、实体片段概率、实体类型概率,以及基于多次采样的自洽性
整句概率在领域内是最强的错误检测信号,但一旦领域偏移就明显退化;实体片段概率和自洽性更稳健,其中自洽性无需额外校准就更准确地反映真实正确率
只自动处理置信度最高的40%句子时,领域内句子错误率从34.3%降到2%以下,在财经新闻上依然有效,但在极端社交媒体数据上找不到足够大的可靠子集
据此提出分阶段部署思路:先检测输入是否发生严重领域偏移,再仅在可靠领域内使用置信度来决定是否自动放行结果

Fig. 1: Reliability diagrams for self-consistency confidence (0.5B, pooled seeds). Near-diagonal in-domain; on both shifted tiers the top-confidence bin inverts.

TABLE I: Dataset statistics after harmonization and filtering.
Split	Domain	Sent.	Ent.	PER	ORG	LOC
Train	FIN (filings) [27]	1014	980	648	175	157
Valid	FIN (filings) [27]	150	177	97	68	12
Test	FIN (filings) [27]	299	295	201	56	38
Test	FiNER-ORD (news) [28]	300	322	78	151	93
Test	TweetNER7 (tweets) [30]	300	619	381	135	103

Fig. 2: Sentence-level risk–coverage (Qwen2.5-0.5B; line = 3-seed mean, band = seed range; lower is better). Abstention sharply reduces risk in-domain and on news, but cannot rescue the far-shift tier.

TABLE II: Main results (%; mean±std over 3 seeds; Qwen2.5-1.5B is a single-seed scale check).
Model	Domain	P	R	F1	Halluc.%	ECE
BERT-base	FIN	66.7±2.3	74.2±1.4	70.3±1.5	–	0.102±0.016
BERT-base	FiNER-ORD	39.1±0.3	37.6±2.3	38.3±1.1	–	0.071±0.017
BERT-base	TweetNER7	35.2±5.2	18.4±1.7	24.1±2.5	–	0.074±0.007
Qwen-0.5B	FIN	40.5±0.4	34.7±2.1	37.4±1.3	3.6±2.4	0.362±0.009
Qwen-0.5B	FiNER-ORD	30.7±1.3	18.0±4.4	22.6±3.8	8.2±0.7	0.407±0.005
Qwen-0.5B	TweetNER7	30.8±0.7	23.7±4.2	26.7±2.8	5.6±1.1	0.370±0.007
Qwen-1.5B	FIN	47.7	44.7	46.2	4.3	0.322
Qwen-1.5B	FiNER-ORD	56.3	48.4	52.1	3.6	0.227
Qwen-1.5B	TweetNER7	44.9	50.9	47.7	6.8	0.358

TABLE III: Error-detection AUROC and ECE per confidence signal (Qwen2.5-0.5B; mean±std over 3 seeds), by test domain.
Signal	FIN	FiNER-ORD	TweetNER7
	AUROC	ECE	AUROC	ECE	AUROC	ECE
Sequence prob.	0.839±0.014	0.512±0.005	0.661±0.015	0.593±0.011	0.626±0.019	0.586±0.006
Token prob.	0.823±0.012	0.531±0.004	0.666±0.019	0.615±0.013	0.605±0.018	0.611±0.006
Span prob.	0.801±0.004	0.362±0.009	0.690±0.030	0.407±0.005	0.721±0.011	0.370±0.007
Type prob.	0.492±0.019	0.506±0.004	0.536±0.022	0.533±0.019	0.511±0.010	0.573±0.002
Self-consistency	0.690±0.019	0.116±0.029	0.671±0.048	0.120±0.015	0.682±0.017	0.100±0.008

TABLE IV: Scale check: error-detection AUROC by signal and domain for Qwen2.5-0.5B (3-seed mean) vs. Qwen2.5-1.5B (single seed).
Signal	0.5B (3 seeds)	1.5B (1 seed)
	FIN	FiNER	Tweet	FIN	FiNER	Tweet
Sequence prob.	0.839	0.661	0.626	0.764	0.720	0.637
Span prob.	0.801	0.690	0.721	0.819	0.729	0.638
Self-consistency	0.690	0.671	0.682	0.735	0.744	0.663

TABLE V: Entity-level selective precision (%) at 100/80/60% coverage (Qwen2.5-0.5B; mean±std over 3 seeds).
Domain	Signal	P@100%	P@80%	P@60%
FIN	Seq. prob.	40.5±0.4	49.0±0.8	58.4±0.6
FIN	Span prob.	40.5±0.4	49.2±0.3	56.4±0.6
FIN	Self-consistency	40.5±0.4	47.7±0.4	52.8±1.0
FiNER	Seq. prob.	30.7±1.3	34.9±3.0	40.1±1.3
FiNER	Span prob.	30.7±1.3	35.3±1.7	40.8±2.6
FiNER	Self-consist.	30.7±1.3	34.9±3.0	41.4±2.3
Tweet	Seq. prob.	30.8±0.7	34.2±1.0	36.8±2.9
Tweet	Span prob.	30.8±0.7	36.6±1.1	42.7±1.1
Tweet	Self-consist.	30.8±0.7	35.5±1.7	40.5±1.5

TABLE VI: TweetNER7 filtering sensitivity, main (drop-sentence) vs. relaxed (ignore-entity) protocol. Encoder: 3-seed mean; Qwen: seed 42.
	Filtered	Relaxed
Sentences dropped (of 2807)	2179	0
Evaluated (cap 300)	300	300
Gold PER/ORG/LOC	619	443
Ignored entities	0	488
Encoder F1	24.1±2.5	19.5±1.0
Encoder MSP AUROC	0.615±0.054	0.650±0.047
Qwen-0.5B F1	29.7	29.1
Qwen-0.5B span AUROC	0.725	0.715

为什么重要

在金融合规、风险监控等场景中,单纯的准确率数字无法告诉从业者何时可以放心让AI自动处理结果。这项研究表明,当输入数据类型发生变化时,AI自身汇报的置信度可能悄悄失效,并为选择合适的置信度指标提供了具体依据。

本文术语

命名实体识别(NER) · 从文本中识别并标注人名、机构名、地名等专有名词的AI任务
领域偏移(distribution shift) · 实际使用时的文本风格或主题与AI训练数据不同的现象
选择性预测(selective prediction) · 让AI在没有把握时拒绝回答、转交人工复核的处理方式
自洽性(self-consistency) · 通过让AI对同一输入多次生成结果、看结果是否一致来衡量置信度的方法
LoRA · 一种轻量级微调方法,只训练额外添加的小部分参数,而不重新训练整个大模型
AUROC / ECE · AUROC衡量置信度分数区分对错预测的能力,ECE衡量AI给出的置信度数值与实际正确率的吻合程度

论文原文摘要（英文）

Financial AI systems often train information extractors on one textual register and deploy them across filings, news, and user-generated content, while standard F1 scores do not indicate which predictions remain safe to automate when the input distribution changes. We study confidence estimation and selective prediction for financial named entity recognition (NER) on a three-tier stress test spanning SEC filings, financial news, and general-topic social media as an extreme out-of-domain condition. We evaluate a BERT tagger and LoRA-tuned Qwen2.5-0.5B/1.5B models using five inference-time confidence signals, three training seeds, and bootstrap intervals. Confidence rankings themselves change under distribution shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, whereas entity-span probability and self-consistency are more robust; self-consistency is also better calibrated without post-hoc fitting. Abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no usefully large clean subset under the extreme social-media shift. These results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating.

作者 · Zihao Zheng, Baichuan Li, Junyi Yao, Jiayu Long

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Zihao Zheng et al., arXiv:2608.19558, arxiv-nonexclusive

Reliable Financial Named Entity Recognition under Domain Shift

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道