CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms

arXiv:2608.280532026-08-31

886、yyds、彳亍这类中文新造词,大模型往往能解释是什么意思,却还原不出它原本的样子

研究团队构建了CNeo-Bench,收录4,759个中文网络新词(如用886谐音代表拜拜、用yyds取自拼音首字母、用彳亍拆解行字),按构词机制分为5大类9小类。团队用两层评测测试了包括GPT-5.1、DeepSeek、Kimi在内的18个大模型,发现在仅要求解释含义的任务上大多数模型准确率不到40%,而在要求还原新词原始形式的任务上,不少模型即便正确解释了含义,也会用近义改写代替原始形式。给模型看1到3个示例后,大约一半难题能被解决,但即便给3个示例,仍有33%到47%的错误无法解决。

METAL LAB 解读图

CNeo-Bench两层评测流程

证据状态已报告实测结果

收集新词从萌娘百科、维基学院等来源收集4,759个中文新词及参考释义,按构词机制分为5大类9小类
第一层:解释含义模型仅看到新词本身,需要生成释义,由大模型评判员与参考释义比对打分
第二层:操作形式模型需要还原出新词的原始形式(谐音、拆字类)或从四个选项中选出正确答案(缩写、语义新词等类),按精确匹配或选项匹配打分
识别-操作差距分析挑出第一层答对但第二层答错的案例,发现模型倾向于用改写代替原始形式的失败模式
少样本恢复实验对三个模型共同答错的1,058个难题,给出1到3个示例句子重新测试,观察恢复率和仍未解决的残留错误

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

团队从萌娘百科、维基学院等来源收集了4,759个中文新词及其参考释义,按构词机制分为5个大类和9个小类,涵盖谐音替代、汉字拆解、拼音缩写、外来借词等类型。
评测分两层:第一层要求模型仅凭新词本身给出释义;第二层要求模型直接还原新词背后的原始形式,或从四个选项中选出正确答案。
研究以温度0(只输出最确定的答案)测试了18个模型,包括GPT-5.1、DeepSeek-V3和V3.2、Kimi-K2-Instruct和K2.5、Qwen3系列、GLM-4系列、InternLM3、Gemma3系列、Llama3.2系列。
表现最好的Kimi-K2.5在释义任务上仅达到67.74%的准确率,低于人类评测者85.63%的平均水平;而在通用基准上表现强劲的GPT-5.1在这里只有50.89%,低于多个中文优化模型。
在谐音和拆字这类需要还原原始形式的任务上,出现了系统性的'识别-操作差距':模型在第一层正确解释了新词含义,却在第二层未能还原原始形式,而是给出一个意思相近的改写。

Figure 1: Categories and tasks in CNeo-Bench.

Table 1: Statistics for CNeo-Bench. “Full” is the total number of items in each subcategory; “Hard” is the number of items on which GPT-5.1, DeepSeek-V3.2, and Kimi-K2.5 jointly fail under definition generation.
Subcategory	Full	Hard	Hard Rate
Pinyin Abbr.	192	35	18.23%
English Abbr.	58	7	12.07%
Chinese Homo.	377	65	17.24%
Number Homo.	29	7	24.14%
Lexical Neo.	1161	298	25.67%
Semantic Neo.	850	185	21.76%
Unconv. Char.	23	3	13.04%
Formulaic.	1667	347	20.82%
Foreign	402	111	27.61%
Total	4759	1058	22.23%

Figure 2: Contingency analysis of Tier 1 (definition generation) vs Tier 2 (category-specific) outcomes for five representative models on three open-ended subcategories. Cells A/B/C/D denote (T1✓, T2✓), (T1✓, T2×), (T1×, T2✓), (T1×, T2×) respectively. The highlighted B/(A+B) column reports the operation failure rate (the fraction of correctly described neologisms on which restoration fails) directly quantifying the recognition-manipulation gap.

Table 2: Full definition generation task results. Scores are reported as accuracy (%). Bold indicates the best score in each column; underline indicates the second best. Avg.∗ is computed as micro-average across all 4,759 instances.
Model	Alphabetic	Homophonic	Lexical	Stylistics	Foreign	Avg.∗
Pinyin	English	Chinese	Number	Lexical	Semantic	Unconv. Char.	Formulaic
GPT-5.1	47.40	74.14	48.81	44.83	49.10	54.94	21.74	52.67	42.54	50.89
DeepSeek-V3	64.06	74.14	55.44	72.41	51.51	54.59	65.22	54.35	45.27	53.81
DeepSeek-V3.2	58.85	79.31	60.21	72.41	54.61	55.65	69.57	54.17	49.00	55.26
Kimi-K2-Instruct	66.15	70.69	61.27	72.41	58.14	64.47	65.22	62.57	53.48	61.27
Kimi-K2.5	76.56	79.31	70.82	72.41	66.06	67.06	82.61	67.55	61.69	67.74
Qwen3-4B	9.38	41.38	15.53	17.24	24.89	28.12	8.70	26.03	13.93	23.49
Qwen3-4B-Instruct-2507	7.81	53.45	16.71	20.69	27.82	32.94	13.04	28.91	16.67	26.69
Qwen3-8B	17.71	53.45	18.04	27.59	31.35	33.65	4.55	32.03	18.16	29.40
Qwen3-32B	21.88	63.79	25.99	34.48	36.69	39.18	17.39	37.79	25.37	35.34
GLM-4-9B-0414	18.75	48.28	20.16	20.69	29.46	34.00	8.70	29.03	21.39	28.35
GLM-4-32B-0414	30.21	63.79	35.28	65.52	41.26	45.88	34.78	41.09	30.60	40.60
InternLM3-8B-Instruct	17.19	58.62	18.30	34.48	25.84	31.06	4.55	20.94	15.17	23.56
Gemma3-4B-it	5.21	37.93	7.69	3.45	17.23	19.29	0.00	16.68	10.45	15.68
Gemma3-12B-it	19.27	48.28	14.32	17.24	25.58	27.06	4.55	23.34	18.16	23.41
Gemma3-27B-it	26.04	63.79	21.22	13.79	31.52	30.00	4.55	29.57	19.40	28.66
Llama3.2-1B-Instruct	1.56	3.45	3.18	10.34	5.77	4.47	0.00	6.24	2.24	5.00
Llama3.2-3B-Instruct	5.21	24.14	3.71	3.45	5.77	8.71	0.00	8.64	6.22	7.33
Llama3.2-11B-Instruct	4.69	34.48	7.16	3.45	14.13	17.29	0.00	13.44	10.70	13.34

Figure 3: Few-shot recovery on 1,058 hard samples. Each bar shows the number of hard samples recovered (i.e., judged correct under the few-shot setting) under 1-shot, 2-shot, and 3-shot prompting. Bars are stacked by subcategory to show the distribution of recoveries across the nine subcategories.

Table 3: Category-specific task results. Scores are reported as accuracy (%). Bold indicates the best score in each column; underline indicates the second best. Tasks with† use open-ended restoration; all other tasks use multiple-choice format. Avg.∗ is computed as micro-average across all 4,759 instances.
Model	Alphabetic	Homophonic	Lexical	Stylistics	Foreign	Avg.∗
Pinyin	English	Chinese†	Number†	Lexical	Semantic	Unconv. Char.†	Formulaic
GPT-5.1	72.40	98.28	56.76	72.41	73.47	96.46	22.73	68.77	86.36	75.70
DeepSeek-V3	77.60	98.28	65.25	82.76	77.43	96.58	45.45	70.10	81.06	77.76
DeepSeek-V3.2	76.04	94.83	64.46	82.76	78.12	95.75	54.55	64.20	81.31	75.61
Kimi-K2-Instruct	78.65	96.55	68.97	93.10	83.03	96.58	59.09	68.17	84.60	79.20
Kimi-K2.5	80.21	96.55	68.17	79.31	82.00	97.76	59.09	75.39	88.64	81.94
Qwen3-4B	23.44	86.21	9.28	0.00	49.53	86.79	13.64	44.65	51.77	50.40
Qwen3-4B-Instruct-2507	22.40	84.48	16.18	6.90	51.34	92.10	13.64	46.03	54.04	52.99
Qwen3-8B	30.21	86.21	19.10	3.45	52.63	90.21	4.55	49.22	58.59	54.97
Qwen3-32B	56.25	94.83	33.69	13.79	69.42	96.11	13.64	58.90	66.92	66.63
GLM-4-9B-0414	38.54	89.66	26.26	10.34	61.41	91.16	4.55	37.67	49.75	53.47
GLM-4-32B-0414	65.62	93.10	37.93	51.72	70.03	93.63	18.18	48.98	66.67	63.79
InternLM3-8B-Instruct	16.15	79.31	15.12	0.00	53.32	92.69	0.00	59.33	46.97	58.60
Gemma3-4B-it	30.73	86.21	9.55	3.45	39.79	91.04	4.55	29.60	44.95	43.22
Gemma3-12B-it	39.58	89.66	21.22	10.34	58.66	92.69	4.55	44.65	57.07	55.78
Gemma3-27B-it	41.67	94.83	29.44	20.69	66.32	95.64	4.55	48.13	62.37	60.71
Llama3.2-1B-Instruct	24.48	48.28	0.27	0.00	25.40	31.13	0.00	4.03	24.24	16.81
Llama3.2-3B-Instruct	26.04	70.69	0.27	0.00	36.61	54.72	0.00	5.60	28.79	25.03
Llama3.2-11B-Instruct	24.48	74.14	6.90	6.90	39.28	83.84	4.55	36.04	45.96	43.57

Figure 4: Accuracy on English-based vs Pinyin-based alphabetic abbreviations in the definition generation task. English-based abbreviations are substantially easier than Pinyin-based ones for all evaluated models.

Table 4: Human performance on a stratified sample of CNeo-Bench Tier 1 definition generation. We leave Kimi-K2.5, the strongest model we evaluate, for comparison.
Subcategory	Human	Kimi-K2.5
Pinyin Abbr.	91.00%	76.56%
English Abbr.	84.48%	79.31%
Chinese Homo.	94.00%	70.82%
Number Homo.	82.76%	72.41%
Lexical Neo.	87.00%	66.06%
Semantic Neo.	83.00%	67.06%
Unconv. Chars.	95.65%	82.61%
Formulaic	78.00%	67.55%
Foreign	80.00%	61.69%
Overall	85.63%	67.74%

Figure 5: Tier 2 minus Tier 1 accuracy (percentage points) across 18 models and 9 subcategories. Positive gaps (red) indicate Tier 2 exceeds Tier 1; negative gaps (blue) indicate Tier 2 underperforms Tier 1. Subcategories marked with † use open-ended restoration; others use multiple-choice.

Table 5: Mechanism-driven taxonomy of Chinese neologisms in CNeo-Bench, with five top-level categories and nine subcategories. ∗Foreign Borrowings has no further subcategorization and is treated as a single class.
Categories	Subcategories	Explanations	Examples
1. Alphabetic Abbreviations	Pinyin Abbr.	Expressions formed by taking the initial letters of the Pinyin transliteration of Chinese phrases to create abbreviated forms.	yyds from “永远的神” (yong yuan de shen), which means forever a legend.
	English Abbr.	Expressions formed by taking the initial letters of the English phrases to create abbreviated forms.	ACG from “Anime, Comics, Games”.
2. Homophonic Expressions	Number Homo.	Expressions that use numbers to convey meaning based on phonetic similarity.	“拜拜咯” becomes “886” as they have similar pronunciations in Chinese, which means bye bye.
	Chinese Homo.	Expressions that replace standard Chinese characters with characters of similar pronunciation.	“与你无关” (yu ni wu guan) becomes “雨女无瓜” (yu nv wu gua), which means none of your business.
3. Lexical Innovations	Lexical Neo.	Newly coined expressions that introduce novel lexical items.	“键盘侠”, which means keyboard warrior, a person who aggressively criticizes or attacks others online without engaging in real-world action.
	Semantic Neo.	Existing words that acquire new meanings in emerging usage.	“养鱼”, literal meaning: fish farming; new meaning: to maintain multiple romantic prospects simultaneously, often by keeping others as backups.
4. Stylistic and Pragmatic Expressions	Unconv. Chars.	Expressions that employ rare, archaic, or visually stylized Chinese characters in non-standard ways to convey meaning or achieve stylistic and expressive effects.	“彳亍”, an expression composed of two archaic characters “彳” and “亍” to form “行”, which means okay.
	Formulaic	Fixed or semi-fixed expressions derived from specific memes or events, which are reused across contexts to convey particular pragmatic meanings.	“没活了可以咬打火机”, a meme-based expression originating from a viral livestream clip, used to humorously suggest that someone has run out of ideas or content.
5. Foreign Borrowings	Foreign∗	Expressions derived from foreign-language sources through phonetic reinterpretation or direct lexical adoptiont.	“逮虾户” from “Déjà Vu” (phonetic, via Japanese anime Initial D), used for high-speed driving scenes; “文化祭”, which means school cultural festival is a direct loanword from Japanese.

Table 6: Representative examples for each Tier 2 tasks in CNeo-Bench. Subcategories marked with † require the model to complete open-ended restoration. English references are not included in our dataset.
Subcategory	Task	Example	Answer
Pinyin Abbr.	Cloze	Context: 这家楼下新开的烧烤摊味道真的太绝了，朋友吃完直接说它是夜宵界的(____)。 English: The new BBQ stall that just opened downstairs is absolutely amazing, my friend straight-up called it the (____) of the late-night food scene. Options: (A) cxk (B) xswl (C) yysy (D) yyds	(D) yyds (“forever legendary”)
English Abbr.	Cloze	Context: 他平时看番、追漫画还爱玩手游，朋友圈里一看就是个(____)重度爱好者。 English: He watches anime, follows manga, and loves playing mobile games, and you can tell he’s a hardcore (____) enthusiast from his social media. Options: (A) KPI (B) ACG (C) PDF (D) CPU	(B) ACG (Anime, Comics, Games)
Chinese Homo.†	Restoration	Neologism: 雨女无瓜 Example context: 这件事是他们部门内部的安排，雨女无瓜，别再到处打听了。 English: This is an internal arrangement within their department, none of your business, so stop asking around about it.	与你无关
Number Homo.†	Restoration	Neologism: 886 Example context: "时间不早了, 我先撤, 886, 明天见。 English: It’s getting late, I’m gonna head out. Bye-bye, see you tomorrow!	拜拜咯
Lexical Neo.	Cloze	Context: 事情刚上热搜、真相还没出来，他就在评论区里对当事人冷嘲热讽、指点江山，活脱脱一个(____)。 English: The story had just started trending and the truth hadn’t even come out yet, but there he was in the comments section mocking the people involved and pontificating about everything, a textbook (____). Options: (A) 程序员 (B) 黑客 (C) 杠精 (D) 键盘侠	(D) 键盘侠 (keyboard warrior)
Semantic Neo.	Semantic Discrimination	Target word: 养鱼 Sentences: (A) 他在院子里挖了个小池子，专门养鱼和种睡莲。(fish farming) (B) 爷爷退休后最大的乐趣就是养鱼，每天一早都去喂食。(fish farming) (C) 她总是同时和好几个人保持暧昧，朋友都说她是在养鱼。 (D) 这家农场除了养鸡养鸭，还养鱼，逢年过节卖得特别好。(fish farming)	(C) maintaining multiple romantic relationships
Unconv. Chars.†	Restoration	Neologism: 彳亍	行 (“okay”)
Formulaic	Scenario Matching	Scenario: 家庭聚会上，62岁的老周本来在陪孙子搭积木，结果看到院里积雪够厚，突然兴奋地拉着几个晚辈去堆“巨型雪怪”，还非要拿胡萝卜做獠牙。被老伴笑着数落一把年纪还这么能闹，他却越玩越起劲。 English: At the family gathering, 62-year-old Zhou had been building blocks with his grandson, but when he saw how thick the snow had piled up in the yard, he suddenly got excited and dragged a few of the younger family members out to build a “giant snow monster”, and he insisted on using carrots for the fangs. His wife laughingly scolded him for being so rowdy at his age, but the more he played, the more into it he got. Options: (A) 老夫聊发少年狂 (B) 人间清醒 (C) 差生文具多 (D) 男人至死是少年	(D) 男人至死是少年 (Men never really grow up)
Foreign†	Source Identification	Neologism: 文化祭 Options: (A) 它是从日语“文化祭(ぶんかさい)”转译扩展的词，“祭”在日语里有集中展示、主题活动之意，“文化祭”本质上是对日本校园文化展示周这一活动与说法的概括吸收。(Expanded translation) (B) 它是从日语“文化祭(ぶんかさい)”直接借入的词，“祭”在日语里有节庆、庆典之意，“文化祭”本质上是对日本校园“学园文化祭”这一制度与说法的直接沿用。(Direct loanword) (C) 它是从日语“文化祭(ぶんかさい)”类推形成的词，“祭”在日语里有公开活动、校内集会之意，“文化祭”本质上是对日本校园“社团发表会”这一形式与说法的重新命名。(Analogical formation) (D) 它是从日语“文化祭(ぶんかさい)”借形改义的词，“祭”在日语里有热闹活动、集体庆祝之意，“文化祭”本质上是对日本校园“校园开放日”这一安排与说法的本地化套用。(Borrowed form, new meaning)	(B) Direct loanword

Table 7: Breakdown of Cell B failure modes (Tier 1 correct, Tier 2 wrong) across five models and three open-ended subcategories. Each row sums to 100%. Copy Verbatim: model outputs the neologism itself. Explanation/Paraphrase: model outputs a semantic gloss or near-synonym of the source form. Wrong Restoration: model outputs a different Chinese expression that is neither the neologism nor a paraphrase. Empty/Other: empty output or unparseable.
Model	Subcategory	Cell B Failure Mode (%)
Copy Verbatim	Explanation/Paraphrase	Wrong Restoration	Empty/Other
Kimi-K2.5	Chinese Homo.	26.6%	38.0%	32.9%	2.5%
Number Homo.	0.0%	60.0%	40.0%	0.0%
Unconv. Chars.	0.0%	71.4%	28.6%	0.0%
DeepSeek-V3.2	Chinese Homo.	25.5%	35.3%	35.3%	3.9%
Number Homo.	0.0%	100%	0.0%	0.0%
Unconv. Chars.	0.0%	62.5%	37.5%	0.0%
Qwen3-32B	Chinese Homo.	13.6%	22.7%	54.5%	9.1%
Number Homo.	0.0%	42.9%	57.1%	0.0%
Unconv. Chars.	50.0%	50.0%	0.0%	0.0%
GLM-4-32B	Chinese Homo.	27.3%	38.2%	32.7%	1.8%
Number Homo.	0.0%	66.7%	33.3%	0.0%
Unconv. Chars.	33.3%	33.3%	33.3%	0.0%
Gemma3-27B	Chinese Homo.	22.2%	44.4%	26.7%	6.7%
Number Homo.	0.0%	100%	0.0%	0.0%
Unconv. Chars.	0.0%	0.0%	100%	0.0%

Table 8: Representative Cell B cases (Tier 1 correct, Tier 2 wrong) across the three open-ended subcategories. In each case, the model correctly describes the neologism in Tier 1 but, in Tier 2, substitutes a semantic equivalent for the source form rather than producing the source form itself (Explanation/Paraphrase), which is the dominant failure mode across all three subcategories (Table 7). English references are provided below the generated definition.
Subcategory	Model	Neologism	Gold	Tier 1 Model Output (correct)	Tier 2 Output (wrong)
Chinese Homo.	Kimi-K2.5	v我50	微我50	网络流行语，指通过微信转账50元人民币。源自肯德基“疯狂星期四”促销活动… Referring to transferring 50 yuan RMB via WeChat. It originated from KFC’s ‘Crazy Thursday’ promotional campaign.	微信我50 (additional characters)
Chinese Homo.	Kimi-K2.5	石乐志	失了智	网络流行语，“失了智”的谐音，指失去理智、做出愚蠢或不理智的行为… A homophonic pun on “失了智” (lost one’s wits), meaning to lose one’s reason and act in a stupid or irrational way.	失智 (dementia)
Chinese Homo.	DeepSeek-V3.2	河蟹	和谐	网络用语中常被用作“删除”或“屏蔽”的委婉说法，因“和谐”与“河蟹”谐音，后者常被用来讽刺内容审查。 Often used as a euphemism for ‘deleting’ or ‘blocking’ content. Because 和谐 (harmony) is a homophone of 河蟹 (river crab).	删除 (delete)
Chinese Homo.	DeepSeek-V3.2	大清药丸	大清要完	大清药丸是网络流行语，源自“大清要完”的谐音，实际用来调侃某事物或局势即将崩溃或失败，带有讽刺或幽默意味。 A homophonic pun on ‘大清要完’ (the Qing dynasty is finished). It’s used to jokingly suggest that something or some situation is on the verge of collapse or failure, with a satirical or humorous undertone.	大清朝要完 (additional characters)
Number Homo.	Kimi-K2.5	3q	thank you	英文“thank you”的谐音数字缩写，“3”对应“th”音、“q”对应“you”音。 ‘3Q’ is a homophonic abbreviation of the English ‘thank you’, using numbers and letters: ‘3’ represents the ‘th’ sound and ‘q’ represents the ‘you’ sound.	谢谢 (Chinese expression for thank you)
Number Homo.	DeepSeek-V3.2	666	溜溜溜	表示非常厉害、令人惊叹的意思，源自数字6的谐音“溜”。 ‘666’ means something is awesome or impressive. It derives from the number 6, a homophone of ‘溜’ (‘slick/skilled’)	牛牛牛 (semantically equivalent)
Unconv. Chars.	Kimi-K2.5	占戈哥欠走已	战歌起	将“战歌起”三字拆解为偏旁部首（战=占+戈，歌=哥+欠，起=走+己/已）的隐晦写法… A cryptic way of writing ‘战歌起’ (cue the battle anthem) by splitting each of the three characters into their radical components.	战歌响起 (additional characters)
Unconv. Chars.	Kimi-K2.5	口区	呕	将汉字“呕”拆分为“口”和“区”二字输入的网络黑话… The Chinese character ‘呕’ (to vomit/retch) is broken apart and typed as its two components, ‘口’ and ‘区’.	呕吐 (additional characters)
Unconv. Chars.	DeepSeek-V3.2	彳亍口巴	行吧	网络用语，由“彳亍”和“口巴”拼接而成。其中“彳亍”是“行”的拆分形式，“口巴”是“吧”的拆分形式。 ‘彳亍’ is the disassembled form of ‘行’ and ‘口巴’ is the disassembled form of ‘吧’.	好吧 (another expression for okay)
Unconv. Chars.	DeepSeek-V3.2	占戈土也	战地	“占戈土也”是“战地”的拆分写法… ‘占戈土也’ is the disassembled-character way of ‘战地’(‘battlefield’)	战争 (war)

研究结果

在释义任务上,表现最好的Kimi-K2.5整体准确率为67.74%,多数开源模型不到40%,而人类评测者平均达到85.63%。
在通用基准上处于前沿水平的GPT-5.1在CNeo-Bench整体上仅得50.89%,低于中文优化模型,在拼音缩写(47.40%对Kimi-K2.5的76.56%)和拆字(21.74%对82.61%)上差距尤为明显。
所有模型在第二层任务上的得分都高于自身在第一层的得分,但在三个需要开放式还原的小类(两种谐音类型加拆字)中,第一层答对的题目里有24.2%到57.1%在第二层还原失败。
分析第一层答对但第二层答错的案例,发现最常见的失败模式是模型给出意思相近的改写或近义词,而不是给出原始形式本身。
在GPT-5.1、DeepSeek-V3.2、Kimi-K2.5三个模型共同答错的1,058个难题上,给1个示例就能让37%到50%的题目被答对,给3个示例能提升到53%到67%,但即便给3个示例,仍有33%到47%的题目答不对。

可应用场景

可作为诊断工具,检验聊天机器人、翻译系统或内容审核工具处理中文网络流行语和梗的能力水平。
可用来定位模型知道新词含义却无法操作其底层形式的薄弱环节,为设计涉及新词的提示词或示例策略提供参考。
可作为参考,判断哪种构词机制(谐音替代、拆字等)对模型最难,从而指导针对性训练数据或微调方案的设计。

局限与待验证事项

第一层的打分依赖大模型评判员判断语义是否恰当,因此即便没有还原出原始形式,释义也可能被判为正确。
数字谐音(29条)和非常规拆字(23条)两个小类样本量很少,针对这两类的模型间细致比较只能作为参考,不能视为确定结论。
本研究专注于诊断这一差距的存在,并未提出解决识别-操作差距的训练方法,留待未来工作。
数据收集截止到2026年1月,之后出现的新词未被纳入。
即便给出3个示例,仍有33%到47%的难题未被解决,这部分残留错误的成因尚未在本文中进一步分析或解决。

为什么重要

新词处于模型训练数据的边缘甚至之外,因此模型处理新词的能力是检验其语言灵活性的一个有效探针。这一结果说明,仅凭释义任务的表现来判断模型是否真正理解某种语言现象是不够的。

本文术语

新词(Neologism) · 近期新造出来的词,或原有词被赋予了新的含义
谐音替代(Homophonic substitution) · 用发音相近但意思无关的数字或汉字替代原本的表达,例如886因发音接近拜拜咯而被使用
非常规拆字(Unconventional Character Decomposition) · 把一个汉字拆成视觉上的组成部分分开书写,例如把行拆成彳和亍
识别-操作差距(Recognition-Manipulation Gap) · 模型能正确说出新词的含义,却无法还原出该新词最初是从哪个原始形式变化而来的现象
大模型评判员(LLM-as-judge) · 用另一个大模型来自动判断模型生成的答案是否与参考答案意思相符

论文原文摘要（英文）

Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character decomposition that are rare in other languages. We introduce CNeo-Bench, a benchmark of 4,759 such neologisms with reference definitions, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression. CNeo-Bench is paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whether it can operate on its underlying mechanism. Evaluating 18 LLMs, we find that Chinese neologisms remain an open challenge; most models fall below 40\% on definition generation, and on several subcategories a systematic recognition-manipulation gap emerges: models describe neologisms correctly but, in source-form restoration tasks, substitute a semantic equivalent (paraphrase) for the source form rather than producing the source form itself. A few-shot analysis on 1,058 hard items shows that in-context examples can solve many difficult cases, but leave a noticeable portion of errors remaining, indicating challenges beyond prompting alone can address.

作者 · Kaiyan Zhao, Zhongtao Miao, Zheyong Xie, Shaosheng Cao, Yoshimasa Tsuruoka

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Kaiyan Zhao et al., arXiv:2608.28053, arxiv-nonexclusive