每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

PersonalBench: Measuring the Authorship Gap in LLM Personalization

arXiv:2608.197462026-08-21

让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调

研究者测试了几种让大语言模型模仿特定人写作风格的方法,并用一个专门判断作者身份的AI模型、一个充当评委的语言模型、以及经典的文体统计手段来打分。结果生成的文本之间确实能按目标人物区分开来,但和那个人真实写的文章相比,相似度还不如两个毫不相干的真人之间的相似度。团队发布了名为PersonalBench的评测基准,基于50位作者和1000篇生成文本,证明基于提示词的个性化方法只是在模型自身的风格空间里做调整,始终没能真正跨越到接近人类作者的水平。

他们做了什么

  1. 问题所在:现有的个性化评测基准只看任务完成度或用户偏好是否匹配,并不检验生成的文字是否真的读起来像目标作者写的
  2. 方法:结合作者身份验证模型LUAR、担任评委的语言模型、以及功能词和标点符号等文体统计三种手段,在Qwen 3和GLM-4两个模型家族上,对50位作者、1000篇生成文本、四种个性化方法(无个性化、少量示例、提取风格档案、对比样本)进行测试
  3. 结果:生成文本之间可以按目标作者区分开(判别指标AUC达0.918),但与真实作者本人写作的相似度只有0.484到0.508,反而低于两个毫不相关的真人之间的相似度基准值0.626
  4. 意外发现:担任评委的语言模型认为提取风格档案的方法表现最好,但这其实是循环论证造成的假象——评委和该方法用的是同一套特征提取流程,用专门的作者身份判别模型检验后发现根本没有优势
  5. 局限性:测试用的两个模型都是320亿参数规模且经过压缩(量化)处理的版本;写作样本只来自2000年代初的博客;担任评委的语言模型尚未与人类评审结果做过对比验证
Figure 1: LUAR authorship similarity by method (5-post aggregation). All methods score below the real-text cross-author floor (0.626), far from the real-author ceiling (0.756). The total spread across methods is only 0.024. The LLM’s authorship fingerprint is so dominant that generated text sits in a distinct regime below human-to-human similarity.
Figure 1: LUAR authorship similarity by method (5-post aggregation). All methods score below the real-text cross-author floor (0.626), far from the real-author ceiling (0.756). The total spread across methods is only 0.024. The LLM’s authorship fingerprint is so dominant that generated text sits in a distinct regime below human-to-human similarity.
Table 1: Effect of prompt construction on baseline scores. Naïve extraction leaks author voice, inflating the unpersonalized baseline by 28pp on same-author judgments.
Prompt typeNon-personalizedFew-shot vs. baseline
TMRSA%ΔTMRWin%
Raw first sentence0.58750%−0.08023%
Content summary0.38422%+0.04933%
Figure 2: Distribution of per-generation LUAR similarity scores by method. All four methods produce overlapping distributions centered near 0.44 (1v5 per-generation scores), well below the real-author distribution centered at 0.58.
Figure 2: Distribution of per-generation LUAR similarity scores by method. All four methods produce overlapping distributions centered near 0.44 (1v5 per-generation scores), well below the real-author distribution centered at 0.58.
Table 2: Method comparison across 50 authors, 1,000 generations. LUAR similarity uses 5-post aggregation (5 generations per author-method pair compared to 5 training posts) and is the primary authorship metric (↑ better). TMR = trait match rate from LLM judge. SA% = same-author rate. FuncCos = function word cosine similarity. Ceiling = real author’s test posts; floor = cross-author random pairs. All methods score below the real-text cross-author floor on LUAR despite appearing differentiated on TMR. CIs use hierarchical bootstrap (resampling authors, then generations within authors) to account for within-author correlation (B=10,000).
MethodLUAR ↑TMR ↑SA% ↑FuncCos ↑
Non-Personalized0.484±.0190.384±.05822%±70.741±.011
Few-Shot0.508±.0200.433±.06131%±80.749±.011
Profile Extraction0.502±.0190.542±.06029%±80.761±.010
Contrastive0.494±.0200.447±.05936%±80.752±.011
Real Author (ceiling)0.7560.42730%—
Cross-Author (floor)0.6260.3907%—
Figure 3: Calibration across three metrics. Each panel shows method scores (bars) with ceiling (real author, green) and floor (cross-author, red) baselines. On LUAR (left), all methods score below the real-text cross-author floor. On TMR (center), profile extraction exceeds the real-author ceiling—evidence of circularity (§4.6.1). On SA% (right), contrastive leads but all methods exceed the chance floor.
Figure 3: Calibration across three metrics. Each panel shows method scores (bars) with ceiling (real author, green) and floor (cross-author, red) baselines. On LUAR (left), all methods score below the real-text cross-author floor. On TMR (center), profile extraction exceeds the real-author ceiling—evidence of circularity (§4.6.1). On SA% (right), contrastive leads but all methods exceed the chance floor.
Table 3: Pearson correlation between evaluation metrics (n=1,000). All pairwise correlations are near zero, indicating the metrics capture fundamentally different constructs. No single metric is a reliable proxy for another.
LUARTMRFuncCos
LUAR1.00——
TMR0.0131.00—
FuncCos0.0260.0671.00
Figure 4: LUAR similarity vs. TMR for all 1,000 generations, colored by method. No systematic relationship (r=0.013). Profile extraction’s high TMR does not correspond to high LUAR.
Figure 4: LUAR similarity vs. TMR for all 1,000 generations, colored by method. No systematic relationship (r=0.013). Profile extraction’s high TMR does not correspond to high LUAR.

为什么重要

这对任何声称能适配特定用户写作风格的写作助手、聊天机器人或品牌语气工具都有警示意义,说明单靠精心设计提示词未必能真正捕捉一个人的独特声音。它也为业界提供了一把校准过的尺子,用来检验个性化方法是否真正有效,而不只是看起来令人信服。

本文术语

  • LUAR · 一个基于数百万条Reddit帖子训练出来的AI模型,用来判断两段文字是否出自同一人之手
  • LLM充当评委 · 用一个语言模型来评价另一个模型的输出,替代人工评审
  • AUC · 衡量区分两组数据能力的指标,数值越接近1说明区分效果越好
  • 推理阶段个性化 · 不重新训练模型参数,只通过提示词和示例在生成时调整输出风格的方法
  • 文体统计 · 通过分析词频、标点习惯等写作特征来识别作者身份的统计方法

论文原文摘要（英文）

Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target author's writing. We introduce PersonalBench, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics. Across 50 authors, 1,000 generations, and two model families (Qwen 3, GLM-4), we find that personalization methods do produce author-differentiated output (LUAR discriminates target authors within generated text at AUC=0.918) but this differentiation never crosses the human-LLM boundary. All methods achieve LUAR similarity to real authors in the range 0.484-0.508, below the cross-author human floor of 0.626 (ceiling 0.756). The LLM's own authorship fingerprint dominates: generated text is more distant from any human author than random humans are from each other. Methods are statistically indistinguishable from each other on LUAR (spread 0.024) despite appearing differentiated on the LLM judge, a discrepancy we trace to circularity between trait extraction and profile extraction. We validate that LUAR reliably measures authorship in our corpus (AUC=0.76 single-post, 0.96 multi-post). We release PersonalBench as a calibrated measuring stick: inference-time personalization modulates the LLM's style but does not bridge the gap to human authorship.

作者 · Yash Ganpat Sawant

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Yash Ganpat Sawant et al., arXiv:2608.19746, CC BY 4.0