每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Are LLMs becoming similarly creative? Evidence from three years of models

arXiv:2608.194372026-08-21

三年数据显示,不同AI聊天机器人给出的答案正变得越来越相似

研究团队测试了2023年以来发布的68个AI模型,让它们回答需要创造力的问题,并测量这些答案彼此之间的差异程度。结果显示一个统计上显著的趋势:不同公司模型的回答随时间推移变得越来越相似。研究者警告,如果这一趋势持续,人们通过AI获得的创意想法范围可能会不断收窄。

他们做了什么

  1. 团队使用两套题目——经典心理学创造力测试Alternate Uses Task(要求为日常物品想出不寻常用途)和收集了100个真实用户问题的Infinity-Chat100——让来自12家公司的68个模型分别作答
  2. 把每个回答用句子嵌入模型转换成数字向量,再用余弦距离衡量回答之间的差异,并按发布时间分成九个区间做回归分析
  3. 只比较不同公司之间的模型(避免同公司模型因架构或训练数据相近而造成干扰),并通过1000次重复抽样检验结果,确保结论不受某一家公司模型过度代表的影响
  4. 在AUT测试中,不同公司模型回答之间的平均差异从最早约0.50降到最近低于0.40;在Infinity-Chat100上降幅较缓,从约0.34降到0.32,且1000次重复抽样结果全部显示下降趋势
Figure 1: Methodology overview: we select open-ended prompts designed to elicit creative behavior from models; run these against different model lineages; embed model outputs in a semantic space to compute their semantic distance; and perform a regression analysis to understand changes in semantic distance over time.
Figure 1: Methodology overview: we select open-ended prompts designed to elicit creative behavior from models; run these against different model lineages; embed model outputs in a semantic space to compute their semantic distance; and perform a regression analysis to understand changes in semantic distance over time.
Table 1: Bootstrap OLS estimates for mean cosine distance across nine model release bins.
DatasetModels (Pairs)Slope per Bin95% CI
Alternate Uses Task68 (273)−0.01385[−0.01695,−0.01044]
Infinity-Chat67 (268)−0.00167[−0.00267,−0.00074]
Figure 2: Regression results and bootstrapped slope estimates for AUT and Infinity-Chat response distances across model generations. Our findings of consistently negative slopes across the observation period indicate semantic convergence across LLM creative outputs over time.
Figure 2: Regression results and bootstrapped slope estimates for AUT and Infinity-Chat response distances across model generations. Our findings of consistently negative slopes across the observation period indicate semantic convergence across LLM creative outputs over time.
Table A1: Observed and missing responses for models with at least one missing response. Each model had 10 possible AUT responses and 100 possible Infinity-Chat100 responses.
AUTInfinity-Chat100
ModelObservedMissingObservedMissing
minimax/minimax-010100100
minimax/minimax-m1640100
mistralai/mistral-small-3.1-24b-instruct1008911
qwen/qwen3.6-max-preview281000
qwen/qwen3-max100946
meta-llama/llama-3.2-3b-instruct100973
anthropic/claude-fable-5100991
minimax/minimax-m2.1100991
qwen/qwen-2.5-72b-instruct100991
All 69 models668226,677223
Figure A1: Distribution of model releases over time across the 12 model providers and 68 versions in our dataset.
Figure A1: Distribution of model releases over time across the 12 model providers and 68 versions in our dataset.

为什么重要

如今AI聊天机器人每天处理数十亿条消息,其中大量用于构思创意和写作,如果不同AI给出的答案越来越趋同,人们接触到的想法多样性可能会缩小。这对未来能否继续依赖AI作为创意合作伙伴提出了值得关注的警示。

Figure A2: OLS Linearity, Normality, Independence, and Influential Observations
Figure A2: OLS Linearity, Normality, Independence, and Influential Observations

本文术语

  • Alternate Uses Task(AUT) · 一种心理学测试,通过要求为常见物品想出不寻常用途来衡量发散性思维
  • Infinity-Chat100 · 收集了100个真实用户向AI聊天机器人提出的开放式问题的数据集
  • 句子嵌入(sentence embedding) · 把文本转换成能表示其含义的数字向量的方法,方便进行数学比较
  • 余弦距离 · 衡量两个向量(此处指两个回答)含义差异程度的指标,数值越小表示越相似
  • 跨公司比较(cross-family) · 只比较不同公司生产的模型,而非同一公司内部的模型

论文原文摘要（英文）

Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs' role in the human creative process.

作者 · Nirav Patel, Josiah Crossman, Eva Aggarwal, Emily Wenger

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Nirav Patel et al., arXiv:2608.19437, CC BY 4.0