每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

LLMs as Acquisition Policies for Finite-Pool Materials Optimization: A Controlled Study

arXiv:2608.197902026-08-21

研究人员测试了大语言模型能否代替传统统计方法来挑选下一个要测试的新材料

在真实实验成本高昂的材料研发场景中,研究团队让五款开源大语言模型自己决定下一个该测试哪种材料成分。结果显示,这些大模型选出的候选材料总是比随机选择更快找到最优解,但在大多数任务上仍不如材料领域长期使用的统计方法高斯过程优化。候选材料的展示方式和是否附带材料专业背景信息,都会让模型表现出现明显波动,说明这一方法目前还不够稳定可靠。

他们做了什么

  1. 研究团队在钢铁合金、Fe-Co-Ni磁性薄膜、压电陶瓷等四类材料优化任务上,让Gemma、DeepSeek、Qwen系列共五款开源大语言模型反复从候选列表中选一个未测试的成分进行评估,记录找到最优解所需的尝试次数
  2. 比较了两种向模型展示候选材料的方式:一次性展示全部剩余候选,或把候选分成若干小组进行淘汰赛式的锦标赛筛选
  3. 大语言模型的表现始终优于随机选择,但在四项任务中的三项上不敌传统的高斯过程优化方法,只有一项任务能与之匹敌或更优
  4. 同一模型的表现会因候选列表的排列顺序、分组大小、是否提供元素名称等材料专业背景信息而出现明显差异,部分模型还表现出明显偏向选择列表靠前位置候选的"位置偏差"
  5. 进一步分析发现,大语言模型的选择模式并不像高斯过程方法那样在"探索新区域"和"深挖已知优区域"之间保持规律的过渡,而是更不规则
Figure 1: Best-so-far optimization trajectories using Gemma 4 31B for the LLM policies. At iteration t, each curve shows the mean percentage gap 100​(y⋆−bt)/|y⋆| across the 25 shared initialization seeds. Shading denotes pointwise normal-approximation 95% confidence intervals, computed as g¯t±1.96​st/25. We show Gemma 4 31B because it ranks among the strongest LLMs across all four tasks.
Figure 1: Best-so-far optimization trajectories using Gemma 4 31B for the LLM policies. At iteration t, each curve shows the mean percentage gap 100​(y⋆−bt)/|y⋆| across the 25 shared initialization seeds. Shading denotes pointwise normal-approximation 95% confidence intervals, computed as g¯t±1.96​st/25. We show Gemma 4 31B because it ranks among the strongest LLMs across all four tasks.
Table 1: Retrospective optimization tasks.
TaskCandidatesFeaturesTarget
Kerr Rotation9213Kerr rotation (mrad)
Coercivity9213coercivity (mT)
Matbench Steels31214yield strength (MPa)
Electrostrain8115electrostrain (%)
Figure 2: Batch-size sensitivity of tournament selection on Matbench Steels and Kerr Rotation for runs of 25 seeds. Curves report the mean number of iterations to the optimum across seeds, with uncertainty shown shaded as the mean ± std/nseeds.
Figure 2: Batch-size sensitivity of tournament selection on Matbench Steels and Kerr Rotation for runs of 25 seeds. Curves report the mean number of iterations to the optimum across seeds, with uncertainty shown shaded as the mean ± std/nseeds.
Table 2: Iterations to the global optimum over 25 initializations (mean ± standard deviation), except for random selection which is initialized over 1000 seeds. Lower is better. Baselines are shown once because they do not depend on an LLM presentation protocol.
ModelKerr RotationCoercivityMatbench SteelsElectrostrain
Random462.48 ± 262.24448.09 ± 269.63159.31 ± 91.2839.89 ± 23.00
GP-EI8.40 ± 2.7891.80 ± 62.7742.72 ± 27.2418.48 ± 14.53
Whole-pool selection
Gemma431.28 ± 17.24209.88 ± 134.0544.16 ± 23.3614.36 ± 7.71
Qw27B27.08 ± 13.59195.20 ± 114.3948.24 ± 23.4319.44 ± 10.40
Qw35B45.72 ± 20.28289.88 ± 127.4450.36 ± 24.8927.64 ± 18.15
Qw397B23.48 ± 10.40∗258.24 ± 151.68∗53.32 ± 32.90∗16.84 ± 5.93
DS110.64 ± 85.31236.28 ± 210.7287.60 ± 52.1013.88 ± 9.98
Batch-tournament (b=20)
Gemma413.12 ± 5.12389.64 ± 92.0078.08 ± 49.8314.80 ± 6.50
Qw27B14.28 ± 7.33468.64 ± 50.6864.52 ± 32.9418.04 ± 5.33
Qw35B22.00 ± 8.73379.80 ± 110.2752.68 ± 20.1220.36 ± 9.89
Qw397B14.48 ± 7.54178.80 ± 82.87†81.96 ± 41.1015.08 ± 6.92
DS16.88 ± 5.52241.84 ± 52.1032.88 ± 31.7112.36 ± 4.71
(a) Position distributions under batch-tournament and whole-pool selection.
(a) Position distributions under batch-tournament and whole-pool selection.
Table 3: Position-bias metrics for all model-task combinations. We report DKL for whole-pool and batch-tournament selection with b=20, except for the proxy configurations marked with an asterisk. Missing measurements are denoted by ’-’.
ElectrostrainMatbench SteelsCoercivityKerr Rotation
ModelsWholeBatchWholeBatchWholeBatchWholeBatch
Gemma40.9982.271.022.111.602.082.212.02
DS1.362.291.472.132.482.352.602.15
Qw27B0.9292.100.8522.042.182.062.172.04
Qw35B1.262.200.7472.121.672.242.772.08
Qw397B1.042.201.36∗2.08–1.84†–2.06
Figure 4: Optimization performance on Matbench Steels, with and without materials context across 25 initializations. Bars report the mean iteration at which the optimum is reached, with normal-approximation 95% confidence intervals, computed as mean ± 1.96​std/nseeds. We also report the mean iterations to optimum of random selection with a dotted line.
Figure 4: Optimization performance on Matbench Steels, with and without materials context across 25 initializations. Bars report the mean iteration at which the optimum is reached, with normal-approximation 95% confidence intervals, computed as mean ± 1.96​std/nseeds. We also report the mean iterations to optimum of random selection with a dotted line.
Table 4: Generation-and-matching performance on Matbench Steels compared with random, GP-EI, and whole-pool selection. We report iterations to the global optimum over 25 initializations (mean ± standard deviation), except for random selection which is initialized over 1000 seeds.
ModelRandomGP-EIWhole-poolGeneration-and-matching
Qw27B159.31 ± 91.2842.72 ± 27.2448.24 ± 23.43123.60 ± 63.60
Gemma444.16 ± 23.3648.80 ± 26.14
Figure 5: Explorative behavior of two selection protocols on Matbench Steels for seed 4 with Gemma 4 31B. Dots represent points of the pool projected onto the first two principal components, with the PCA being scaled over every pool entry. On this seed, GP-EI and batch-tournament reach the optimum after 60 and 56 iterations, respectively.
Figure 5: Explorative behavior of two selection protocols on Matbench Steels for seed 4 with Gemma 4 31B. Dots represent points of the pool projected onto the first two principal components, with the PCA being scaled over every pool entry. On this seed, GP-EI and batch-tournament reach the optimum after 60 and 56 iterations, respectively.

为什么重要

在材料研发这类每次真实实验都费时费钱的领域,弄清楚通用人工智能模型是否可以在无需额外训练的情况下替代专门的统计优化工具,具有很强的现实意义。这项研究既展示了潜力也暴露了明显局限,提示相关团队现阶段不宜贸然用大语言模型取代成熟的优化方法,而应谨慎验证。

Figure 7: Batch-size sensitivity of tournament selection on Coercivity and Electrostrain. Curves report the mean number of iterations to the optimum across seeds, with uncertainty shown shaded as the mean ± std/nseeds.
Figure 7: Batch-size sensitivity of tournament selection on Coercivity and Electrostrain. Curves report the mean number of iterations to the optimum across seeds, with uncertainty shown shaded as the mean ± std/nseeds.

本文术语

  • 主动学习(Active Learning) · 利用已有观测结果来决定下一步测试什么,以节省昂贵实验成本的方法
  • 高斯过程(Gaussian Process, GP) · 一种能同时预测结果和不确定性的经典统计模型,广泛用于指导材料搜索
  • 获取函数(Acquisition Function) · 用于给候选材料打分、决定下一个测试对象的规则,常见的有期望改进量(EI)
  • 开源权重大语言模型(Open-weight LLM) · 内部参数公开发布、任何人都可下载运行的大语言模型
  • 位置偏差(Position Bias) · 人工智能倾向于选择列表中特定位置(如靠前)的项目,而不管其实际质量如何的现象
Figure 8: Comparison of evolution of fitted β^t on Matbench Steels between Gemma 4 31B, Random and GP-EI. Results are averaged across seeds for each iteration, when at least half of the seeds have not converged yet.
Figure 8: Comparison of evolution of fitted β^t on Matbench Steels between Gemma 4 31B, Random and GP-EI. Results are averaged across seeds for each iteration, when at least half of the seeds have not converged yet.

论文原文摘要（英文）

Discovering materials with desirable properties often requires searching large candidate spaces while experimental or computational evaluations remain costly. Active learning addresses this challenge by using previous observations to select which candidate to evaluate next, typically through probabilistic surrogate models. We investigate whether open-weight large language models (LLMs) can serve as standalone acquisition policies in this setting. We evaluate five LLMs across four retrospective finite-pool materials optimization tasks under different candidate-presentation strategies and compare them with random selection and conventional Gaussian-process methods. LLM policies generally reach the global optimum in fewer iterations than random selection, indicating that they provide a useful acquisition signal without task-specific training. Their performance relative to Gaussian-process methods is mixed: conventional acquisition performs better on most tasks, while LLMs match or outperform it in some settings. Performance varies substantially across tasks, models, initializations, and candidate presentations, with no LLM approach performing best across all tasks. Overall, open-weight LLMs show potential as acquisition policies for finite-pool materials search, although their reliability remains sensitive to the task and to how candidates and scientific context are presented.

作者 · Dino-Rober Demir, Florian Le Bronnec, Rio Yokota

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Dino-Rober Demir et al., arXiv:2608.19790, CC BY 4.0