LLMs as Acquisition Policies for Finite-Pool Materials Optimization: A Controlled Study

arXiv:2608.197902026-08-21

Researchers tested whether LLMs can pick which new material to try next, instead of classic statistical methods

The team asked five open-weight large language models to act as decision-makers choosing which untested material composition to evaluate next, in a setting where real experiments are expensive. LLMs consistently beat random guessing but generally fell short of the long-standing statistical method called Gaussian-process optimization on most tasks. Performance swung a lot depending on how candidates were listed and whether material-specific context was given, showing the approach works but is not yet reliable enough to trust blindly.

What they did

Using four material-optimization benchmarks (steel alloys, magnetic Fe-Co-Ni films, piezoelectric ceramics), the team had five open-weight LLMs (Gemma, DeepSeek, Qwen family) repeatedly pick one untested candidate from a list until they found the best one, and counted how many tries it took
They compared two ways of showing candidates to the LLM: presenting the entire remaining pool at once, or splitting it into small groups and running a tournament where the LLM picks a winner from each group
LLMs always needed fewer tries than random selection, but lost to the classic Gaussian-process method on three of the four tasks, only matching or beating it on one task
Results varied a lot depending on how the candidate list was ordered, how large the groups were, and whether material-specific labels (like element names) were included; some models also showed a clear bias toward picking candidates near the top of the list regardless of quality
When the researchers examined the pattern of choices, LLMs did not explore and exploit the search space in the same steady, structured way as the Gaussian-process method

Figure 1: Best-so-far optimization trajectories using Gemma 4 31B for the LLM policies. At iteration t, each curve shows the mean percentage gap 100(y⋆−bt)/|y⋆| across the 25 shared initialization seeds. Shading denotes pointwise normal-approximation 95% confidence intervals, computed as g¯t±1.96st/25. We show Gemma 4 31B because it ranks among the strongest LLMs across all four tasks.

Table 1: Retrospective optimization tasks.
Task	Candidates	Features	Target
Kerr Rotation	921	3	Kerr rotation (mrad)
Coercivity	921	3	coercivity (mT)
Matbench Steels	312	14	yield strength (MPa)
Electrostrain	81	15	electrostrain (%)

Figure 2: Batch-size sensitivity of tournament selection on Matbench Steels and Kerr Rotation for runs of 25 seeds. Curves report the mean number of iterations to the optimum across seeds, with uncertainty shown shaded as the mean ± std/nseeds.

Table 2: Iterations to the global optimum over 25 initializations (mean ± standard deviation), except for random selection which is initialized over 1000 seeds. Lower is better. Baselines are shown once because they do not depend on an LLM presentation protocol.
Model	Kerr Rotation	Coercivity	Matbench Steels	Electrostrain
Random	462.48 ± 262.24	448.09 ± 269.63	159.31 ± 91.28	39.89 ± 23.00
GP-EI	8.40 ± 2.78	91.80 ± 62.77	42.72 ± 27.24	18.48 ± 14.53
Whole-pool selection
Gemma4	31.28 ± 17.24	209.88 ± 134.05	44.16 ± 23.36	14.36 ± 7.71
Qw27B	27.08 ± 13.59	195.20 ± 114.39	48.24 ± 23.43	19.44 ± 10.40
Qw35B	45.72 ± 20.28	289.88 ± 127.44	50.36 ± 24.89	27.64 ± 18.15
Qw397B	23.48 ± 10.40∗	258.24 ± 151.68∗	53.32 ± 32.90∗	16.84 ± 5.93
DS	110.64 ± 85.31	236.28 ± 210.72	87.60 ± 52.10	13.88 ± 9.98
Batch-tournament (b=20)
Gemma4	13.12 ± 5.12	389.64 ± 92.00	78.08 ± 49.83	14.80 ± 6.50
Qw27B	14.28 ± 7.33	468.64 ± 50.68	64.52 ± 32.94	18.04 ± 5.33
Qw35B	22.00 ± 8.73	379.80 ± 110.27	52.68 ± 20.12	20.36 ± 9.89
Qw397B	14.48 ± 7.54	178.80 ± 82.87†	81.96 ± 41.10	15.08 ± 6.92
DS	16.88 ± 5.52	241.84 ± 52.10	32.88 ± 31.71	12.36 ± 4.71

(a) Position distributions under batch-tournament and whole-pool selection.

Table 3: Position-bias metrics for all model-task combinations. We report DKL for whole-pool and batch-tournament selection with b=20, except for the proxy configurations marked with an asterisk. Missing measurements are denoted by ’-’.
	Electrostrain	Matbench Steels	Coercivity	Kerr Rotation
Models	Whole	Batch	Whole	Batch	Whole	Batch	Whole	Batch
Gemma4	0.998	2.27	1.02	2.11	1.60	2.08	2.21	2.02
DS	1.36	2.29	1.47	2.13	2.48	2.35	2.60	2.15
Qw27B	0.929	2.10	0.852	2.04	2.18	2.06	2.17	2.04
Qw35B	1.26	2.20	0.747	2.12	1.67	2.24	2.77	2.08
Qw397B	1.04	2.20	1.36∗	2.08	–	1.84†	–	2.06

Figure 4: Optimization performance on Matbench Steels, with and without materials context across 25 initializations. Bars report the mean iteration at which the optimum is reached, with normal-approximation 95% confidence intervals, computed as mean ± 1.96std/nseeds. We also report the mean iterations to optimum of random selection with a dotted line.

Table 4: Generation-and-matching performance on Matbench Steels compared with random, GP-EI, and whole-pool selection. We report iterations to the global optimum over 25 initializations (mean ± standard deviation), except for random selection which is initialized over 1000 seeds.
Model	Random	GP-EI	Whole-pool	Generation-and-matching
Qw27B	159.31 ± 91.28	42.72 ± 27.24	48.24 ± 23.43	123.60 ± 63.60
Gemma4	44.16 ± 23.36	48.80 ± 26.14

Figure 5: Explorative behavior of two selection protocols on Matbench Steels for seed 4 with Gemma 4 31B. Dots represent points of the pool projected onto the first two principal components, with the PCA being scaled over every pool entry. On this seed, GP-EI and batch-tournament reach the optimum after 60 and 56 iterations, respectively.

Why it matters

For fields like materials discovery where every real test is costly and slow, knowing whether a general-purpose AI model can replace specialized statistical tools without extra training matters a lot for practical adoption. This study shows real promise but also concrete limitations, meaning teams should not yet swap in LLMs for proven optimization methods without careful checks.

Figure 7: Batch-size sensitivity of tournament selection on Coercivity and Electrostrain. Curves report the mean number of iterations to the optimum across seeds, with uncertainty shown shaded as the mean ± std/nseeds.

Terms in this paper

Active Learning · A method where a system uses past results to decide what to test next, to save costly experiments
Gaussian Process (GP) · A classic statistical model that predicts both an outcome and its uncertainty, widely used to guide materials search
Acquisition Function · A scoring rule (like Expected Improvement, EI) used to decide which untested candidate to try next
Open-weight LLM · A large language model whose internal parameters are publicly released so anyone can run it
Position Bias · A tendency of an AI to favor items in certain positions of a list (e.g., near the top) regardless of their actual quality

Figure 8: Comparison of evolution of fitted β^t on Matbench Steels between Gemma 4 31B, Random and GP-EI. Results are averaged across seeds for each iteration, when at least half of the seeds have not converged yet.

Original abstract (English)

Discovering materials with desirable properties often requires searching large candidate spaces while experimental or computational evaluations remain costly. Active learning addresses this challenge by using previous observations to select which candidate to evaluate next, typically through probabilistic surrogate models. We investigate whether open-weight large language models (LLMs) can serve as standalone acquisition policies in this setting. We evaluate five LLMs across four retrospective finite-pool materials optimization tasks under different candidate-presentation strategies and compare them with random selection and conventional Gaussian-process methods. LLM policies generally reach the global optimum in fewer iterations than random selection, indicating that they provide a useful acquisition signal without task-specific training. Their performance relative to Gaussian-process methods is mixed: conventional acquisition performs better on most tasks, while LLMs match or outperform it in some settings. Performance varies substantially across tasks, models, initializations, and candidate presentations, with no LLM approach performing best across all tasks. Overall, open-weight LLMs show potential as acquisition policies for finite-pool materials search, although their reliability remains sensitive to the task and to how candidates and scientific context are presented.

Authors · Dino-Rober Demir, Florian Le Bronnec, Rio Yokota

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Dino-Rober Demir et al., arXiv:2608.19790, CC BY 4.0