AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
arXiv:2608.213822026-08-25
The same model scores anywhere from 31% to 89% depending only on how you grade the multiple-choice test, and that hidden choice is quietly picking leaderboard winners
The researchers gave 12 open-weight LLMs the exact same 3,679 questions from ARC, HellaSwag, MMLU and TruthfulQA, but scored each answer 26 different equally reasonable ways (option order, prompt wording, and whether answers come from generated text or likelihood scores). They found that the items two neighboring models both answer reliably are basically tied, while the gap that ranks them comes almost entirely from items that flip depending on the scoring setup. Four different models can each claim the number-one spot depending only on which of the 26 setups is used, and the compression techniques meant to shrink benchmarks tend to keep exactly these unstable items.
METAL LAB explanatory visual
Fragility Grid: 같은 문항, 같은 모델, 26가지 채점 방식
Evidence statusMeasured results reported
고정 요소3,679개 문항, 12개 모델의 동일한 가중치, greedy decoding은 모두 고정
가변 요소: 하네스 26종선택지 순서(6종) x 프롬프트 형식(4종) = 24가지 생성 설정 + 확률 기반 채점 2가지 = 26가지 조합
문항 분류모든 설정에서 항상 맞히면 robust-correct, 항상 틀리면 robust-wrong, 설정에 따라 뒤집히면 config-fragile
인접 쌍 격차 분해리더보드 순위가 붙은 두 모델의 정확도 차이를 robust 문항에서 나오는 부분과 fragile 문항에서 나오는 부분으로 나눔
챔피언 집합26개 설정 중 하나라도 1위를 차지한 모델들을 모아 harness가 승자를 어떻게 바꾸는지 확인
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
12 instruction-tuned open-weight LLMs from 4 families (Qwen, Llama, Gemma, Mistral) answered the same 3,679 fixed items under 26 different but all-defensible harness configurations, varying only option order, prompt format, and whether scoring used generated text or per-option likelihood.
For each model, item and configuration they recorded a single correct/wrong bit, holding items, model weights, and greedy decoding fixed so only the harness setup changed.
On items where two neighboring models on the leaderboard both answered consistently across all 26 setups, the two models turned out to be statistically tied; on average 95.7% of the accuracy gap between adjacent models came from items whose correctness flipped depending on the harness.
Four of the 12 models reached rank 1 under at least one of the 26 configurations, and switching just the scoring method (generated text vs. likelihood) preserved only 31% of a model's credited correct answers on average, far less than switching prompt format (40%) or option order (60%).
Benchmark-compression methods that keep only the most 'discriminative' items (the ones that best separate strong from weak models) turned out to correlate with fragility at 0.28, and the most discriminative items had 96% average fragility versus 85% overall, meaning compression tends to retain the very items whose correctness depends on scoring setup.
Figure 1: The same model, the same items, and many scores. Models run down the axis in reference-leaderboard order, best at the top. Each row runs from robust accuracy, the items a model answers correctly under all 26 harness configurations (left tick), to optimistic accuracy, the items it answers correctly under at least one (right tick), with the reference-configuration score marked by the dot. Only the harness differs along a row. For gemma4-31b the row runs from 0.04 to 0.98, and the 0.88 it is credited with under the reference harness is one point inside that range. Averaged across the 12 models, 85 percent of credited answers can be flipped to wrong by an equally defensible configuration, which is why a single leaderboard number is a point drawn from a wide band rather than a property of the model.
Table 1: The reference leaderboard, with the lowest and highest single-configuration accuracy across the 26 configurations. Spread is their difference, computed before rounding, so for two rows it differs by 0.001 from the difference of the rounded columns shown. Config-lucky is the share of reference-correct answers that flip to wrong under some valid configuration. Accuracy is pooled over the same 3,679 items for every model. The spread reported here is the range of single-configuration scores and is narrower than the robust-to-optimistic band drawn in Figure 1.
Rank
Model
Reference
Min-config
Max-config
Spread
Config-lucky
1
gemma4-31b
0.880
0.306
0.886
0.580
0.958
2
llama3.3-70b
0.832
0.520
0.832
0.311
0.764
3
gemma4-26b-a4b
0.825
0.302
0.840
0.538
0.964
4
qwen3-30b-a3b
0.822
0.514
0.835
0.321
0.791
5
gemma4-12b
0.800
0.284
0.810
0.526
0.923
6
qwen3-14b
0.799
0.559
0.804
0.245
0.731
7
qwen3-32b
0.799
0.380
0.802
0.422
0.939
8
qwen3-4b
0.747
0.398
0.754
0.356
0.873
9
gemma4-e4b
0.699
0.289
0.705
0.416
0.921
10
llama3.1-8b
0.641
0.458
0.645
0.186
0.753
11
mixtral-8x7b
0.629
0.476
0.661
0.185
0.806
12
llama3.2-3b
0.547
0.395
0.608
0.213
0.818
Figure 2: Every leaderboard gap is fragile mass. For each adjacent pair, the orange bar is the reference-config accuracy gap, the ordering the leaderboard prints. The blue marker is the accuracy gap on the items both models answer stably, each model’s answer unchanged across all 26 configurations, with its 95 percent bootstrap confidence interval. Every marker sits on zero and every interval contains it, so the two models are tied once the fragile items are set aside, while the orange bar they are ranked by remains. The label on each bar gives the fragile share of that gap. The qwen3-14b and qwen3-32b pair has no gap to decompose, since the two are tied on the full item set as well.
Table 2: Adjacent-pair gap decomposition. Reference gap is the accuracy difference under the reference configuration. Fragile share is the part of that gap carried by config-fragile items. Robust tie indicates that the two models are indistinguishable on the items both answer stably, where five pairs answer identically on every such item, four differ on exactly one item, and two disagree in both directions and net to zero. Robust items is the size of that jointly stable set. Reversing counts the configurations under which the lower model outscores the higher one.
Higher
Lower
Ref gap
Fragile share
Robust tie
Robust items
Reversing / 26
gemma4-31b
llama3.3-70b
0.048
0.994
yes
75
8
llama3.3-70b
gemma4-26b-a4b
0.006
1.000
yes
101
10
gemma4-26b-a4b
qwen3-30b-a3b
0.003
0.917
yes
90
17
qwen3-30b-a3b
gemma4-12b
0.022
0.988
yes
142
10
gemma4-12b
qwen3-14b
0.001
0.667
yes
158
21
qwen3-14b
qwen3-32b
0.000
n/a
yes
151
10
qwen3-32b
qwen3-4b
0.052
1.000
yes
72
1
qwen3-4b
gemma4-e4b
0.048
1.000
yes
143
12
gemma4-e4b
llama3.1-8b
0.058
1.000
yes
193
8
llama3.1-8b
mixtral-8x7b
0.013
1.000
yes
336
19
mixtral-8x7b
llama3.2-3b
0.082
1.000
yes
230
0
Figure 3: The configuration selects the champion. Models run down the axis in reference-leaderboard order, best at the top, and each bar spans the range of ranks a model attains across the 26 configurations, with a dot at its reference rank. The four models that reach rank one under some configuration are drawn in orange, and the dashed line marks rank one. mixtral-8x7b places 11th of 12 under the reference harness yet reaches first, spanning ranks 1 to 11, and qwen3-14b spans 1 to 7. The winner of the leaderboard is selected by the configuration, not only by the model.
Table 3: Axis-robust accuracy: the accuracy that survives varying one axis with the others held at the reference. Joint varies all three at once. Lower is worse, since it means the axis destroys more credited answers. Scoring is the most destructive single axis for 9 of the 12 models.
Model
Ordering
Format
Scoring
Joint
gemma4-12b
0.673
0.373
0.177
0.062
gemma4-26b-a4b
0.718
0.310
0.186
0.029
gemma4-31b
0.803
0.412
0.213
0.037
gemma4-e4b
0.520
0.379
0.152
0.055
llama3.1-8b
0.434
0.391
0.363
0.158
llama3.2-3b
0.294
0.316
0.269
0.100
llama3.3-70b
0.689
0.442
0.516
0.196
mixtral-8x7b
0.411
0.392
0.370
0.122
qwen3-14b
0.662
0.529
0.470
0.215
qwen3-30b-a3b
0.710
0.449
0.480
0.172
qwen3-32b
0.662
0.486
0.172
0.048
qwen3-4b
0.602
0.271
0.403
0.095
Mean
0.598
0.396
0.314
0.108
Figure 4: The scoring choice does the damage. For each model, four bars give the accuracy that survives varying one harness axis with the other axes held at the reference: option order, prompt format, scoring method, and the joint grid. Averaged across the 12 models, shuffling the options preserves 0.60 of credited accuracy, prompt format preserves 0.40, and the generation-versus-likelihood choice preserves only 0.31. Varying all three together leaves 0.11. Scoring is the most destructive single axis for 9 of the 12 models, and the joint bar is the smallest for all 12. The axis that evaluation protocols most often fix is the one that matters least.
Table 4: The six option orderings. Entry j gives the index of the original option shown in display slot j, so p0 is the identity and leaves the benchmark’s own order intact. The five non-identity orderings are a fixed draw from the 24 permutations of four options, taken under a separate constant seed (99) so that the same five appear in every run and for every model. A model’s prediction is mapped back through the permutation before it is scored, so a correct answer counts as correct whatever slot it was shown in.
Ordering
Slot A
Slot B
Slot C
Slot D
p0
0
1
2
3
identity; used by the reference configuration
p1
1
3
2
0
p2
0
1
3
2
p3
0
2
3
1
p4
3
0
2
1
p5
2
3
1
0
Table 5: The six prompt templates. Newlines are written \n, and the option block renders the four options one per line in the order fixed by the configuration’s permutation, with the label style shown. The four generation templates are crossed with the six orderings. The two cloze stems carry no option labels, since each option is scored as a continuation of the stem, so the ordering axis cannot act on them.
Template
Labels
Text
Generation: the model emits an answer and a regular expression reads it
letter_plain
A.
Question: [q]\n[options]\nAnswer with the letter only:
letter_paren
(A)
[q]\n[options]\nThe correct option is
digit_labels
1.
Question: [q]\n[options]\nAnswer (1-4):
instruction
A)
Read carefully and pick the single best choice.\n[q]\n[options]\nYour choice:
Likelihood: each option is scored as a continuation and the argmax wins
cloze_plain
none
[q]\nAnswer:
cloze_question
none
Question: [q]\nAnswer:
Table 6: Every step in the pipeline that consumes randomness, and what it decides. Nothing else in the pipeline is stochastic: the correctness tensor itself is produced by greedy decoding and by likelihood scoring, both of which are deterministic given the weights.
Step
Seed
Draws
What it decides
Item sampling
1234
once per benchmark
which items are evaluated
Option orderings
99
once
the five non-identity permutations
Generation
1234
per call
nothing, since the temperature is 0
Bootstrap, robust-item gap
1234
10,000
the intervals behind Table 2
Bootstrap, correlation
1234
2,000
the interval on 0.28, not the 0.28
Discrimination tie-break
1234
200
which 64 items complete the top-100 subset
Random-subset baseline
1234
200
the 5.1 champion baseline
Table 7: Seed sensitivity of the compression statistics. Each row reruns the analysis end to end under a different seed. The champion count on the discriminative subset is a mean over 200 tie-breaks of the 102 items on the cutoff. The random baseline is a mean over 200 subsets of 100 items drawn without replacement. The discriminative subset yields more champions than the random subset under every seed, and both yield more than the full benchmark, which yields 4 under all of them.
Seed
corr(disc, fragility)
Discriminative 100
Random 100
Full benchmark
1234 (reported)
0.2767
5.67
5.08
4
1
0.2767
5.67
4.81
4
2
0.2767
5.68
4.93
4
7
0.2767
5.68
4.85
4
42
0.2767
5.70
4.80
4
99
0.2767
5.67
4.89
4
2024
0.2767
5.72
4.98
4
31337
0.2767
5.59
4.88
4
Table 8: The winner of every configuration and its margin over the runner-up, in accuracy and in items out of 3,679. The 24 generation configurations are decided by margins of 7 to 292 items. The 2 likelihood configurations are decided by 1 item and 3 items, and are the only first places in the grid that a handful of items would overturn. The letter blocks and the digit block are unanimous, which is the split the champion claim rests on.
Configuration
Winner
Acc.
Runner-up
Margin
Items
gen|letter_plain|p0
gemma4-31b
0.8796
llama3.3-70b
0.0478
176
gen|letter_plain|p1
gemma4-31b
0.8820
qwen3-30b-a3b
0.0470
173
gen|letter_plain|p2
gemma4-31b
0.8804
gemma4-26b-a4b
0.0560
206
gen|letter_plain|p3
gemma4-31b
0.8731
llama3.3-70b
0.0468
172
gen|letter_plain|p4
gemma4-31b
0.8834
gemma4-26b-a4b
0.0429
158
gen|letter_plain|p5
gemma4-31b
0.8856
qwen3-30b-a3b
0.0546
201
gen|letter_paren|p0
gemma4-31b
0.8230
qwen3-32b
0.0652
240
gen|letter_paren|p1
gemma4-31b
0.8247
qwen3-32b
0.0508
187
gen|letter_paren|p2
gemma4-31b
0.8190
qwen3-32b
0.0587
216
gen|letter_paren|p3
gemma4-31b
0.8146
qwen3-32b
0.0546
201
gen|letter_paren|p4
gemma4-31b
0.8203
qwen3-32b
0.0652
240
gen|letter_paren|p5
gemma4-31b
0.8247
qwen3-32b
0.0565
208
gen|digit_labels|p0
qwen3-14b
0.6268
qwen3-32b
0.0247
91
gen|digit_labels|p1
qwen3-14b
0.6222
qwen3-30b-a3b
0.0794
292
gen|digit_labels|p2
qwen3-14b
0.6276
qwen3-32b
0.0272
100
gen|digit_labels|p3
qwen3-14b
0.6203
qwen3-32b
0.0264
97
gen|digit_labels|p4
qwen3-14b
0.6013
llama3.3-70b
0.0767
282
gen|digit_labels|p5
qwen3-14b
0.6099
gemma4-12b
0.0769
283
gen|instruction|p0
gemma4-31b
0.7513
qwen3-14b
0.0038
14
gen|instruction|p1
qwen3-14b
0.7521
gemma4-31b
0.0027
10
gen|instruction|p2
gemma4-31b
0.7518
qwen3-14b
0.0068
25
gen|instruction|p3
gemma4-31b
0.7442
qwen3-14b
0.0073
27
gen|instruction|p4
gemma4-31b
0.7418
qwen3-14b
0.0019
7
gen|instruction|p5
qwen3-14b
0.7543
gemma4-31b
0.0060
22
ll|cloze_plain
llama3.3-70b
0.6099
mixtral-8x7b
0.0003
1
ll|cloze_question
mixtral-8x7b
0.6053
llama3.3-70b
0.0008
3
Table 9: Every headline number recomputed with TruthfulQA removed. The three claims of the paper hold on either item set. The two quantities that move are the champion count and the minimum Kendall correlation, both of which fall on the reduced set. The mean fragile share exceeds 1 without TruthfulQA because one pair’s jointly robust items favor the lower-ranked model.
Quantity
4 benchmarks (reported)
3 benchmarks (no TruthfulQA)
Items per model
3,679
3,000
Mean config-lucky fraction
0.853
0.859
Adjacent pairs whose order is manufactured
10 of 11
11 of 11
Mean fragile share of gap
0.957
1.004
Distinct champions
4
3
Models reaching the top three
8
9
corr(discrimination, fragility)
0.28 [0.25, 0.30]
0.32 [0.29, 0.35]
Mean fragility, 100 most discriminative items
0.960
0.968
Mean fragility, all items
0.851
0.863
Axis-robust accuracy, option order
0.598
0.605
Axis-robust accuracy, prompt format
0.396
0.367
Axis-robust accuracy, scoring method
0.314
0.328
Kendall τ against the reference, mean
0.42
0.41
Kendall τ against the reference, minimum
−0.09
+0.03
Findings
gemma4-31b는 동일한 가중치와 문항으로도 채점 설정에 따라 31%에서 89% 사이의 정확도를 기록했고, 12개 모델 평균으로는 정답 처리된 답의 85%가 다른 정당한 설정 하에서 오답으로 뒤집힐 수 있었다.
인접한 리더보드 순위 쌍 11개 중 10개는 두 모델이 안정적으로 맞힌 문항에서는 사실상 동점이었고(11개 중 5개는 완전히 동일, 4개는 문항 1개 차이), 그 격차의 평균 95.7%(문항 단위로 합산 시 99.7%)가 채점 설정에 따라 뒤집히는 문항에서 나왔다.
26개 설정 중 4개 모델(gemma4-31b, qwen3-14b, llama3.3-70b, mixtral-8x7b)이 각각 적어도 한 번 1위를 차지했고, 기준 리더보드에서 11위였던 mixtral-8x7b도 특정 설정에서 1위에 올랐다.
채점 방식(생성 텍스트 vs 확률값)만 바꿨을 때 평균적으로 정답의 31%만 유지되었는데, 이는 프롬프트 형식을 바꿀 때의 40%, 선택지 순서를 바꿀 때의 60%보다 낮아 가장 파괴적인 축이었다.
문항 판별력(item discrimination)과 config-fragility의 상관은 0.28(95% CI 0.25-0.30)이었고, 판별력이 가장 높은 상위 100개 문항의 평균 fragility는 0.96으로 전체 평균 0.85보다 높아, 벤치마크 압축이 오히려 불안정한 문항을 남기는 경향을 보였다.
Where it can be used
리더보드를 운영하는 팀이 순위를 발표하기 전에 같은 방식으로 harness를 여러 개 돌려 문항 단위로 순위가 얼마나 안정적인지 점검하는 사전 체크로 활용할 수 있다.
벤치마크를 소수 문항으로 압축하려는 연구자가 판별력 기준으로 문항을 고를 때, 그 문항들이 채점 설정에 따라 얼마나 불안정한지 함께 확인하는 참고 자료로 쓸 수 있다.
모델 비교 논문을 쓸 때 단일 정확도 숫자 대신 robust accuracy와 optimistic accuracy 같은 범위를 함께 보고하는 관행을 제안하는 근거로 인용할 수 있다.
Limits and open work
실험은 4개 벤치마크(ARC, HellaSwag, MMLU, TruthfulQA)의 4지선다 문항, 12개 오픈웨이트 모델, zero-shot 그리디 디코딩이라는 특정 조건에 한정되어 있어 few-shot이나 다른 형태의 벤치마크로 일반화되는지는 검증되지 않았다.
26개 설정은 모두 실제 평가 코드에서 쓰이는 방어 가능한 조합이지만, 저자들이 고른 축(선택지 순서, 프롬프트 형식, 채점 방식)에 한정된 것이며 다른 종류의 하네스 변화는 포함되지 않았다.
TruthfulQA를 제외하고 재계산했을 때 champion 수는 4에서 3으로, 최소 Kendall 상관은 -0.09에서 +0.03으로 바뀌어, 일부 수치는 벤치마크 구성에 따라 달라질 수 있음을 저자들도 인정했다.
판별력 기준 상위 100개 문항 선정 시 720개의 서로 다른 판별력 값만 존재해 100위 근처에 동점 문항이 많아, champion 수 관련 수치는 200회 무작위 동점 처리 평균에 의존한다.
논문은 코드와 문항별 기록을 공개했지만 본 요약은 저자가 보고한 완료된 실험 결과만을 다루며, 추가 벤치마크나 더 많은 모델군으로의 확장 실험은 아직 수행되지 않았다.
Why it matters
Anyone reading an LLM leaderboard treats a single accuracy number as a fact about a model, but this work shows that number is really one sample from a wide band that a different, equally legitimate scoring choice could move by dozens of points. Because the paper releases the per-item data and a script that reproduces every number on a CPU in seconds, any leaderboard maintainer can run the same fragility check before publishing a ranked order.
Terms in this paper
하네스(harness) · 모델의 답을 채점하기 위해 실제로 실행하는 구체적 절차: 선택지 순서, 프롬프트 문구, 정답을 생성 텍스트에서 읽을지 확률값에서 읽을지 등을 정하는 설정
robust item · 26가지 채점 설정 전부에서 항상 같은 정답/오답 결과가 나오는 문항
item discrimination · 어떤 문항이 강한 모델과 약한 모델을 얼마나 잘 구분해 주는지를 나타내는 통계량, 벤치마크 압축 기법이 우선적으로 남기는 문항의 기준
champion set · 26개 채점 설정 중 적어도 하나에서 1위를 차지한 모델들의 집합
Original abstract (English)
Multiple-choice benchmarks fix the questions and the correct answers, but not the harness: the order of the options, the wording of the prompt, and whether a language model's answer is read from generated text or from per-option likelihoods. Work on this harness sensitivity reports it as aggregate score variance, leaving unexamined which items the variance falls on and whether they are the items that separate one model from the next. We treat the evaluation harness of large language models (LLMs) as an independent variable and resolve its effect to single items. We introduce the \textit{fragility grid}: 12 open-weight instruction-tuned LLMs from 4 families answer the same 3{,}679 items from 4 benchmarks (ARC, HellaSwag, MMLU, TruthfulQA) under 26 equally defensible harness configurations, recording one correctness bit for every model, item, and configuration. The comparison is matched, since the items, the weights, and the greedy decoding stay fixed while only the harness varies. Under the grid a model's score is a band rather than a point: gemma4-31b scores between 31 and 89 percent depending only on the harness. Three results follow. On the items that two adjacent models both answer stably the pair is tied, and config-fragile items carry 95.7 percent of a pair's gap on average. Four of the 12 models reach rank one under some configuration, so the harness selects the winner. Item discrimination, the property that benchmark-compression methods maximize, correlates with fragility at 0.28 (95 percent CI 0.25 to 0.30), so compression keeps the fragile items rather than removing them. The scoring choice, not the option order that protocols usually fix, is the load-bearing axis. We release the per-item records and the analysis script, from which every number regenerates on a CPU in seconds, and we position the fragility grid as a check a leaderboard can run before it reports an order.
