Figure 1: The same model, the same items, and many scores. Models run down the axis in reference-leaderboard order, best at the top. Each row runs from robust accuracy, the items a model answers correctly under all 26 harness configurations (left tick), to optimistic accuracy, the items it answers correctly under at least one (right tick), with the reference-configuration score marked by the dot. Only the harness differs along a row. For gemma4-31b the row runs from 0.04 to 0.98, and the 0.88 it is credited with under the reference harness is one point inside that range. Averaged across the 12 models, 85 percent of credited answers can be flipped to wrong by an equally defensible configuration, which is why a single leaderboard number is a point drawn from a wide band rather than a property of the model.
Table 1: The reference leaderboard, with the lowest and highest single-configuration accuracy across the 26 configurations. Spread is their difference, computed before rounding, so for two rows it differs by 0.001 from the difference of the rounded columns shown. Config-lucky is the share of reference-correct answers that flip to wrong under some valid configuration. Accuracy is pooled over the same 3,679 items for every model. The spread reported here is the range of single-configuration scores and is narrower than the robust-to-optimistic band drawn in Figure 1.
Rank
Model
Reference
Min-config
Max-config
Spread
Config-lucky
1
gemma4-31b
0.880
0.306
0.886
0.580
0.958
2
llama3.3-70b
0.832
0.520
0.832
0.311
0.764
3
gemma4-26b-a4b
0.825
0.302
0.840
0.538
0.964
4
qwen3-30b-a3b
0.822
0.514
0.835
0.321
0.791
5
gemma4-12b
0.800
0.284
0.810
0.526
0.923
6
qwen3-14b
0.799
0.559
0.804
0.245
0.731
7
qwen3-32b
0.799
0.380
0.802
0.422
0.939
8
qwen3-4b
0.747
0.398
0.754
0.356
0.873
9
gemma4-e4b
0.699
0.289
0.705
0.416
0.921
10
llama3.1-8b
0.641
0.458
0.645
0.186
0.753
11
mixtral-8x7b
0.629
0.476
0.661
0.185
0.806
12
llama3.2-3b
0.547
0.395
0.608
0.213
0.818
Figure 2: Every leaderboard gap is fragile mass. For each adjacent pair, the orange bar is the reference-config accuracy gap, the ordering the leaderboard prints. The blue marker is the accuracy gap on the items both models answer stably, each model’s answer unchanged across all 26 configurations, with its 95 percent bootstrap confidence interval. Every marker sits on zero and every interval contains it, so the two models are tied once the fragile items are set aside, while the orange bar they are ranked by remains. The label on each bar gives the fragile share of that gap. The qwen3-14b and qwen3-32b pair has no gap to decompose, since the two are tied on the full item set as well.
Table 2: Adjacent-pair gap decomposition. Reference gap is the accuracy difference under the reference configuration. Fragile share is the part of that gap carried by config-fragile items. Robust tie indicates that the two models are indistinguishable on the items both answer stably, where five pairs answer identically on every such item, four differ on exactly one item, and two disagree in both directions and net to zero. Robust items is the size of that jointly stable set. Reversing counts the configurations under which the lower model outscores the higher one.
Higher
Lower
Ref gap
Fragile share
Robust tie
Robust items
Reversing / 26
gemma4-31b
llama3.3-70b
0.048
0.994
yes
75
8
llama3.3-70b
gemma4-26b-a4b
0.006
1.000
yes
101
10
gemma4-26b-a4b
qwen3-30b-a3b
0.003
0.917
yes
90
17
qwen3-30b-a3b
gemma4-12b
0.022
0.988
yes
142
10
gemma4-12b
qwen3-14b
0.001
0.667
yes
158
21
qwen3-14b
qwen3-32b
0.000
n/a
yes
151
10
qwen3-32b
qwen3-4b
0.052
1.000
yes
72
1
qwen3-4b
gemma4-e4b
0.048
1.000
yes
143
12
gemma4-e4b
llama3.1-8b
0.058
1.000
yes
193
8
llama3.1-8b
mixtral-8x7b
0.013
1.000
yes
336
19
mixtral-8x7b
llama3.2-3b
0.082
1.000
yes
230
0
Figure 3: The configuration selects the champion. Models run down the axis in reference-leaderboard order, best at the top, and each bar spans the range of ranks a model attains across the 26 configurations, with a dot at its reference rank. The four models that reach rank one under some configuration are drawn in orange, and the dashed line marks rank one. mixtral-8x7b places 11th of 12 under the reference harness yet reaches first, spanning ranks 1 to 11, and qwen3-14b spans 1 to 7. The winner of the leaderboard is selected by the configuration, not only by the model.
Table 3: Axis-robust accuracy: the accuracy that survives varying one axis with the others held at the reference. Joint varies all three at once. Lower is worse, since it means the axis destroys more credited answers. Scoring is the most destructive single axis for 9 of the 12 models.
Model
Ordering
Format
Scoring
Joint
gemma4-12b
0.673
0.373
0.177
0.062
gemma4-26b-a4b
0.718
0.310
0.186
0.029
gemma4-31b
0.803
0.412
0.213
0.037
gemma4-e4b
0.520
0.379
0.152
0.055
llama3.1-8b
0.434
0.391
0.363
0.158
llama3.2-3b
0.294
0.316
0.269
0.100
llama3.3-70b
0.689
0.442
0.516
0.196
mixtral-8x7b
0.411
0.392
0.370
0.122
qwen3-14b
0.662
0.529
0.470
0.215
qwen3-30b-a3b
0.710
0.449
0.480
0.172
qwen3-32b
0.662
0.486
0.172
0.048
qwen3-4b
0.602
0.271
0.403
0.095
Mean
0.598
0.396
0.314
0.108
Figure 4: The scoring choice does the damage. For each model, four bars give the accuracy that survives varying one harness axis with the other axes held at the reference: option order, prompt format, scoring method, and the joint grid. Averaged across the 12 models, shuffling the options preserves 0.60 of credited accuracy, prompt format preserves 0.40, and the generation-versus-likelihood choice preserves only 0.31. Varying all three together leaves 0.11. Scoring is the most destructive single axis for 9 of the 12 models, and the joint bar is the smallest for all 12. The axis that evaluation protocols most often fix is the one that matters least.
Table 4: The six option orderings. Entry j gives the index of the original option shown in display slot j, so p0 is the identity and leaves the benchmark’s own order intact. The five non-identity orderings are a fixed draw from the 24 permutations of four options, taken under a separate constant seed (99) so that the same five appear in every run and for every model. A model’s prediction is mapped back through the permutation before it is scored, so a correct answer counts as correct whatever slot it was shown in.
Ordering
Slot A
Slot B
Slot C
Slot D
p0
0
1
2
3
identity; used by the reference configuration
p1
1
3
2
0
p2
0
1
3
2
p3
0
2
3
1
p4
3
0
2
1
p5
2
3
1
0
Table 5: The six prompt templates. Newlines are written \n, and the option block renders the four options one per line in the order fixed by the configuration’s permutation, with the label style shown. The four generation templates are crossed with the six orderings. The two cloze stems carry no option labels, since each option is scored as a continuation of the stem, so the ordering axis cannot act on them.
Template
Labels
Text
Generation: the model emits an answer and a regular expression reads it
letter_plain
A.
Question: [q]\n[options]\nAnswer with the letter only:
letter_paren
(A)
[q]\n[options]\nThe correct option is
digit_labels
1.
Question: [q]\n[options]\nAnswer (1-4):
instruction
A)
Read carefully and pick the single best choice.\n[q]\n[options]\nYour choice:
Likelihood: each option is scored as a continuation and the argmax wins
cloze_plain
none
[q]\nAnswer:
cloze_question
none
Question: [q]\nAnswer:
Table 6: Every step in the pipeline that consumes randomness, and what it decides. Nothing else in the pipeline is stochastic: the correctness tensor itself is produced by greedy decoding and by likelihood scoring, both of which are deterministic given the weights.
Step
Seed
Draws
What it decides
Item sampling
1234
once per benchmark
which items are evaluated
Option orderings
99
once
the five non-identity permutations
Generation
1234
per call
nothing, since the temperature is 0
Bootstrap, robust-item gap
1234
10,000
the intervals behind Table 2
Bootstrap, correlation
1234
2,000
the interval on 0.28, not the 0.28
Discrimination tie-break
1234
200
which 64 items complete the top-100 subset
Random-subset baseline
1234
200
the 5.1 champion baseline
Table 7: Seed sensitivity of the compression statistics. Each row reruns the analysis end to end under a different seed. The champion count on the discriminative subset is a mean over 200 tie-breaks of the 102 items on the cutoff. The random baseline is a mean over 200 subsets of 100 items drawn without replacement. The discriminative subset yields more champions than the random subset under every seed, and both yield more than the full benchmark, which yields 4 under all of them.
Seed
corr(disc, fragility)
Discriminative 100
Random 100
Full benchmark
1234 (reported)
0.2767
5.67
5.08
4
1
0.2767
5.67
4.81
4
2
0.2767
5.68
4.93
4
7
0.2767
5.68
4.85
4
42
0.2767
5.70
4.80
4
99
0.2767
5.67
4.89
4
2024
0.2767
5.72
4.98
4
31337
0.2767
5.59
4.88
4
Table 8: The winner of every configuration and its margin over the runner-up, in accuracy and in items out of 3,679. The 24 generation configurations are decided by margins of 7 to 292 items. The 2 likelihood configurations are decided by 1 item and 3 items, and are the only first places in the grid that a handful of items would overturn. The letter blocks and the digit block are unanimous, which is the split the champion claim rests on.
Configuration
Winner
Acc.
Runner-up
Margin
Items
gen|letter_plain|p0
gemma4-31b
0.8796
llama3.3-70b
0.0478
176
gen|letter_plain|p1
gemma4-31b
0.8820
qwen3-30b-a3b
0.0470
173
gen|letter_plain|p2
gemma4-31b
0.8804
gemma4-26b-a4b
0.0560
206
gen|letter_plain|p3
gemma4-31b
0.8731
llama3.3-70b
0.0468
172
gen|letter_plain|p4
gemma4-31b
0.8834
gemma4-26b-a4b
0.0429
158
gen|letter_plain|p5
gemma4-31b
0.8856
qwen3-30b-a3b
0.0546
201
gen|letter_paren|p0
gemma4-31b
0.8230
qwen3-32b
0.0652
240
gen|letter_paren|p1
gemma4-31b
0.8247
qwen3-32b
0.0508
187
gen|letter_paren|p2
gemma4-31b
0.8190
qwen3-32b
0.0587
216
gen|letter_paren|p3
gemma4-31b
0.8146
qwen3-32b
0.0546
201
gen|letter_paren|p4
gemma4-31b
0.8203
qwen3-32b
0.0652
240
gen|letter_paren|p5
gemma4-31b
0.8247
qwen3-32b
0.0565
208
gen|digit_labels|p0
qwen3-14b
0.6268
qwen3-32b
0.0247
91
gen|digit_labels|p1
qwen3-14b
0.6222
qwen3-30b-a3b
0.0794
292
gen|digit_labels|p2
qwen3-14b
0.6276
qwen3-32b
0.0272
100
gen|digit_labels|p3
qwen3-14b
0.6203
qwen3-32b
0.0264
97
gen|digit_labels|p4
qwen3-14b
0.6013
llama3.3-70b
0.0767
282
gen|digit_labels|p5
qwen3-14b
0.6099
gemma4-12b
0.0769
283
gen|instruction|p0
gemma4-31b
0.7513
qwen3-14b
0.0038
14
gen|instruction|p1
qwen3-14b
0.7521
gemma4-31b
0.0027
10
gen|instruction|p2
gemma4-31b
0.7518
qwen3-14b
0.0068
25
gen|instruction|p3
gemma4-31b
0.7442
qwen3-14b
0.0073
27
gen|instruction|p4
gemma4-31b
0.7418
qwen3-14b
0.0019
7
gen|instruction|p5
qwen3-14b
0.7543
gemma4-31b
0.0060
22
ll|cloze_plain
llama3.3-70b
0.6099
mixtral-8x7b
0.0003
1
ll|cloze_question
mixtral-8x7b
0.6053
llama3.3-70b
0.0008
3
Table 9: Every headline number recomputed with TruthfulQA removed. The three claims of the paper hold on either item set. The two quantities that move are the champion count and the minimum Kendall correlation, both of which fall on the reduced set. The mean fragile share exceeds 1 without TruthfulQA because one pair’s jointly robust items favor the lower-ranked model.
Multiple-choice benchmarks fix the questions and the correct answers, but not the harness: the order of the options, the wording of the prompt, and whether a language model's answer is read from generated text or from per-option likelihoods. Work on this harness sensitivity reports it as aggregate score variance, leaving unexamined which items the variance falls on and whether they are the items that separate one model from the next. We treat the evaluation harness of large language models (LLMs) as an independent variable and resolve its effect to single items. We introduce the \textit{fragility grid}: 12 open-weight instruction-tuned LLMs from 4 families answer the same 3{,}679 items from 4 benchmarks (ARC, HellaSwag, MMLU, TruthfulQA) under 26 equally defensible harness configurations, recording one correctness bit for every model, item, and configuration. The comparison is matched, since the items, the weights, and the greedy decoding stay fixed while only the harness varies. Under the grid a model's score is a band rather than a point: gemma4-31b scores between 31 and 89 percent depending only on the harness. Three results follow. On the items that two adjacent models both answer stably the pair is tied, and config-fragile items carry 95.7 percent of a pair's gap on average. Four of the 12 models reach rank one under some configuration, so the harness selects the winner. Item discrimination, the property that benchmark-compression methods maximize, correlates with fragility at 0.28 (95 percent CI 0.25 to 0.30), so compression keeps the fragile items rather than removing them. The scoring choice, not the option order that protocols usually fix, is the load-bearing axis. We release the per-item records and the analysis script, from which every number regenerates on a CPU in seconds, and we position the fragility grid as a check a leaderboard can run before it reports an order.
