A method for deciding how many AI judges to call, and when to stop calling more
Instead of always calling every available AI judge to score model outputs, this method uses a small labeled audit set to estimate whether each judge adds new information given the judges already in use. Judges that duplicate existing information get dropped, judges that broadly help get always called, judges that help only in specific situations get called only there, and the process stops once no more gains are worth the cost. Tested across reasoning, code, safety, preference, reward-model, summarization, and math evaluation tasks, the method matches or beats standard baselines while calling far fewer judges.
What they did
Classifies candidate judges (LLM prompts, reward models, safety classifiers, verifiers, etc.) into three roles: copy (redundant), complement (helps broadly), and specialist (helps only on specific slices)
Uses a small labeled audit set to measure each judge's validation-gain, adding it globally if the gain clears a threshold, routing it only to relevant slices if the gain is slice-specific, and dropping it otherwise
Compared across seven evaluation settings (GSM8K reasoning, MBPP code, JailbreakBench safety, LLMBar preference, RewardBench, SummEval, MATH-500) against single judges, flat panels, diversity-matched panels, full-call stacking, reliability juries, and frugal cascades
On LLMBar, routing specialists lifted accuracy from 0.6692 (flat panel) or 0.6822 (single best judge) to 0.7334 while using only about 3.46 calls; on a safety audit, the routed policy reached 0.1094 risk with 2.29 calls versus 0.1069 risk from a full 7-judge stack
Only pre-observable signals such as metadata, verifier or classifier outputs, or judge disagreement are allowed as routing signals at deployment time -- human ground-truth labels are used only for offline analysis, not for live routing
Figure 1: Action-oriented role taxonomy. A copy is redundant after conditioning on the current panel, a complement adds broad residual information, and a specialist adds value mainly on a deployable slice.
Table 1: Role taxonomy as a policy interface. Roles are target-relative, conditional on the current panel, and may overlap.
Role
Signal pattern
Policy implication
Copy
Broad and slice gains are below threshold.
Do not invoke by default.
Complement
Broad gain CP(j∣S) is above threshold.
Add to the global panel.
Specialist
Cost-adjusted slice gain clears the slice threshold.
Route to examples in the corresponding slice.
Comp. + spec.
Broad gain is high and concentrated on one or more slices.
Invoke globally; optionally prioritize on the specialist slice.
Figure 2: Risk-cost frontier across representative settings. Each point is a held-out policy evaluation averaged over 10 splits. Full-call stacking and full-call jury can be low-risk endpoints in broad-ensemble regimes, but require invoking every judge. Role policies occupy useful frontier regions when specialists or cheap verifiers matter.
Table 2: Experimental matrix. Each setting is included because it exercises a different deployment decision: add complements, route specialists, stop early, drop copies, or accept a full-call boundary. Human labels may define audit slices for analysis, but only metadata, verifier outputs, classifier outputs, or judge-disagreement proxies are deployable route signals.
Setting
Why it stresses allocation
Slice or route signal
Deployment status
Hard GSM8K rationale
Answer checking saturates, but rationale validity requires complementary LLM judgments.
Candidate generator and verifier agreement.
Available before final audit label.
MBPP public-overfit
A cheap hidden-test verifier can dominate some LLM signals but not all code-audit cases.
Public-test pass/fail and verifier agreement.
Available before final hidden-test label.
JailbreakBench
Safety judges have conditional value on unsafe and classifier-disagreement regions.
Classifier/disagreement proxy slices; human safety label is audit-only.
Deployable only for proxy slices, not for human-label slices.
LLMBar
Preference failures differ across natural and adversarial subsets, making specialist routing central.
Natural, adversarial instruction, adversarial output, and neighbor subsets.
Dataset metadata available before routing.
RewardBench / Arena100K
Broad preference comparisons test whether stopped panels should give way to full-call aggregation.
Saturated verifier cases test whether the method refuses unnecessary expansion.
Unit-test or answer-verifier result.
Verifier output available before routing.
Table 3: Main held-out policy comparison across hard reasoning audits, code overfit audits, safety, pairwise preference, reward modeling, and scalar summarization. Role policies expose few-judge complement panels, one-step stopping, specialist routing, and broad-ensemble endpoints.
Dataset
Single best
Flat all
Role routed stop
Risk
Acc.
Risk
Acc.
Risk
Acc.
Cost
Judges
Hard GSM8K rationale
0.2350
0.6253
0.2106
0.6670
0.2137
0.6843
2.90
2.90
MBPP public-overfit
0.0226
0.9767
0.0158
0.9617
0.0097
0.9900
1.52
1.70
JBB-7
0.1183
0.8349
0.1291
0.8409
0.1094
0.8527
2.29
2.29
LLMBar-7
0.2180
0.6822
0.2118
0.6692
0.1884
0.7334
3.46
3.46
RewardBench-7
0.0308
0.9678
0.0280
0.9615
0.0291
0.9660
1.80
1.80
Arena100K-7
0.2321
0.6257
0.2462
0.6186
0.2321
0.6257
1.00
1.00
SummEval-7 scalar
0.0450
–
0.0601
–
0.0450
–
1.00
1.00
MATH-500-5
0.0731
0.9167
0.0537
0.9309
0.0678
0.9202
1.70
1.70
Table 4: Strong baseline comparison. Full-call aggregation can be the best risk endpoint, but it invokes every judge. Role policies solve the deployment problem of deciding when to buy a small stopped panel, when to route specialists, and when to keep the full-call endpoint.
Setting
Best full-call
Best matched non-role
Best matched
non-role
Role policy
Best matched
non-role
Risk
Cost
Risk
Cost
Risk
Cost
Hard GSM8K rationale
0.1963
6.10
0.2114
2.90
0.2137
2.90
MBPP public-overfit
0.0067
6.10
0.0117
1.61
0.0097
1.52
JBB-7 DeepSeek
0.1069
7.00
0.1151
1.90
0.1094
2.29
LLMBar-7 DeepSeek
0.1804
7.00
0.1967
2.50
0.1884
3.46
LLMBar-7 Qwen3
0.2034
7.00
0.2190
2.20
0.2033
3.28
LLMBar-7 JudgeLM
0.1999
7.00
0.2215
2.30
0.2040
3.48
RewardBench-7 DeepSeek
0.0201
7.00
0.0284
1.50
0.0291
1.80
Arena100K-7 DeepSeek
0.2286
7.00
0.2321
1.00
0.2321
1.00
SummEval-7 DeepSeek
0.0446
7.00
0.0450
1.00
0.0450
1.00
MATH-500-5
0.0536
5.00
0.0668
1.70
0.0678
1.70
Table 5: SOTA-style allocation baselines. Reliability jury is full-call multi-annotator aggregation; frugal cascade is confidence-triggered budgeted routing. Role policies are most informative when useful judges are slice-conditional, as in deployable LLMBar slices and safety proxy/audit slices.
Setting
Reliability jury
Frugal cascade
Role policy
Risk
Cost
Risk
Cost
Risk
Cost
Hard GSM8K rationale
0.1957
6.10
0.2182
2.76
0.2137
2.90
MBPP public-overfit
0.0059
6.10
0.0225
1.06
0.0097
1.52
JBB-7 DeepSeek
0.1382
7.00
0.1213
1.43
0.1094
2.29
LLMBar-7 DeepSeek
0.2058
7.00
0.2107
2.52
0.1884
3.46
LLMBar-7 JudgeLM
0.2113
7.00
0.2337
2.60
0.2040
3.48
LLMBar-7 Qwen3
0.2232
7.00
0.2333
1.94
0.2033
3.28
RewardBench-7 DeepSeek
0.0268
7.00
0.0308
1.00
0.0291
1.80
MATH-500-5
0.0631
5.00
0.0658
1.67
0.0678
1.70
Table 6: Mechanism evidence for the four actions induced by the taxonomy: route, stop, drop, and select broad-ensemble endpoints.
Policy action
Evidence
Interpretation
Route specialists
LLMBar improves from flat-all risk 0.2118/accuracy 0.6692 to role risk 0.1884/accuracy 0.7334 at 3.46 calls; repeated routes appear on adversarial and natural subsets across DeepSeek, Qwen3, and JudgeLM anchors.
Useful judges are conditional on declared slices, not merely next in a global quality order.
Stop
Increasing τ reduces calls on hard GSM8K (3.10 to 1.30), MBPP (1.80 to 1.10), JBB (3.68 to 1.00), and MATH-500 (2.70 to 1.00); HumanEval and ordinary GSM8K stop after the verifier or strong single judge.
The method produces a practical validation-based stopping report for saturated targets.
Drop copies
Adding four exact copies to LLMBar and JBB leaves role risk/cost unchanged (LLMBar 0.1884/3.46; JBB 0.1094/2.29), while full-call jury cost rises to 11 and risk worsens (LLMBar 0.2860; JBB 0.1594).
Conditional gain identifies redundant signals even when nominal panel size and model count grow.
Expose boundaries
RewardBench and MATH-500 role policies are cheaper, but full-call stacking gives lower risk (0.0201 vs 0.0291; 0.0536 vs 0.0678).
The method is a regime detector: when broad ensemble information remains valuable and cost is acceptable, call the full panel.
Table 7: Representative routed specialists on LLMBar. Frequencies count how often a judge-slice route appears across 10 random splits.
Anchor
Slice
Routed judge
Frequency
DeepSeek
adversarial_gptinst
llama3_8b_v
6/10
DeepSeek
adversarial_neighbor
llama3_8b_v
5/10
DeepSeek
natural
gemma3_12b_v
6/10
Qwen3
adversarial_gptout
gemma3_12b_v
5/10
Qwen3
adversarial_neighbor
mistral_7b_v
6/10
Qwen3
natural
gemma3_12b_v
5/10
JudgeLM
adversarial_gptout
gemma3_12b_v
5/10
JudgeLM
adversarial_neighbor
mistral_7b_v
6/10
JudgeLM
natural
gemma3_12b_v
5/10
Table 8: Threshold sensitivity for role-routed stopping. Conservative thresholds reduce calls and provide an explicit stopping condition: add no remaining judge whose validation gain is below τ.
Dataset and threshold
Risk
Acc.
Cost
Judges
Hard GSM8K rationale, τ=0.001
0.2129
0.6790
3.10
3.10
Hard GSM8K rationale, τ=0.005
0.2137
0.6843
2.90
2.90
Hard GSM8K rationale, τ=0.020
0.2318
0.6373
1.30
1.30
MBPP public-overfit, τ=0.001
0.0078
0.9920
1.53
1.80
MBPP public-overfit, τ=0.005
0.0097
0.9900
1.52
1.70
MBPP public-overfit, τ=0.020
0.0206
0.9787
1.01
1.10
JBB, τ=0.001
0.1078
0.8688
3.68
3.68
JBB, τ=0.005
0.1094
0.8527
2.29
2.29
JBB, τ=0.020
0.1183
0.8349
1.00
1.00
LLMBar, τ=0.001
0.1954
0.7303
4.10
4.10
LLMBar, τ=0.005
0.1884
0.7334
3.46
3.46
LLMBar, τ=0.020
0.1834
0.7443
2.00
2.00
MATH-500, τ=0.001
0.0617
0.9209
2.70
2.70
MATH-500, τ=0.005
0.0678
0.9202
1.70
1.70
MATH-500, τ=0.020
0.0731
0.9167
1.00
1.00
Table 9: Redundant-copy stress test. Four exact copies of an existing judge increase the apparent pool size, but role-conditioned stopping ignores them after their conditional gain vanishes.
Setting
Condition
Flat all
Jury all
Cascade
Role
Risk
Cost
Risk
Cost
Risk
Cost
Risk
Cost
LLMBar
base
0.2118
7.00
0.2058
7.00
0.2107
2.52
0.1884
3.46
LLMBar
+4 copies
0.2118
11.00
0.2860
11.00
0.2107
3.80
0.1884
3.46
JBB
base
0.1291
7.00
0.1382
7.00
0.1213
1.43
0.1094
2.29
JBB
+4 copies
0.1291
11.00
0.1594
11.00
0.1213
1.65
0.1094
2.29
Table 10: JailbreakBench safety proxy-routing audit. The route signal is the dataset’s GPT-4 classifier field gpt4_cf (7; 35), which is stored separately from human-majority labels and is used here as a deployable proxy slice signal. Across 10 splits, the routed policy selected extra specialists on the classifier-safe proxy slice in 4 splits and on the classifier-unsafe proxy slice in 2 splits.
Policy
Risk (95% CI)
Acc.
Cost
Deployment reading
Single best
0.1183±0.0105
0.8349
1.00
Cheapest one-call reference.
Flat all
0.1291±0.0052
0.8409
7.00
All safety judges on every item.
Frugal cascade
0.1213±0.0115
0.8376
1.43
Uncertainty-triggered global order.
Full-call stack
0.1069±0.0034
0.8450
7.00
Best full-call risk endpoint.
Role global stop
0.1199±0.0098
0.8258
1.90
Stopped panel without proxy routing.
Role routed stop
0.1094±0.0106
0.8527
2.29
Specialists routed on gpt4_cf proxy slices.
Table 11: Judge-pool disclosure for the main experiments. The table reports the signals available to the allocation policy before it selects global calls, routed specialists, or stopped candidates.
DeepSeek/Qwen3/JudgeLM anchor plus Gemma-3, Llama-3, Mistral, Prometheus, Qwen2.5, and Selene preference judges.
subset
none
RewardBench / Arena100K / SummEval
DeepSeek, Gemma-3, Llama-3, Mistral, Prometheus, Qwen2.5, and Selene preference or scalar-summary judges.
subset or none
none
MATH-500
Llama-3.1, Mistral, Prometheus, Qwen2.5, and Selene math-correctness judges.
candidate model
none
HumanEval / GSM8K answer
Task verifier plus Llama-3.1, Mistral, Prometheus, Qwen2.5, Selene, and where available DeepSeek/Gemma correctness judges.
candidate model or kind
unit-test or answer verifier
Table 12: Complete-case filtering before policy construction. Rows with missing or unparseable outputs for the declared judge pool are excluded before the construction/final-test split.
Setting
Rows
Judges
Complete
Dropped
Main unparseable source
LLMBar-7
838
7
837
1
Selene 1
JBB-7
300
7
298
2
Prometheus 2
MBPP public-overfit
300
7
300
0
–
SummEval-7
1600
7
1600
0
–
Math/GSM8K correctness
300
8
300
0
–
LLMBar prompt variants
838
10
680
158
Prometheus-letter 155
Table 13: Split-level 95% confidence intervals for the main held-out risk comparisons. The role column uses the routed policy when routing is selected and the global stopped policy in one-step stopping regimes.
Setting
Single best risk
Flat-all risk
Role policy risk
Hard GSM8K rationale
0.2350±0.0055
0.2106±0.0116
0.2137±0.0082
MBPP public-overfit
0.0226±0.0039
0.0158±0.0050
0.0097±0.0096
JBB-7
0.1183±0.0105
0.1291±0.0052
0.1094±0.0106
LLMBar-7
0.2180±0.0116
0.2118±0.0098
0.1884±0.0143
RewardBench-7
0.0308±0.0015
0.0280±0.0019
0.0291±0.0019
Arena100K-7
0.2321±0.0034
0.2462±0.0046
0.2321±0.0034
SummEval-7 scalar
0.0450±0.0008
0.0601±0.0022
0.0450±0.0008
MATH-500-5
0.0731±0.0033
0.0537±0.0049
0.0678±0.0058
Table 14: Pattern-table sparsity for the selected role policy. Each row averages 10 random construction/test splits. Cells are occupied response-pattern cells in the refit construction calibrator; route cells report the largest selected slice-specific table in the split. Fallback is the fraction of validation or final-test examples whose invoked pattern was unseen in the corresponding fit table.
Setting
Fit/Val/Test
Calls
Global cells
Max route cells
Val fallback
Test fallback
LLMBar-7
209/209/419
3.46
8.3
23.7
8.7%
4.4%
JBB-7
74/75/149
2.29
4.8
5.0
4.3%
2.2%
MBPP public-overfit
75/75/150
1.70
3.4
0.0
0.0%
0.0%
MATH-500
200/201/401
1.70
3.4
0.0
0.2%
0.1%
SummEval-7
400/400/800
1.00
5.0
0.0
0.0%
0.0%
Table 15: Audit-size stability. The held-out test split is fixed for each seed; the construction audit set is reduced before fitting the role policy. Plan Jaccard compares the global and routed call set with the full-audit plan for the same split. Entries are means over 10 random splits.
Setting
Audit fraction
Risk
Accuracy
Cost
Plan Jaccard
LLMBar
50%
0.1904
0.7371
3.07
0.246
LLMBar
100%
0.1884
0.7334
3.46
1.000
JBB proxy
50%
0.1066
0.8570
1.98
0.238
JBB proxy
100%
0.1094
0.8527
2.29
1.000
MBPP public-overfit
50%
0.0306
0.9680
0.65
0.083
MBPP public-overfit
100%
0.0097
0.9900
1.52
1.000
Table 16: Slice-mix shift audit. Construction and deployment splits are disjoint and intentionally use different route-signal proportions. Entries are mean held-out risk/accuracy over 10 random shifted splits. The table is read as a deployment check on whether the audit labels match the next evaluation batch.
Setting
Construction mix
Deployment mix
Single R/A
Flat R/A
Role R/A
Calls
LLMBar-7
natural-heavy
adversarial-heavy
.220/.699
.237/.646
.233/.681
3.21
LLMBar-7
adversarial-heavy
natural-heavy
.175/.782
.215/.686
.175/.782
2.91
JBB-7
proxy-safe-heavy
proxy-unsafe-heavy
.147/.823
.154/.803
.147/.809
1.90
JBB-7
proxy-unsafe-heavy
proxy-safe-heavy
.125/.828
.147/.724
.109/.855
2.20
Table 17: Frequency audit for conservative deployment. Calls are listed when they appear in at least 5/10 random splits. Route entries use slice:judge; unqualified entries are global calls. Low-frequency routes are not deployment failures; they are prompts to collect more audit labels before relying on that specialist. This table is not a statistical guarantee.
Table 18: Production-cost sensitivity. Costs are normalized but non-uniform: API anchors are expensive, local judges are cheaper, and deterministic verifiers are cheapest. Entries are role-routed policy means over 10 random splits.
Setting
λ
Risk
Accuracy
Cost
Calls
LLMBar
0.000
0.1884
0.7334
7.46
3.46
LLMBar
0.002
0.1857
0.7468
7.08
3.08
LLMBar
0.005
0.1871
0.7302
5.05
2.62
JBB proxy
0.000
0.1094
0.8527
3.09
2.29
JBB proxy
0.002
0.1146
0.8389
1.82
1.82
JBB proxy
0.005
0.1169
0.8326
1.52
1.52
MBPP public-overfit
0.000
0.0097
0.9900
5.91
1.70
MBPP public-overfit
0.002
0.0141
0.9853
3.93
1.60
MBPP public-overfit
0.005
0.0224
0.9767
3.33
1.40
Table 19: Near-duplicate prompt-variant audit on LLMBar. The pool contains JSON-schema and letter-prompt outputs from five judge families. Role routing keeps prompt variants as conditional specialists rather than treating them as automatic diversity.
Policy
Risk ± 95% CI
Accuracy
Cost
Single best
0.2350±0.0035
0.6126
1.00
Flat all prompt variants
0.2633±0.0076
0.5668
10.00
Matched top-k
0.2320±0.0055
0.6335
2.10
Frugal confidence cascade
0.2351±0.0086
0.6300
1.92
Full-call logistic stack
0.2028±0.0042
0.6862
10.00
Role-routed stop
0.2122±0.0080
0.7143
3.64
Table 20: Pair-addition audit after greedy stopping. A pair-only move is a pair whose validation gain exceeds τ=0.005 after every remaining single addition falls below threshold.
Audit setting
Splits
Pair-only moves
Mean single gain
Max pair gain
LLMBar, seven judges
10
0
0.0008
0.0039
MBPP public-overfit
10
0
-0.0008
0.0008
JBB safety
10
3
0.0011
0.0116
LLMBar prompt variants
10
2
0.0011
0.0112
Why it matters
Calling every possible AI judge on every evaluation batch is expensive, while relying on just one judge risks missing failure cases; this method gives researchers an auditable, cost-aware policy for exactly which judges to call and when to stop. It turns a vague notion of 'judge diversity' into a concrete deployment decision that balances risk and cost for the next evaluation batch.
Terms in this paper
LLM-as-a-judge · using a language model to score or evaluate another model's outputs
judge panel · the set of judges (models, classifiers, or rules) assembled to evaluate outputs
slice · a subset of evaluation examples sharing some property, e.g. adversarial prompts only
reward model · a model trained to score outputs by predicting human preference
frugal cascade · a cost-saving strategy that calls extra, pricier judges only when the current prediction is uncertain
reliability jury · a method that statistically estimates each annotator's trustworthiness before combining their votes
Original abstract (English)
LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers. The deployment question is not only which judge is best, but which judges should be called, on which examples, and when panel construction should stop. We formulate judge-panel design as a role-conditioned allocation problem. From a small labeled audit set, declared slices, and judge costs, the method estimates target-relative roles: copies add no conditional information, complements improve the global panel, and specialists help only on slices. These roles induce a policy: drop copies, add complements globally, route specialists conditionally, and stop when validation gain falls below a threshold. Across reasoning, code, safety, preference, reward-model, summarization, and math audits, the method is compared with single judges, flat panels, matched diversity heuristics, full-call stacking, reliability juries, and frugal cascades. The result is a regime map for judge calls: route specialists on deployable slices, stop in saturated verifier regimes, keep broad ensembles when their risk benefit is worth the cost, and ignore conditional copies. The output is a reusable, auditable call plan for the next evaluation batch.
