METAL LAB

FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes

arXiv:2608.261292026-08-28

A dataset that finally teaches AI what biology, chemistry, and physics peer reviewers actually argue about, not just CS reviewers

Every peer-review dataset used to train AI came exclusively from Computer Science and Machine Learning venues, so models never saw a biology reviewer demand contamination controls or a chemist question NMR spectral assignments. FIRSTPASS draws on Nature Communications' mandatory transparent peer review to compile 3,668 multi-round dialogue records across biology, chemistry, neuroscience, physics, and earth science. Each record carries an outcome label taken directly from real editorial decisions (STANDARD for two rounds, EXTENDED for three or more), letting evaluation check alignment with actual editorial outcomes instead of stylistic resemblance.

METAL LAB explanatory visual

FIRSTPASS construction pipeline and three-task curriculum

Evidence statusMeasured results reported

  1. CollectionPapers from Nature Communications (Jan 2023-Dec 2025) pulled via Springer Nature API, parsed with Gemini, filtered to 3,668 records with 4 complete sections and 2+ review rounds
  2. LabelingOutcome label set by round count: STANDARD for two rounds, EXTENDED for three or more, consistent at 62.8-71.2% STANDARD across all five domains
  3. Task 1: Review GenerationModel generates Round 1 reviewer comments from paper content alone
  4. Task 2: Reviewer UpdatingModel generates Round 2 reviews given paper content, Round 1 reviews, and author responses, learning which rebuttals are convincing
  5. Task 3: Outcome PredictionModel predicts STANDARD or EXTENDED from the full dialogue; fine-tuned Qwen2.5-7B-Instruct reached 80.5% accuracy, 78.2% F1-macro
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. Prior datasets like PeerRead, ReviewMT, and MARG relied entirely on ACL, NIPS, and ICLR reviews, leaving models blind to review norms outside Computer Science and Machine Learning.
  2. The team queried the Springer Nature OpenAccess API for papers published January 2023 to December 2025 from Nature Communications, which has required transparent peer review since November 2022, then parsed PDFs with Gemini-3.1-flash-lite-preview, keeping only records where all four paper sections (abstract, introduction, methods, results) exceeded 20 words and at least two complete review rounds were present.
  3. Outcome labels were assigned by round count (two rounds = STANDARD, three or more = EXTENDED), and this label stayed consistent across all five domains at 62.8% to 71.2% STANDARD, indicating it captures a structural pattern of reviewer-author tension rather than a single field's procedural norm.
  4. An automated audit of all 3,668 retained records found zero hollow files and 100% content integrity, and expert reviews averaged 2,155 words, more than five times longer than typical ICLR reviews (about 400 words).
  5. Each paper produced three training tasks: review generation, reviewer updating, and outcome prediction; a fine-tuned Qwen2.5-7B-Instruct reached 80.5% accuracy and 78.2% F1-macro on the outcome prediction task, outperforming zero-shot Gemini-3.1-flash-lite-preview by 10.4 percentage points.
Figure 1: The FirstPass three-task training curriculum.
Figure 1: The FirstPass three-task training curriculum.
Table 1: FIRSTPASS dataset statistics by domain. STD = STANDARD (2 rounds), EXT = EXTENDED (3+ rounds).
DomainTotalTrainValTestSTD (%)EXT (%)
Biology741593747463.236.8
Chemistry744595757462.837.2
Neuroscience739591747464.635.4
Physics727582737266.733.3
Earth Sci.717573727271.228.8
Total3,6682,93436836665.434.6

Findings

  • An automated audit of all 3,668 retained Nature Communications records found zero hollow files and 100% content integrity.
  • The STANDARD label proportion stayed consistent across all five domains, ranging from 62.8% to 71.2%.
  • Expert reviews averaged 2,155 words, more than five times longer than typical ICLR reviews (about 400 words).
  • A fine-tuned Qwen2.5-7B-Instruct achieved 80.5% accuracy and 78.2% F1-macro on the outcome prediction task, outperforming zero-shot Gemini-3.1-flash-lite-preview by 10.4 percentage points.

Where it can be used

  • Training data for building AI review-assistance tools across multiple scientific disciplines rather than just Computer Science venues.
  • A benchmark for evaluating whether AI outcome-prediction models align with real editorial decisions.
  • A methodological template for validating AI judgment against real expert decisions in other domains such as clinical triage, grant evaluation, or regulatory review.

Limits and open work

  • The dataset is drawn from a single journal source, Nature Communications, so its editorial practices and biases may be embedded in the data.
  • The release covers only accepted papers, so it does not capture the review process for rejected submissions.
  • There is a risk of misuse for generating synthetic peer reviews, which the authors address with an explicit datasheet documenting known limitations and mitigations.
  • Labels rely purely on round count since decision-letter text was unavailable for 97.7% of records, limiting the granularity of the outcome label.

Why it matters

AI tools built to assist or automate paper review risk applying Computer-Science-shaped judgment to fields with entirely different evaluation norms, and FIRSTPASS provides a way to test and correct that bias against real editorial outcomes rather than surface-level review style. The full release of data, parsing pipelines, evaluation scripts, and model weights lets other researchers reproduce and extend the benchmark directly.

Terms in this paper

  • peer review · the process by which experts in a field assess a submitted paper and decide whether it should be published
  • F1-macro · the unweighted average of F1 scores computed separately for each class, treating all classes equally
  • STANDARD / EXTENDED · outcome labels marking whether a paper went through two review rounds (STANDARD) or three or more (EXTENDED)
  • zero-shot · having a model perform a task without any task-specific fine-tuning

Original abstract (English)

Scientific peer review datasets have trained AI systems exclusively on Computer Science and Machine Learning venues, producing models that critique ablation studies yet have never seen a biology reviewer demand contamination controls or a chemist question Nuclear Magnetic Resonance (NMR) spectral assignments. We introduce FIRSTPASS, the first large-scale peer review dataset built on complete multi-round editorial dialogues from a multidisciplinary high-impact journal. Curated from Nature Communications mandatory transparent peer review (instituted November 2022), FIRSTPASS comprises 3,668 records spanning five scientific domains (biology, chemistry, neuroscience, physics, and earth science), capturing the full iterative structure of scientific validation: initial referee reports, author point-by-point responses, and updated reviewer assessments. Each record carries an outcome label derived directly from editorial decisions (STANDARD for two-round review; EXTENDED for three or more rounds), providing ground truth absent in all prior corpora. An automated audit confirms 100% content integrity. Expert reviews average 2,155 words, substantially denser than conference venue reviews. All data, parsing pipelines, and evaluation scripts are released to enable reproducible benchmarking of AI scientific judgment across disciplines.

Authors · Prabhjot Singh, Somnath Luitel, Manmeet Singh, Josh Durkee

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Prabhjot Singh et al., arXiv:2608.26129, CC BY 4.0