매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis

arXiv:2608.199022026-08-21

AI가 뇌영상 분석을 대신할 때, '왜 이 결론을 믿어도 되는가'까지 기록하게 만든 시스템

Brain Researcher는 AI 에이전트가 뇌영상 데이터를 분석할 때 어떤 도구를 썼고 어떤 대안을 검토했으며 결론이 어디까지 유효한지를 자동으로 기록하고 검증하는 연구 플랫폼이다. 벤치마크에서 올바른 분석 도구를 첫 시도에 고르는 정확도가 23.3%에서 93.6%로, 근거를 실제로 확인할 수 있는 비율이 4.6%에서 22.0%로 올랐다. 실제 협업 연구 사례에서는 분석 방식을 수백 가지로 바꿔가며 결과가 얼마나 민감하게 달라지는지 드러냈고, 그 결과를 승인, 조건부 승인, 수정, 보류, 반려 등으로 분류해 심사했다.

무엇을 했나

  1. AI 에이전트가 과학 분석을 실행할 수는 있지만, 대안을 비교하지 않고 성급하게 성공을 선언하거나 입맛에 맞는 결과만 고르는 문제가 있었다. Brain Researcher는 이를 막기 위해 분석 전에 계획을 확정하고(커밋먼트 카드), 분석 후 결과와 근거를 기록한 문서(클레임 카드)를 남겨 다른 사람이 재실행 없이도 검증할 수 있게 했다.
  2. 745,949개 노드와 2,461,469개 연결로 이루어진 지식그래프(BR-KG)를 만들어 뇌 영역, 실험 과제, 논문 근거 등을 서로 연결하고, 모든 사실에 출처와 원문 인용을 붙여 추적 가능하게 했다.
  3. 7개의 최신 언어모델(Claude Opus, GPT-5.5, Gemini 3.1 Pro 등)로 도구 선택 정확도를 테스트한 결과, Brain Researcher 없이는 23.3%였던 정확도가 있을 때는 93.6%로 올랐고, 인용한 근거가 실제로 확인 가능한 비율도 4.6%에서 22.0%로 늘었다.
  4. 정신분열증, 코카인 사용 장애, 문화 간 사회인지 연구 등 실제 협업 연구에서 하나의 질문을 수백 가지 분석 방식(다중우주 분석)으로 반복 검증해 결과가 분석 선택에 얼마나 민감한지 드러냈고, 한 사례에서는 자동 검토가 놓친 부호 오류를 사람이 직접 찾아내기도 했다.
  5. 인간 커넥톰 프로젝트 데이터로 116개의 분석 파이프라인을 탐색해 인지능력 예측 성능(r=.487)을 찾아냈고, TRIBE라는 AI 모델의 내부 표현을 탐구하며 스스로 새로운 가설을 세우고 검증하는 자기진화형 연구 과정도 보여줬다.
Figure 1: Brain Researcher: workspace-centric infrastructure for auditable neuroimaging research. Brain Researcher runs inside the researcher’s existing computational environment and exposes neuroimaging analyses as structured, auditable operations: every choice, check, and input is recorded as it happens, so that a frozen record of the completed analysis can be read and audited by someone other than the person who ran it, without re-executing it. (a) The tool ecosystem: established neuroimaging software for preprocessing, modeling, meta-analysis, machine learning, quality control, and reporting, each represented by a machine-readable specification and executed in a version-pinned container. (b) Each specification declares its inputs, outputs, parameters, version, evidence anchors, and validation rules; the rule checker tests every proposed call against these clauses before it runs, and records which clauses passed or failed. (c) The episode workflow: the researcher frames a question and approves the plan at the commitment gate, valid actions are dispatched to version-pinned executors, and the resulting audit bundle, containing the committed plan, tool versions, evidence consulted, artifacts, logs, provenance, and the checks each claim passed, feeds the review layer, which writes condition-tagged claims back to memory. This audit bundle is what makes an analysis auditable: a completed run is a fully inspectable research object rather than a one-off result, exported as a compact claim card a reviewer can reopen field by field. Methodological judgment remains the researcher’s; the system makes it visible at each stage.
Figure 1: Brain Researcher: workspace-centric infrastructure for auditable neuroimaging research. Brain Researcher runs inside the researcher’s existing computational environment and exposes neuroimaging analyses as structured, auditable operations: every choice, check, and input is recorded as it happens, so that a frozen record of the completed analysis can be read and audited by someone other than the person who ran it, without re-executing it. (a) The tool ecosystem: established neuroimaging software for preprocessing, modeling, meta-analysis, machine learning, quality control, and reporting, each represented by a machine-readable specification and executed in a version-pinned container. (b) Each specification declares its inputs, outputs, parameters, version, evidence anchors, and validation rules; the rule checker tests every proposed call against these clauses before it runs, and records which clauses passed or failed. (c) The episode workflow: the researcher frames a question and approves the plan at the commitment gate, valid actions are dispatched to version-pinned executors, and the resulting audit bundle, containing the committed plan, tool versions, evidence consulted, artifacts, logs, provenance, and the checks each claim passed, feeds the review layer, which writes condition-tagged claims back to memory. This audit bundle is what makes an analysis auditable: a completed run is a fully inspectable research object rather than a one-off result, exported as a compact claim card a reviewer can reopen field by field. Methodological judgment remains the researcher’s; the system makes it visible at each stage.
Figure 2: BR-KG: provenance-linked semantic integration for grounded, auditable neuroimaging reasoning. BR-KG integrates existing ontologies, repositories, data resources, and literature into a single graph (745,949 nodes, 2,461,469 edges; 2026-07-07 release snapshot) aligned to the OpenNeuro Vocabulary (ONVOC) (44), which normalizes heterogeneous terms to shared identifiers so a query resolves consistently across sources. The central graph links neuroimaging concepts (tasks, contrasts, cognitive constructs), neural representations (brain regions, statistical maps), and research resources (datasets, tools) through typed relationships, with literature evidence attached. Crucially, source-backed facts carry explicit provenance (their source, and where available a verbatim supporting quote and grounding label), so a retrieved claim can be traced back to the study and passage that support it rather than taken on trust; coverage is partial and tracked, and this is what makes retrieval here auditable. Downstream panels show the payoff: grounded query answering and multi-hop reasoning over concept-task-map paths, each hop inspectable down to its underlying nodes, edges, and cited evidence, which lets the review layer attach a recommendation’s method-condition checks (cohort, paradigm, preprocessing, statistical model) before it is accepted.
Figure 2: BR-KG: provenance-linked semantic integration for grounded, auditable neuroimaging reasoning. BR-KG integrates existing ontologies, repositories, data resources, and literature into a single graph (745,949 nodes, 2,461,469 edges; 2026-07-07 release snapshot) aligned to the OpenNeuro Vocabulary (ONVOC) (44), which normalizes heterogeneous terms to shared identifiers so a query resolves consistently across sources. The central graph links neuroimaging concepts (tasks, contrasts, cognitive constructs), neural representations (brain regions, statistical maps), and research resources (datasets, tools) through typed relationships, with literature evidence attached. Crucially, source-backed facts carry explicit provenance (their source, and where available a verbatim supporting quote and grounding label), so a retrieved claim can be traced back to the study and passage that support it rather than taken on trust; coverage is partial and tracked, and this is what makes retrieval here auditable. Downstream panels show the payoff: grounded query answering and multi-hop reasoning over concept-task-map paths, each hop inspectable down to its underlying nodes, edges, and cited evidence, which lets the review layer attach a recommendation’s method-condition checks (cohort, paradigm, preprocessing, statistical model) before it is accepted.

왜 중요한가

AI가 과학 연구를 돕는 시대에 결과의 신뢰성을 어떻게 검증할지는 핵심 과제다. 이 연구는 AI 에이전트의 결론을 사람이 재현하지 않고도 검토할 수 있는 감사 가능한 기록 체계를 제시해, 연구 재현성과 책임성 문제에 실질적인 해법을 제공한다.

Figure 3: Summary of Brain Researcher effects across quantitative benchmark tasks. Without-BR (gray) and with-BR (blue) benchmark performance. Capability@k is mean coverage of required task capabilities after the first k non-neutral actions. The left column reports Capability@1 and @3 across the seven model variants (Claude Opus 4.8, Codex GPT-5.5, Gemini 3.1 Pro, GLM-5.1, DeepSeek-V4-Pro, Kimi K2.5, Qwen3.6-Plus). Upper-right panels break Capability@1 down by task domain. Lower panels report Handoff score@1 and @3 (whether the first route carries enough information for another agent to continue) and a Gemini 2.5 Flash single-judge safeguard: precision among claims marked grounded (fraction whose cited evidence was both locatable and judged supportive). Correct route/tool@1, the first-action selection accuracy reported in the text (23.3% to 93.6%), is detailed in Supplementary Methods S11.1.1. Metrics are interpreted within panel, as denominators and scoring rules differ across benchmarks.
Figure 3: Summary of Brain Researcher effects across quantitative benchmark tasks. Without-BR (gray) and with-BR (blue) benchmark performance. Capability@k is mean coverage of required task capabilities after the first k non-neutral actions. The left column reports Capability@1 and @3 across the seven model variants (Claude Opus 4.8, Codex GPT-5.5, Gemini 3.1 Pro, GLM-5.1, DeepSeek-V4-Pro, Kimi K2.5, Qwen3.6-Plus). Upper-right panels break Capability@1 down by task domain. Lower panels report Handoff score@1 and @3 (whether the first route carries enough information for another agent to continue) and a Gemini 2.5 Flash single-judge safeguard: precision among claims marked grounded (fraction whose cited evidence was both locatable and judged supportive). Correct route/tool@1, the first-action selection accuracy reported in the text (23.3% to 93.6%), is detailed in Supplementary Methods S11.1.1. Metrics are interpreted within panel, as denominators and scoring rules differ across benchmarks.
Figure 4: Multiverse sensitivity and claim-review outcomes across three collaborator episodes. (A–C) Schizophrenia NeuroMark audit: (A) group-mean functional connectivity for controls (HC, N=181), patients (SZ, N=182), and their difference across four estimators; (B) NM-H2 (between- versus within-domain) specification curve over the 480-specification multiverse; after sign-aware rescoring, its estimand comprises 24 unique connectivity–confound–domain contrasts, with favorable support at 100% for Pearson and Spearman and 0% for partial correlation and mutual information. This complete estimator partition, rather than the pooled 12-of-24 fraction, is the informative result: NM-H2 is measure-dependent, and the mechanism underlying the partition remains unresolved. (C) Marginal influence of each analytic choice on NM-H2. (D, E) Cocaine-use-disorder episode: (D) multiverse stability of systemic-segregation associations across 36 specifications with SDMA-GLS consensus; (E) single-specification versus multiverse SDMA-GLS maps for five network–outcome pairs. (F) Cross-cultural social cognition: culture-stratified ALE maps contrasting Euro-American trust networks with East Asian social-cognition networks.
Figure 4: Multiverse sensitivity and claim-review outcomes across three collaborator episodes. (A–C) Schizophrenia NeuroMark audit: (A) group-mean functional connectivity for controls (HC, N=181), patients (SZ, N=182), and their difference across four estimators; (B) NM-H2 (between- versus within-domain) specification curve over the 480-specification multiverse; after sign-aware rescoring, its estimand comprises 24 unique connectivity–confound–domain contrasts, with favorable support at 100% for Pearson and Spearman and 0% for partial correlation and mutual information. This complete estimator partition, rather than the pooled 12-of-24 fraction, is the informative result: NM-H2 is measure-dependent, and the mechanism underlying the partition remains unresolved. (C) Marginal influence of each analytic choice on NM-H2. (D, E) Cocaine-use-disorder episode: (D) multiverse stability of systemic-segregation associations across 36 specifications with SDMA-GLS consensus; (E) single-specification versus multiverse SDMA-GLS maps for five network–outcome pairs. (F) Cross-cultural social cognition: culture-stratified ALE maps contrasting Euro-American trust networks with East Asian social-cognition networks.

이 논문의 용어

  • 다중우주 분석(multiverse analysis) · 하나의 연구 질문에 대해 가능한 여러 분석 방법을 모두 시도해 결과가 분석 선택에 얼마나 민감한지 확인하는 방법
  • 커밋먼트 카드 · 분석을 실행하기 전에 질문, 허용된 대안, 성공·실패 기준을 미리 고정해 기록한 문서
  • 클레임 카드 · 분석이 끝난 뒤 결과와 그 결과가 통과한 검증 항목을 기록한 문서로, 다른 사람이 재실행 없이 검토할 수 있게 함
  • 지식그래프(BR-KG) · 뇌 영역, 실험 과제, 논문 근거 등을 서로 연결해 출처를 추적할 수 있게 만든 대규모 데이터 구조
  • 그라운딩(grounding) · AI가 인용한 근거가 실제로 찾을 수 있고 내용을 뒷받침하는지 확인하는 것
Figure 5: Brain Researcher searches 116 HCP prediction pipelines and identifies a workflow that consistently exceeds a matched reference. A. Brain Researcher first evaluated 20 candidate pipelines for Cognition prediction, reaching a best discovery score of r=.373. Brain researcher then launched a 96-candidate expansion; 84 candidates returned scores and 12 ended in transport failure. Within the expanded episode, Brain Researcher adapted its proposals to the accumulating results: 27 candidates exceeded the initial search maximum, and the highest discovery score was r=.487, obtained with whole-band coherence and ridge regression. Following a selector audit, the researcher froze a related coherence-based workflow for matched evaluation. Across 10 repeated family-grouped 5×3 nested-cross-validation runs, this workflow achieved median r=.332, compared with .235 for the matched reference (median Δ​r=.098; conditional one-sided p=.006), and was higher in all 10 runs. B. The same frozen selected workflow was then refit for each of four additional behavioural outcomes without target-specific retuning. It produced a higher median correlation for every outcome and exceeded the matched reference in 37 of 40 repeat-level comparisons, giving 47 of 50 directional wins across all five outcomes.
Figure 5: Brain Researcher searches 116 HCP prediction pipelines and identifies a workflow that consistently exceeds a matched reference. A. Brain Researcher first evaluated 20 candidate pipelines for Cognition prediction, reaching a best discovery score of r=.373. Brain researcher then launched a 96-candidate expansion; 84 candidates returned scores and 12 ended in transport failure. Within the expanded episode, Brain Researcher adapted its proposals to the accumulating results: 27 candidates exceeded the initial search maximum, and the highest discovery score was r=.487, obtained with whole-band coherence and ridge regression. Following a selector audit, the researcher froze a related coherence-based workflow for matched evaluation. Across 10 repeated family-grouped 5×3 nested-cross-validation runs, this workflow achieved median r=.332, compared with .235 for the matched reference (median Δ​r=.098; conditional one-sided p=.006), and was higher in all 10 runs. B. The same frozen selected workflow was then refit for each of four additional behavioural outcomes without target-specific retuning. It produced a higher median correlation for every outcome and exceeded the matched reference in 37 of 40 repeat-level comparisons, giving 47 of 50 directional wins across all five outcomes.
Figure 6: Brain Researcher turns an open question about TRIBE into successive tests with new sounds and collections. A. Brain Researcher began by asking how TRIBE changes natural-sound representations from early to late layers. It screened category contrasts by their change in held-out distinguishability (AUC), without choosing a target in advance. Tools–voice changed most, but the pattern varied across collections. Rather than simply following the top-ranked result, Brain Researcher identified speech–tools as a clearer lead: the categories moved closer in later layers while usually keeping the same representational direction. The contrast could also be retested with new, acoustically matched sounds from several collections. The researcher approved this direction and froze the prediction and analysis. B. Brain Researcher then evaluated three non-overlapping 48-item panels. All three showed a smaller speech–tools separation on average in later layers. In 11 of 12 collection-by-panel comparisons, the categories became less separated while retaining the prespecified direction. C. After the pattern recurred across all three panels, Brain Researcher extended the test to four previously unused sound collections. Three of four showed the same geometry, and the late-layer separation was again smaller on average (Δ​S=−0.198). In all geometry plots, horizontal position shows the late-minus-early change in normalized separation (Δ​S), and vertical position shows late directional alignment (C); the upper-left quadrant therefore marks smaller separation with retained direction. A uses fold-specific references, whereas B and C use the frozen speech–tools reference.
Figure 6: Brain Researcher turns an open question about TRIBE into successive tests with new sounds and collections. A. Brain Researcher began by asking how TRIBE changes natural-sound representations from early to late layers. It screened category contrasts by their change in held-out distinguishability (AUC), without choosing a target in advance. Tools–voice changed most, but the pattern varied across collections. Rather than simply following the top-ranked result, Brain Researcher identified speech–tools as a clearer lead: the categories moved closer in later layers while usually keeping the same representational direction. The contrast could also be retested with new, acoustically matched sounds from several collections. The researcher approved this direction and froze the prediction and analysis. B. Brain Researcher then evaluated three non-overlapping 48-item panels. All three showed a smaller speech–tools separation on average in later layers. In 11 of 12 collection-by-panel comparisons, the categories became less separated while retaining the prespecified direction. C. After the pattern recurred across all three panels, Brain Researcher extended the test to four previously unused sound collections. Three of four showed the same geometry, and the late-layer separation was again smaller on average (Δ​S=−0.198). In all geometry plots, horizontal position shows the late-minus-early change in normalized separation (Δ​S), and vertical position shows late directional alignment (C); the upper-left quadrant therefore marks smaller separation with retained direction. A uses fold-specific references, whereas B and C use the frozen speech–tools reference.

논문 원문 초록 (영문)

AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence supports. Agents may reproduce failures including selective analysis, premature declarations of success and optimization of imperfect criteria. We present Brain Researcher, an agentic research harness operating in a neuroimaging researcher's computational environment under rules for admissible analyses, required checks and claim scope. In benchmarks, Brain Researcher increased first-choice tool-selection accuracy across seven models by 70.2 percentage points (23.3% without it versus 93.6% with it) and verifiable grounding from 4.6% to 22.0%. In collaborator-led and self-evolving studies, multiverse analyses exposed analytic-choice sensitivity, and scientific review classified claims as accepted, qualified, revised, blocked, rejected or deferred. By linking decisions to evidence and provenance, Brain Researcher embeds methodological judgment within the workflow, not after it.

저자 · Zijiao Chen, Nicholas Lu, Xinhui Li, Jocelyn A. Ricard, Ce Ju, Huan H. Wang, Christian Kindermann, Jeanette A. Mumford

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Zijiao Chen et al., arXiv:2608.19902, arxiv-nonexclusive