HealMed: Multilingual Evaluation of Large Language Models in Medicine

arXiv:2608.199812026-08-21

医疗AI用英语很厉害,换成斯瓦希里语、祖鲁语就大幅失手

研究团队构建了医疗领域多语言评测基准HealMed,涵盖九种语言、九个源数据集和三类任务:选择题、句子关系判断、开放式问答。九个国家和地区的23名医生及医学专家历时两年,对机器翻译的题目进行了两轮审校和修改。结果显示,顶级商业模型在各语言间表现稳定,而开源模型和医疗专用模型在泰语、斯瓦希里语、祖鲁语等资源较少的语言上明显掉分。

他们做了什么

HealMed在九种语言(英语、德语、西班牙语、葡萄牙语、日语、汉语、泰语、斯瓦希里语、祖鲁语)中各收录1000道题,涵盖选择题(MCQA)、自然语言推理(NLI)、开放式问答三种任务形式。
每条机器翻译内容都经过23名医生和医学专家的两阶段审校修订,从而把翻译错误和模型自身能力的差距区分开来。
五个商业模型(包括GPT-5.4、Claude、Gemini系列)在资源较少的语言上准确率最多只下降5.5个百分点,而部分开源模型和医疗专用模型的降幅最高达28.9个百分点。
专门针对医疗训练的模型(HuatuoGPT-o1、MedGemma、MediPhi)同样不能保证跨语言表现稳定。
同一批题目分别用未经审校的机器翻译版和专家审校版测试,得分结果并不一致,说明翻译质量本身会影响评测结果的可信度。

Figure 1: Overview of HealMed. a. Benchmark scope, comprising MCQA, NLI and open-ended QA across nine languages and nine source datasets. b. Construction pipeline, including English source-example selection, machine translation into eight target languages, two-stage review and revision by bilingual medical experts, and final consistency checking and curation.

Table 1: Language-specific refusal rates in open-ended QA. Values are percentages of 400 zero-shot responses per language. A refusal was counted only when the model explicitly declined to answer and provided no substantive response. Overall rates were calculated across 3,600 responses per model. Gemini-3 denotes Gemini-3-Flash.
Language	GPT-4o	GPT-5.4	o4-mini
English	0.00	0.00	0.00
German	0.00	0.00	0.00
Spanish	0.00	0.00	0.00
Portuguese	0.00	0.25	0.00
Japanese	0.00	0.25	0.00
Chinese	0.25	0.00	0.00
Thai	0.25	0.00	0.25
Swahili	1.50	0.00	0.00
Zulu	2.00	0.00	0.00
Overall	0.44	0.06	0.03

Figure 2: Multilingual performance on the MCQA and NLI tasks in HealMed. a. Macro-average accuracy across four MCQA and two NLI datasets for 14 models. Blue bars show the lower-resource mean, and red extensions show the gap to the higher-resource mean. Circles, squares and diamonds denote proprietary, open-source and medically specialized models, respectively. b. Language-level accuracy across models. c. Within-model accuracy shifts relative to English. Grey points represent individual models; colored points and horizontal lines show the mean and interquartile range. Black, blue and red denote English, other higher-resource languages and lower-resource languages, respectively. Higher-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese; lower-resource languages comprise Thai, Swahili and Zulu.

Table 2: Expert assessment and revision of machine-translated data by language. Accuracy (Acc.), fluency (Flu.) and completeness (Comp.) are mean expert ratings on five-point scales. Revision is the mean normalized word-level edit distance between the original machine translations and reviewer-submitted revisions. Each language contains 1,000 instances.
Language	Acc.	Flu.	Comp.	Revision (%)
German	4.87	4.75	4.83	12.1
Spanish	4.92	4.92	4.85	1.0
Portuguese	4.74	4.84	4.93	2.7
Japanese	4.70	4.62	4.96	5.7
Chinese	4.73	4.51	4.98	5.2
Thai	4.71	4.79	4.93	9.1
Swahili	4.50	4.57	4.63	1.8
Zulu	4.60	4.68	4.92	1.1
Overall	4.72	4.71	4.88	4.8

Figure 3: Open-ended QA performance on HealMed. a. Mean LLM-as-judge scores for ten models, macro-averaged across the three QA datasets. Blue and red points show higher- and lower-resource means; diamonds show means across all languages. b. Score shifts relative to each model’s English score. Positive values indicate higher scores than the English baseline. Grey lines show individual models and the dark line shows their mean. c. Proportions of responses with an overall score below 3, a safety score of 2 or lower, or specific language and applicability issues. Other higher-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese. Bubble size and color indicate the proportion; values are shown when at least 10%.

Table 3: Criterion-level comparison of expert and LLM-based evaluations. Scores range from 1 to 5. Δ denotes the LLM score minus the expert score and was calculated before rounding.
Criterion	Expert	LLM	Δ
Chinese
Completeness	4.22	3.80	−0.42
Reference alignment	4.24	3.62	−0.62
Clinical consensus	4.33	4.04	−0.29
Clinical appropriateness	4.31	3.93	−0.38
Safety	4.69	4.20	−0.49
Overall	4.36	3.92	−0.44
Japanese
Completeness	4.20	3.44	−0.76
Reference alignment	4.76	3.71	−1.04
Clinical consensus	4.82	4.20	−0.62
Clinical appropriateness	4.78	4.02	−0.76
Safety	4.87	4.27	−0.60
Overall	4.68	3.93	−0.76
Thai
Completeness	3.28	3.38	+0.10
Reference alignment	3.64	3.62	−0.02
Clinical consensus	4.12	4.11	−0.01
Clinical appropriateness	3.80	3.91	+0.11
Safety	4.36	4.22	−0.13
Overall	3.84	3.85	+0.01

Figure 4: Evaluation shifts between expert-reviewed HealMed and machine-translated (MT) data. a. English-adjusted mean accuracy shifts (HealMed minus MT) across 14 models and six MCQA and NLI datasets. Cells are labelled when |Δ|≥3 percentage points (pp). b. English-adjusted mean LLM-as-judge score shifts across five models and three open-ended QA datasets. Symbols denote datasets, and horizontal lines span their mean shifts. In both panels, Overall reports the mean absolute model-level shift across the corresponding models and datasets. Positive values indicate higher performance on expert-reviewed data; English is shown unadjusted as a same-source control.

Table 4: Language use in observable reasoning traces generated by HuatuoGPT-o1-72B. The “Traces” column denotes the number of responses containing an explicit reasoning trace among 15 responses examined per language. ”Target” and ”English” report the number and percentage of traces written predominantly in the target language or English, respectively.
Language	Traces	Target	English
Japanese	9/15	7 (77.8%)	2 (22.2%)
Chinese	15/15	15 (100%)	0 (0%)
Thai	7/15	0 (0%)	7 (100%)

Table 5: Benchmark components and sample allocation in HealMed. Sample counts denote the number of aligned examples included in each language.
Task	Component	Samples per language, n	Expected model output
MCQA	HeadQA	75	Correct option
	MedQA	75	Correct option
	MedExpQA	75	Correct option
	MMLU-Pro	75	Correct option
NLI	BioNLI	150	Relation label
	MedNLI	150	Relation label
QA	ExpertQA-Bio	40	Free-form answer
	ExpertQA-Med	160	Free-form answer
	LiveQA	200	Free-form answer
Total	1,000

Table 6: Comparison of selected multilingual medical benchmarks. Reported scales use source-specific units and are not directly comparable. Quality metrics indicates whether aggregate translation-quality scores or revision statistics were reported; score effects indicates whether model performance was compared between machine-translated and expert-reviewed versions. WorldMedQA-V identified four country-level validators and seven contributors to English-translation validation, but did not report the number of unique reviewers because these roles may overlap. EN, English; MT, machine translation; N/A, not applicable; NR, not reported. For BRIDGE, reference standards were inherited from the source datasets, benchmark-wide human-review coverage and reviewer numbers were not reported.
	Benchmark scope	Data construction	Human review	Translation audit
Benchmark	Reported scale	Lang.	Tasks	Language design	Review coverage (%)	Reviewers, n	Quality metrics	Score effects
MMedBench	53,566 QA pairs	6	MCQA; rationales	Aggregated; non-aligned	14.1	3	No	No
XMedBench	21,326 records	6	MCQA	Native + MT; non-aligned	10.2	NR	No	No
MedExpQA	2,488 records	4	MCQA	MT + manual revision; aligned	100	NR	No	No
WorldMedQA-V	568 evaluation items	4 + EN	Multimodal MCQA	Local–English pairs	100	4 country 7 EN	No	No
MultiMed-X	2,450 translations	7 + EN	NLI; open QA	MT + expert revision; aligned	100	∼12	No	No
BRIDGE	1,418,042 samples	9	8 task types	Native-source aggregation; non-aligned	Varies by source; NR	NR	N/A	N/A
HealMed	9,000 instances	9	MCQA; NLI; open QA	MT + two-expert medical revision; aligned	100	23	Yes	Yes

Table 7: Models and evaluation configurations used in HealMed.
Model	Parameters	Release date	Model type
Proprietary models
GPT-5.4	Not disclosed	5 Mar 2026	Proprietary
o4-mini	Not disclosed	16 Apr 2025	Proprietary
Gemini-3-Flash	Not disclosed	17 Dec 2025	Proprietary
Claude-Sonnet-5	Not disclosed	30 Jun 2026	Proprietary
Claude-Opus-4.8	Not disclosed	28 May 2026	Proprietary
Open-source models
DeepSeek-V3	671B	26 Dec 2024	open-source, general-purpose
Qwen2.5-72B-Instruct	72.7B	19 Sep 2024	open-source, general-purpose
Qwen3-32B	32.8B	29 Apr 2025	open-source, general-purpose
Qwen3-32B-thinking	32.8B	29 Apr 2025	open-source, general-purpose
LLaMA3.3-70B-Instruct	70B	6 Dec 2024	open-source, general-purpose
Gemma-3-27B-it	27B	12 Mar 2025	open-source, general-purpose
Medically specialized models
HuatuoGPT-o1-72B	72.7B	28 Dec 2024	open-source, medical
MedGemma-27B-text-it	27B	20 May 2025	open-source, medical
MediPhi	3.8B	3 Feb 2025	open-source, medical

Table 8: Five-point rubric used by medical experts to assess machine translations. Each dimension was scored separately.
Score	Accuracy	Fluency	Completeness
5	All concepts and medical terms are translated correctly and precisely. Terminology is professional, contextually appropriate and consistent with established usage in the target language.	The translation is natural and easy to read. Its grammar, wording and sentence structure conform to professional conventions in the target language.	The meaning and all relevant details of the source are retained. There are no omissions or unsupported additions.
4	Most concepts and terms are translated correctly. Minor errors, imprecise wording or simplified terminology may occur but do not affect overall understanding.	The translation is generally natural and clear. Minor stiffness, awkward wording or grammatical errors do not affect comprehension.	The main meaning is retained. Only minor or non-essential details are omitted or expressed unclearly.
3	The main concepts are conveyed, but some errors or imprecise terms may cause partial misunderstanding. The reader may need to infer the intended meaning of some terms.	The translation is understandable but noticeably unnatural in places. Rigid sentence structures, unsuitable word choices or grammatical errors require some effort from the reader.	Most of the source meaning is conveyed, but some information is missing, added or unclear. Important details may require inference.
2	Several important concepts or terms are mistranslated, substantially affecting comprehension. Terminology may be incorrect or inconsistent.	The translation is difficult to read smoothly. It contains awkward transitions, unclear connections or frequent grammatical and structural errors.	Core information is not fully preserved. Noticeable omissions or unnecessary additions reduce correspondence with the source and affect comprehension.
1	Frequent and severe mistranslations prevent the source meaning from being conveyed. Much of the content or terminology does not correspond to the source.	The translation is highly unnatural or difficult to understand. Literal phrasing, disorganized sentence structure and severe grammatical errors may make it unreadable.	Substantial omissions or incorrect additions prevent the translation from reflecting the source. Important passages are missing, and the intended meaning is difficult to recover.

Table 9: Inference settings used for model evaluation.
Setting	MCQA	BioNLI	MedNLI	Open-ended QA
Prompting strategy	zero-shot	zero-shot	zero-shot	zero-shot
Temperature	0.7 where supported; otherwise omitted
Maximum output tokens	32	32	32	2,048
Top-p	Not set; model- or provider-default
Top-k	Not set; model- or provider-default where supported
Random seed	Not set
Stop sequence	Not set

Table 10: Complete zero-shot accuracy (%) on the four MCQA datasets. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined.
Dataset	Model	EN	DE	ES	PT	JA	ZH	TH	SW	ZU
HeadQA	Proprietary Models
Gemini-3-Flash	94.67	94.67	96.00	96.00	92.00	92.00	90.67	92.00	88.00	92.89
GPT-5.4	90.67	93.33	92.00	93.33	89.33	89.33	86.67	90.67	84.00	89.93
o4-mini	96.00	94.67	97.33	96.00	92.00	90.67	94.67	85.33	85.33	92.44
Claude-Opus-4.8	96.00	96.00	96.00	94.67	90.67	90.67	89.33	93.33	82.67	92.15
Claude-Sonnet-5	90.67	96.00	94.67	94.67	92.00	90.67	93.33	90.67	73.33	90.67
open-source Models
DeepSeek-V3	82.67	84.00	80.00	84.00	82.67	82.67	72.00	60.00	54.67	75.85
Gemma-3-27B-it	80.00	78.67	82.67	78.67	78.67	73.33	74.67	66.67	54.67	74.22
LLaMA3.3-70B-Instruct	85.33	80.00	84.00	84.00	77.33	69.33	73.33	65.33	46.67	73.92
Qwen2.5-72B-Instruct	80.00	80.00	81.33	82.67	82.67	78.67	76.00	37.33	9.33	67.56
Qwen3-32B	76.00	74.67	73.33	74.67	66.67	72.00	58.67	45.33	8.00	61.04
Qwen3-32B-thinking	92.00	90.67	92.00	93.33	90.67	86.67	88.00	65.33	24.00	80.30
Specialized Models
HuatuoGPT-o1-72B	89.33	84.00	90.67	81.33	86.67	86.67	86.67	56.00	41.33	78.07
MedGemma-27B	90.67	78.67	81.33	78.67	80.00	74.67	72.00	60.00	40.00	72.89
MediPhi	70.67	49.33	60.00	60.00	45.33	41.33	37.33	26.67	16.00	45.18
MedQA	Proprietary Models
Gemini-3-Flash	94.67	92.00	94.67	93.33	90.67	93.33	93.33	93.33	92.00	93.04
GPT-5.4	92.00	96.00	96.00	97.33	93.33	90.67	97.33	97.33	88.00	94.22
o4-mini	98.67	97.33	96.00	97.33	93.33	96.00	98.67	94.67	92.00	96.00
Claude-Opus-4.8	98.67	93.33	98.67	97.33	92.00	92.00	88.00	89.33	78.67	92.00
Claude-Sonnet-5	88.00	89.33	88.00	94.67	86.67	90.67	85.33	90.67	74.67	87.56
open-source Models
DeepSeek-V3	76.00	77.33	72.00	74.67	69.33	72.00	72.00	58.67	42.67	68.30
Gemma-3-27B-it	65.33	61.33	60.00	64.00	58.67	68.00	57.33	57.33	49.33	60.15
LLaMA3.3-70B-Instruct	85.33	82.67	80.00	86.67	72.00	80.00	81.33	69.33	34.67	74.67
Qwen2.5-72B-Instruct	76.00	78.67	74.67	74.67	72.00	73.33	69.33	42.67	22.67	64.89
Qwen3-32B	62.67	57.33	65.33	54.67	53.33	62.67	69.33	46.67	4.00	52.89
Qwen3-32B-thinking	89.33	90.67	89.33	94.67	84.00	85.33	93.33	70.67	36.00	81.48

Table 11: Complete zero-shot accuracy (%) on the two NLI datasets. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined.
Dataset	Model	EN	DE	ES	PT	JA	ZH	TH	SW	ZU
BioNLI	Proprietary Models
Gemini-3-Flash	79.33	71.33	74.00	72.67	71.33	71.33	74.67	73.33	70.00	73.11
GPT-5.4	73.33	68.00	70.00	68.67	68.00	68.67	68.00	67.33	66.67	68.74
o4-mini	74.00	71.33	71.33	71.33	71.33	70.67	70.00	66.67	63.33	70.00
Claude-Opus-4.8	78.00	70.67	74.67	72.00	72.00	76.00	74.00	74.67	70.67	73.63
Claude-Sonnet-5	72.67	68.00	71.33	68.00	68.00	69.33	68.67	67.33	62.00	68.37
open-source Models
DeepSeek-V3	77.33	65.33	64.67	62.67	62.00	70.67	62.67	60.00	49.33	63.85
Gemma-3-27B-it	66.67	64.67	64.00	62.00	63.33	65.33	60.00	58.00	56.00	62.22
LLaMA3.3-70B-Instruct	71.33	56.00	64.00	64.67	63.33	62.00	61.33	59.33	49.33	61.26
Qwen2.5-72B-Instruct	75.33	67.33	72.67	66.67	60.67	65.33	57.33	58.67	58.67	64.74
Qwen3-32B	70.67	61.33	70.67	66.00	58.00	60.67	58.00	48.67	55.33	61.04
Qwen3-32B-thinking	74.00	66.67	70.00	68.00	68.00	66.00	67.33	62.67	35.33	64.22
Specialized Models
HuatuoGPT-o1-72B	66.67	64.00	66.67	64.00	62.00	60.67	63.33	58.67	61.33	63.04
MedGemma-27B	59.33	58.00	64.67	67.33	59.33	50.67	53.33	51.33	54.00	57.55
MediPhi	66.67	63.33	63.33	62.67	56.67	61.33	57.33	47.33	52.67	59.04
MedNLI	Proprietary Models
Gemini-3-Flash	91.33	88.67	88.67	87.33	86.67	82.67	86.67	82.00	76.67	85.63
GPT-5.4	85.33	84.67	82.00	81.33	82.67	80.67	84.00	82.00	79.33	82.44
o4-mini	90.67	90.00	87.33	86.67	84.00	81.33	84.00	84.67	79.33	85.33
Claude-Opus-4.8	89.33	88.00	88.00	86.67	86.00	83.33	89.33	86.67	84.67	86.89
Claude-Sonnet-5	91.33	93.33	90.00	92.67	87.33	87.33	88.00	85.33	81.33	88.52
open-source Models
DeepSeek-V3	84.67	68.67	75.33	72.00	78.67	75.33	76.67	62.67	44.00	70.89
Gemma-3-27B-it	88.67	86.67	86.00	83.33	84.00	76.00	78.00	76.67	57.33	79.63
LLaMA3.3-70B-Instruct	81.33	74.67	76.00	77.33	80.00	70.00	70.67	68.00	30.67	69.85
Qwen2.5-72B-Instruct	86.67	82.00	84.00	84.00	82.67	78.00	80.00	57.33	32.67	74.15
Qwen3-32B	75.33	75.33	82.67	83.33	81.33	74.00	67.33	34.67	30.67	67.18
Qwen3-32B-thinking	83.33	81.33	82.67	80.00	80.00	79.33	80.67	65.33	36.00	74.30

Table 12: Complete LLM-as-judge scores on the three open-ended QA datasets. Scores range from 1 to 5 and are calculated as the mean of completeness, reference alignment, clinical consensus, clinical appropriateness and safety. Avg. denotes the macro-average across the nine languages. Within each dataset and language, the highest value is shown in bold and the second-highest is underlined.
Dataset	Model	EN	DE	ES	PT	JA	ZH	TH	SW	ZU
ExpertQA-Bio	Proprietary Models
GPT-5.4	4.08	4.29	4.22	4.28	4.19	4.53	4.52	4.12	4.03	4.25
Gemini-3-Flash	3.81	3.90	3.91	3.98	3.99	4.19	4.01	3.89	4.02	3.97
open-source Models
DeepSeek-V3	4.38	3.94	3.96	4.08	3.79	4.08	3.99	3.55	3.30	3.89
Gemma-3-27B-it	3.87	3.73	3.90	3.78	3.59	4.02	3.72	3.38	3.00	3.66
Qwen3-32B-thinking	4.29	3.87	3.92	4.16	3.85	4.17	3.92	2.59	2.21	3.66
Qwen2.5-72B-Instruct	4.10	3.97	4.05	4.03	3.85	4.21	3.95	2.09	1.97	3.58
LLaMA3.3-70B-Instruct	4.19	3.64	3.80	3.93	3.40	3.72	3.32	3.36	2.30	3.51
Specialized Models
HuatuoGPT-o1-72B	4.04	3.86	3.97	4.00	3.82	3.93	3.85	2.71	2.25	3.60
MedGemma-27B	3.94	3.81	3.98	4.13	3.71	3.97	3.68	3.60	2.90	3.74
MediPhi	3.96	3.52	3.57	3.59	3.09	3.19	2.31	2.02	1.87	3.01
ExpertQA-Med	Proprietary Models
GPT-5.4	4.04	4.06	4.24	4.41	4.30	4.50	4.46	4.33	4.19	4.28
Gemini-3-Flash	3.82	3.99	4.00	4.06	4.08	4.14	4.07	3.82	4.00	4.00
open-source Models
DeepSeek-V3	4.40	3.90	4.04	3.95	3.93	4.00	3.72	3.63	3.35	3.88
Gemma-3-27B-it	3.98	3.74	3.96	3.81	3.79	3.97	3.81	3.37	3.05	3.72
Qwen3-32B-thinking	4.14	3.88	3.95	4.02	4.00	3.96	3.87	2.37	1.94	3.57
Qwen2.5-72B-Instruct	4.26	3.76	3.85	3.80	3.93	4.16	3.64	2.19	1.83	3.49
LLaMA3.3-70B-Instruct	4.12	3.64	3.85	3.83	3.46	3.80	3.39	3.18	2.00	3.47
Specialized Models
HuatuoGPT-o1-72B	4.16	3.89	4.03	3.94	3.90	3.93	3.76	2.72	1.98	3.59
MedGemma-27B	3.95	3.86	4.00	3.90	3.86	4.01	3.72	3.44	2.96	3.74
MediPhi	3.89	3.29	3.49	3.36	3.04	3.09	2.14	1.83	1.70	2.87
LiveQA	Proprietary Models
GPT-5.4	4.19	4.04	4.39	4.56	4.52	4.68	4.56	4.45	4.46	4.43
Gemini-3-Flash	3.59	3.70	3.77	3.90	3.97	3.95	3.91	3.77	3.93	3.83
open-source Models

为什么重要

如果只看英语成绩来评价医疗AI,那么非英语使用者在实际就医场景中得到的服务质量可能大打折扣。这项研究也提醒开发者和使用者,未经人工审校的机器翻译评测基准得分不能完全当真。

本文术语

MCQA · 从固定选项中选出正确答案的选择题形式
NLI · 判断两句话之间是相互支持、矛盾还是无关的推理任务
LLM-as-judge · 用另一个大语言模型代替人来给答案质量打分的评测方式
低资源语言 · 网络文本和训练数据较少,因此AI模型训练得相对不充分的语言
机器翻译(MT) · 由软件自动完成、未经人工校对的翻译文本

论文原文摘要（英文）

We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.

作者 · Yingjian Chen (Drew), Fan Gao (Drew), Sherry T. Tong (Drew), Haoyu Zhang (Drew), Aosong Feng (Drew), Kevin W. Jin (Drew)

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Yingjian Chen (Drew) et al., arXiv:2608.19981, arxiv-nonexclusive

HealMed: Multilingual Evaluation of Large Language Models in Medicine

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道