Figure 1: Motivating results for LSC: the performance of Aya (with a more language-balanced pretraining dataset) versus Qwen (with a more English-dominant pretraining dataset) on multilingual HotPotQA (11). We see that although Aya performs more consistently across languages, Aya degrades in performance (see Qwen’s performance in English).
Table 1: HOTFIXR has the best in-distribution performance, the best out-of-distribution performance among the training-based baselines, and remains multilingually consistent. This table shows the cross-lingual consistency with respect to ID and OOD performance. Language spread is the standard deviation of performance across languages. Low spread indicates more consistent performance across languages. Note: the training-based baselines are SelectionGT, SelectionGEN, Filtered, Untrained, and DataEnvGym.
Performance
Language
Method
ID (↑)
OOD (↑)
Spread (↓)
Base
51.9
65.6
9.6
EngReason
50.8
65.4
8.8
SelectionGT
49.2
60.9
9.8
SelectionGEN
50.2
55.8
9.5
Filtered
48.9
58.2
10.6
Untrained
50.4
63.4
9.7
DataEnvGym
49.0
57.4
10.2
HOTFIXR (ours)
56.2
64.7
9.4
Figure 2: Intuition behind HOTFIXR. The “data space” represents the data samples generated by the question generator. Essentially, we want to hill climb through the data space to find samples that have high lingual deficit scores, as they will be most informative to the student model. By fine-tuning the question generation model to generate samples according to the student model’s lingual deficit, we can improve the information embedded within data, improving our models on downstream tasks.
Table 2: Average performance difference of HOTFIXR versus each baseline. The average is over nine settings: three students (Qwen 7B, Qwen 14B, Llama 8B) × three ID/OOD tasks. “Wins” counts the number of settings where HOTFIXR scores higher.
In-Distribution
Out-of-Distribution
Baseline
Δ
Wins
Δ
Wins
Base
+4.3
9/9
-0.9
3/9
EngReason
+5.4
9/9
-0.7
4/9
SelectionGT
+7.0
9/9
+3.8
9/9
SelectionGEN
+6.0
9/9
+8.9
8/9
Filtered
+7.3
9/9
+6.6
8/9
Untrained
+5.8
9/9
+1.4
7/9
DataEnvGym
+7.1
9/9
+7.3
9/9
Average
+6.2
9/9
+3.7
7/9
Figure 3: An illustration of how to train the question generation model in HOTFIXR, along with the reward design.
Table 3: OOD language generalization: performance differences relative to the Base model, averaged across all models and corresponding tasks. Averaging the difference in HOTFIXR and other methods, we can compute how much HOTFIXR avoids catastrophic forgetting. For example, averaged across all languages, HOTFIXR avoids catastrophic forgetting by 8.7% compared to DataEnvGym (10.1-1.4).
Method
De
Ja
Ru
Zh
Avg
SelectionGT
-3.3
-4.7
-9.5
-10.5
-7.0
SelectionGEN
-1.9
-4.0
-22.5
-21.5
-12.5
Filtered
-3.8
-4.8
-14.7
-16.3
-9.9
Untrained
-2.9
-2.0
-2.3
-5.3
-3.1
DataEnvGym
-6.8
-6.0
-12.2
-15.4
-10.1
HOTFIXR
-2.6
-0.7
-0.9
-1.3
-1.4
Figure 4: Performance of data curation methods in-distribution.
Table 4: Cross-lingual spread of student performance, averaged over three student models (Qwen2.5-7B, Qwen2.5-14B, Llama-3.1-8B) and three multilingual tasks (OPUS, MMMLU, mHotpotQA) evaluated with each task’s primary metric. Lower is more consistent for all dispersion columns; Bold marks the best value among data-generation methods. Range is the max − min performance across all languages. Trimmed Range is the best minus the 2nd worst language accuracy. Std is standard deviation of the performance. IQR is the Inter-Quartile Range (difference in the 25% and 75% quartile). CV is the coeffiecient of variation (the standard devation divided by the mean).
Method
Range
Trim. Range
Trim.
Range
Std
IQR
CV
Trim.
Range
Base
20.2
16.0
7.2
9.6
0.12
EngReason
20.7
15.4
7.2
8.8
0.12
SelectionGT
22.3
17.1
7.9
9.8
0.13
SelectionGEN
18.8
14.9
6.9
9.5
0.13
Filtered
22.6
17.1
8.0
10.6
0.14
Untrained
20.1
15.0
7.1
9.7
0.12
DataEnvGym
22.8
17.0
8.0
10.2
0.14
HOTFIXR
19.4
14.6
6.9
9.4
0.11
Figure 5: Performance of data curation methods for factual (MMMLU) queries, an out-of-distribution task.
Table 5: Reward-signal ablation. All rows include the format reward. “Nemo.” is Nemotron, “Trans.” is Translation, “Fact.” is Factual. Highlights the impact each reward has—there is more effect of the reward functions combined, than individually. LAI is the “Language Agnostic Incompetency” and LSI is the “Language Specific Incompetency”, as described in Section 3.
ID
OOD
Model
Config
Nemo.
Trans.
Fact.
RAG
Qwen 7B
Format
52.0
57.6
64.0
75.8
+ LAI
50.9
57.9
64.3
77.2
+ LSI
50.8
57.8
63.6
77.7
+ LAI + LSI
57.8
59.1
64.3
76.7
Llama 8B
Format
48.2
57.8
49.9
70.2
+ LAI
48.5
57.2
49.2
69.2
+ LSI
47.5
57.5
49.3
67.8
+ LAI + LSI
52.5
58.9
49.3
69.7
Qwen 14B
Format
49.3
58.0
71.5
79.8
+ LAI
50.9
58.6
66.3
76.0
+ LSI
48.8
57.2
62.3
76.8
+ LAI + LSI
58.3
58.2
67.9
78.3
Figure 6: Performance of data curation methods for RAG (multilingual HotPotQA) queries, an out-of-distribution task.
Table 6: Minimum and maximum standard deviation (across 3 runs) for the Nemotron task (STEM, MATH, and CHAT), per model and method. This explains the small error bars in Figure 8.
Qwen 7B
Qwen 14B
Llama 8B
Method
min
max
min
max
min
max
Base
0.30
1.50
0.10
0.40
0.20
2.00
EngReason
0.25
1.23
0.16
0.75
0.34
1.00
SelectionGT
0.20
3.40
0.10
1.00
0.40
2.20
SelectionGEN
0.30
3.10
0.00
0.90
0.10
2.20
Filtered
0.70
0.90
0.30
0.80
0.30
1.70
Untrained
0.40
1.90
0.00
1.50
0.30
0.70
DataEnvGym
0.40
1.20
0.10
0.70
0.20
1.70
HOTFIXR
0.10
0.90
0.20
0.40
0.05
0.50
Figure 7: Performance of data curation methods for translation (OPUS-100) queries, an out-of-distribution task.
Table 7: Minimum and maximum standard deviation (across 3 runs) for the Translation task (OPUS-100), per model and method. This explains the small error bars in Figure 10.
Qwen 7B
Qwen 14B
Llama 8B
Method
min
max
min
max
min
max
Base
0.43
0.85
0.41
1.55
0.58
1.56
EngReason
0.40
0.83
0.43
1.05
0.55
1.56
SelectionGT
0.10
1.40
0.20
1.00
0.40
1.60
SelectionGEN
0.00
0.80
0.00
1.50
0.10
1.50
Filtered
0.10
1.60
0.10
0.80
0.10
1.60
Untrained
0.20
0.50
0.00
0.70
0.30
1.70
DataEnvGym
0.00
1.10
0.20
0.70
0.20
0.80
HOTFIXR
0.00
0.90
0.00
0.80
0.20
1.10
Figure 8: Performance of data curation methods for Nemotron queries, per task.
Table 8: Minimum and maximum standard deviation (across 3 runs) for the Factual task (MMMLU), per model and method. This explains the small error bars in Figure 9.
Qwen 7B
Qwen 14B
Llama 8B
Method
min
max
min
max
min
max
Base
0.10
1.50
0.10
1.10
0.00
2.30
EngReason
0.09
1.27
0.28
1.09
0.33
1.73
SelectionGT
0.30
2.30
0.20
1.80
0.10
1.60
SelectionGEN
0.40
2.60
0.10
1.80
0.10
2.30
Filtered
0.10
1.30
0.20
1.20
0.00
1.20
Untrained
0.30
1.00
0.10
1.00
0.20
2.20
DataEnvGym
0.30
2.40
0.10
1.40
0.10
1.20
HOTFIXR
0.30
1.40
0.30
1.30
0.30
2.60
Figure 9: Performance of data curation methods for factual (MMMLU) queries, per language.
Table 9: Minimum and maximum standard deviation (across 3 runs) for the RAG task (mHotPotQA), per model and method. This explains the small error bars in Figure 11.
Qwen 7B
Qwen 14B
Llama 8B
Method
min
max
min
max
min
max
Base
0.10
0.90
0.10
0.80
0.80
2.50
EngReason
0.33
0.99
0.16
1.05
0.53
1.71
SelectionGT
0.30
1.20
0.10
0.80
0.30
1.80
SelectionGEN
0.80
1.40
0.30
1.50
0.00
2.50
Filtered
0.10
1.10
0.40
1.50
0.00
1.30
Untrained
0.20
1.10
0.30
1.10
0.10
1.30
DataEnvGym
0.20
1.40
0.30
0.60
0.30
1.20
HOTFIXR
0.00
0.60
0.20
1.50
0.00
1.40
Figure 10: Performance of data curation methods for translation (OPUS-100) queries, per language.
Table 10: Average performance per model, with the added Distillation method. Here, we are able to show that a lot of HOTFIXR’s empirical success comes from the targeted question generation, rather than having correct, distilled labels from a large model.
Method
Nemotron
Factual
Translation
RAG
Qwen2.5-7B-Instruct
Base
52.3
67.0
59.2
79.0
EngReason
51.2
68.2
59.0
79.3
SelectionGT
48.5
63.9
53.8
61.5
SelectionGEN
50.3
59.9
56.8
41.4
Filtered
48.6
61.7
56.6
66.5
mCOT
47.9
55.3
54.4
53.8
Untrained
51.2
59.2
57.1
72.3
DataEnvGym
49.3
46.2
56.9
47.5
Distillation (32B)
49.6
64.7
58.9
77.3
HOTFIXR
57.8
64.3
59.1
76.7
Qwen2.5-14B-Instruct
Base
55.2
73.7
59.4
79.9
EngReason
54.1
73.0
59.5
81.0
SelectionGT
51.4
67.3
54.2
73.8
SelectionGEN
52.1
70.3
57.4
42.8
Filtered
50.9
59.1
59.1
45.5
Untrained
52.1
71.0
57.9
77.5
DataEnvGym
50.3
62.6
57.4
70.4
Distillation (32B)
53.3
69.5
59.0
78.4
HOTFIXR
58.3
67.9
58.2
78.3
Llama-3.1-8B-Instruct
Base
48.1
48.5
56.3
67.7
EngReason
47.0
47.3
56.4
65.2
SelectionGT
47.6
48.8
57.6
67.2
SelectionGEN
48.1
48.6
57.6
67.2
Filtered
47.2
48.8
57.7
68.4
Untrained
47.7
49.5
57.1
68.6
DataEnvGym
47.6
48.8
57.5
69.1
Distillation (32B)
47.7
51.1
58.0
70.3
Figure 11: Performance of data curation methods for RAG (multilingual HotPotQA) queries, per language.
Figure 12: The delta in performance of student models trained with two rounds of HOTFIXR versus one round. The improvement with two rounds is much more than the improvement with one round.
本文术语
大语言模型(LLM) · 通过海量文本训练、能够回答问题和进行对话的AI模型
语言能力差异(LSC) · 模型对同一语义问题因提问语言不同而准确率不同的现象
学生模型 · 被HOTFIXR生成的数据训练、实际需要被改进的语言模型
GRPO · 一种强化学习算法,用于根据奖励信号优化出题模型
分布外(OOD)任务/语言 · 训练时没有见过的任务类型或语言,用来检验模型的泛化能力
论文原文摘要（英文）
Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. In other words, a language model outputs different (and potentially incorrect) responses to the same semantic query when prompted in different languages. Prior work attributes this to an internal misalignment of semantic representation across languages. Currently, there are two main approaches to address LSC in the literature: (1) routing all queries through English, improving performance, but limiting language expressivity to English; or (2) training on language-balanced data, equalizing model performance across languages, but reducing overall performance. In this work, we take a data centric perspective and introduce HOTFIXR: Hardness Optimized Training data For Improving X-Lingual Reasoning. It is a data generation framework that uses models to probe and learn a student model's multilingual weaknesses, and generates data to mitigate them. HOTFIXR can generate multilingual synthetic training data that can improve multilingual performance. We evaluate on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution languages. On average, HOTFIXR (1) improves in-distribution performance by 6.2%, (2) reduces catastrophic forgetting (induced by fine-tuning) on OOD tasks by 3.7%, and (3) on OOD languages by 7.1%. Overall, as many real-world applications requires multilingual LLMs, our work contributes to the efforts of making LLMs multilingually proficient. We will release code upon acceptance.
