The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline

arXiv:2608.249522026-08-27

AI语言模型对AAVE等非标准英语方言征收的隐性'方言税',不只出现在分词环节,而是贯穿训练与推理全流程

研究者检验了语言模型是否对AAVE(非裔美国人英语)等方言存在系统性不公平待遇,以及这种不公平究竟源自流程中的哪个环节。通过用标准美国英语(SAE)和方言写成、意思完全相同的配对文本进行测试,他们发现模型确实认为两者语义等价,但在分词、预训练、后训练奖励模型和推理这四个环节,方言文本始终被处理得更差、学习得更差、评分也更差。即便强制模型逐字符而非按常规分词方式读取文本,输入端的差距有所缩小,但准确率和输出端的差距依然存在,说明偏差已经固化在模型学到的参数里,而不仅仅是分词器的问题。

METAL LAB 解读图

方言税在语言模型流程各环节的累积过程

证据状态已报告实测结果

1. 语义核验嵌入模型确认SAE与方言(如AAVE)文本语义相同(相似度0.92至0.98以上),排除后续差距源于语义不同的可能
2. 分词环节BPE、Unigram、WordPiece等所有分词器都把AAVE、阿巴拉契亚、奇卡诺文本切分成比SAE更多的token(token生成率差距约+0.07),该排序七年未变
3. 字符级分词实验绕过子词分词后,推理准确率与输出熵的差距依然存在,提示偏差同样存在于模型已学到的参数中
4. 预训练梯度与损失配对的SAE-AAVE文本产生的训练梯度比完全无关的SAE文档对差异更大(平均z=-2.64),AAVE持续获得更高的预测损失(高0.47至0.66 nats)
5. 后训练奖励模型奖励模型的方言偏好随任务不稳定变化:孤立方言词打分高于SAE词,但完整语境下的AAVE回答常被扣分

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

语义核验:嵌入模型(EmbeddingGemma)显示,SAE与方言配对文本的语义相似度高于随机字符噪声或外语翻译对照组(MultiVALUE上超过0.98,ParallelAAVE上为0.92),证明后续发现的差距并非源于语义本身不同。
分词环节:在GPT、Gemma、Llama、Qwen、T5、BERT等模型使用的BPE、Unigram、WordPiece三种分词算法中,AAVE文本平均每个单词比对应SAE文本多需要约0.07个token(数值越高效率越低),且'阿巴拉契亚方言>AAVE>奇卡诺英语>SAE>印度英语>新加坡英语'这一不利程度排序在七年的分词器发展中基本保持不变。
字符级分词实验:绕过子词分词、改为逐字符输入后,模型对输入的'意外程度'差距有所缩小但并不一致,而实际推理准确率差距、输出随机性(熵)差距,以及仅凭模型内部状态判断方言的能力(常规分词下超过99%,字符级分词下仍超过86%)基本没有变化,说明偏差同样存在于模型已学到的参数中。
预训练环节:通过测量一条训练样本对模型参数的更新幅度(梯度)发现,语义相同的SAE-AAVE配对文本产生的梯度更新,反而比两篇完全无关的SAE文档之间的差异更大(九个模型平均z分数为-2.64);同时九个模型全部对AAVE输入给出更高的预测损失(高出0.47至0.66 nats,效应量很大),说明模型学习方言文本比学习无关的标准文本更吃力。
后训练奖励模型环节:来自Ai2、QRM、Skywork的十个奖励模型中有八个表现出统计显著的方向性方言偏好,且方向随任务翻转(算法、数学任务偏向SAE,逻辑、规划任务偏向AAVE);孤立的方言专属词(如'wanna'、'lil')打分反而高于SAE专属词(如'Calculate',平均高出约0.55分),但完整推理语境下的AAVE回答却常被扣分,且同一模型经过指令微调后偏好方向甚至发生反转。

Figure 1: Models understand semantic equivalence yet penalize surface form. We visualize the semantic equivalence of various text transformations on MultiVALUE. All dialect pairs achieve high similarities exceeding every perturbation and translation baseline.

Table 1: Dialects are linearly decodable from hidden states under both tokenizations. We list the five-fold cross-validation accuracy of logistic regression predicting dialect from the answer-step hidden state. Character tokenization slightly reduces separability, but all models remain far above chance (50%).
	Llama	Gemma	Qwen
	1B	3B	8B	1B	4B	12B	1.7B	4B	8B
Canonical (%)	99.0	99.9	99.4	99.8	99.8	100.0	99.9	100.0	100.0
Character (%)	94.5	98.3	99.6	86.4	99.5	99.5	98.5	97.3	97.5
Δ	−4.5	−1.6	+0.2	−13.4	−0.3	−0.5	−1.3	−2.7	−2.5

Figure 2: Modern tokenizers consistently exhibit dialectal biases. (2(a)) The mean fertility of the three tokenization algorithms on ParallelAAVE shows a statistically significant difference between SAE and AAVE dialects. (2(b)) The ratio of BPE token lengths on various dialects to that of SAE on MultiVALUE reveals a consistent dialectal tokenization performance gap that loosely parallels current income gaps of minority groups within the US (§D.1).

Table 2: Dialect gradient divergence sits within the range of character-level perturbations. Across nine base models and four ReDial tasks, we report the grand-mean paired cosine similarity (μ+) and z-score against the unrelated-SAE baseline. Higher μ+ indicates greater similarity to SAE gradients; lower values indicate stronger gradient divergence.
Paired condition with SAE	μ+(↑)	z
Baseline (unrelated SAE)	0.57	0.00
Capitalize (alternating)	0.29	−3.36
Capitalize (random)	0.34	−2.88
Drop (ℙ=0.15)	0.36	−2.58
Insert (ℙ=0.05)	0.47	−1.40
Swap (ℙ=0.05)	0.50	−1.06
Dialect (AAVE)	0.35	−2.64

Table 3: Source datasets used to reconstruct ReDial.
Category	Source	Items
Algorithm (26%)	HumanEval (Chen et al., 2021)	164
MBPP (Sanitized) (Austin et al., 2021)	149
Logic (30%)	FOLIO (Han et al., 2024; Wu et al., 2024)	162
LogicBench (Parmar et al., 2024)	200
Math (25%)	GSM8K (Cobbe et al., 2021)	150
SVAMP (Patel et al., 2021)	150
Planning (19%)	AsyncHow (Lin et al., 2024)	225
Total	-	1,200

Figure 3: Character tokenization does not systematically narrow dialect gaps. We plot per-model SAE − AAVE effect sizes under canonical (blue) and character-level (orange) tokenization, split by reasoning strategy (△=CoT, ○=naïve). Grey lines connect paired points for each model. Left: The accuracy gap does not systematically change under character tokenization. Center: The input entropy gap ΔH changes inconsistently, confirming that character tokenization fails to equalize how the model processes dialect inputs. Right: The output entropy gap ΔH persists under both tokenizations, suggesting generation-time dialect bias is independent of the tokenizer.

Table 4: Datasets for parallel SAE and AAVE texts.
Dataset	Source	Pairs	Dialects
ParallelAAVE	Groenwold et al. (2020)	2019	SAE, AAVE
MultiVALUE	Ziems* et al. (2023)	429	SAE, AAVE, Appalachian, Chicano, Indian, Singapore

Figure 4: Dialectal form outweighs semantic mismatch in gradient geometry. Violin plots of the gradient cosine similarity between paired and unrelated documents on ReDial reveal that matched SAE-AAVE pairs produce lower similarity than unrelated SAE-SAE pairs, implying that a meaning-preserving dialect shift can perturb the training signal more than changing the document content within SAE. In the plot, an LM name is indicated by its first-letter abbreviation and size.

Table 5: Tokenizer names used in our analysis.
Tokenization	Name	Model / Encoding
BPE	GPT-5	o200k_base (tiktoken)
GPT-2	openai-community/gpt2
Gemma	google/gemma-3-27b-it
Llama	meta-llama/Llama-3.3-70B-Instruct
Qwen	Qwen/Qwen3-32B
Unigram	T5	t5-small
WordPiece	BERT	bert-base-uncased

Figure 5: Dialectal text incurs higher prediction loss. We plot the mean cross-entropy loss by dialect. All nine models assign significantly higher loss to AAVE inputs than to their SAE counterparts (one-sided paired Wilcoxon test, n=1,200 per model, Bonferroni-corrected p<0.001), with per-model mean gaps of 0.47 to 0.66 nats (Cohen’s dz∈[2.12,2.55]). An LM name is indicated by its first-letter abbreviation and size.

Table 6: Language models used in our analysis.
Family	Name	Size	Type	Model ID
Llama	Llama 3.2	1B	Base	meta-llama/Llama-3.2-1B
Llama-3.2	1B	Instruct	meta-llama/Llama-3.2-1B-Instruct
Llama-3.2	3B	Base	meta-llama/Llama-3.2-3B
Llama-3.2	3B	Instruct	meta-llama/Llama-3.2-3B-Instruct
Llama-3.1	8B	Base	meta-llama/Llama-3.1-8B
Llama-3.1	8B	Instruct	meta-llama/Llama-3.1-8B-Instruct
	Llama-3.1	70B	Base	meta-llama/Llama-3.1-70B
	Llama-3.1	70B	Instruct	meta-llama/Llama-3.1-70B-Instruct
Gemma	Gemma 3	1B	Base	google/gemma-3-1b-pt
Gemma-3	1B	Instruct	google/gemma-3-1b-it
Gemma-3	4B	Base	google/gemma-3-4b-pt
Gemma-3	4B	Instruct	google/gemma-3-4b-it
Gemma-3	12B	Base	google/gemma-3-12b-pt
Gemma-3	12B	Instruct	google/gemma-3-12b-it
	Gemma-3	27B	Base	google/gemma-3-27b-pt
	Gemma-3	27B	Instruct	google/gemma-3-27b-it
Qwen	Qwen 3	1.7B	Base	Qwen/Qwen3-1.7B-Base
Qwen-3	1.7B	Instruct	Qwen/Qwen3-1.7B
Qwen-3	4B	Base	Qwen/Qwen3-4B-Base
Qwen-3	4B	Instruct	Qwen/Qwen3-4B
Qwen-3	8B	Base	Qwen/Qwen3-8B-Base
Qwen-3	8B	Instruct	Qwen/Qwen3-8B
	Qwen-3	32B	Instruct	Qwen/Qwen3-32B

Figure 6: Transformations result in similar hidden-state similarity curves. Layer-wise cosine similarities between SAE hidden states and those of transformed texts are pooled across MultiVALUE and ParallelAAVE. All text transformations follow similar trajectories across normalized LM layers, with dialectal transformations most similar to SAE.

Table 7: Reward models used in our analysis.
Provider	Base Model	Size	Model ID
Skywork	Llama-3.2 (Instruct)	3B	Skywork/Skywork-Reward-V2-Llama-3.2-3B
Llama-3.1 (Instruct)	8B	Skywork/Skywork-Reward-V2-Llama-3.1-8B
Qwen-3 (Instruct)	4B	Skywork/Skywork-Reward-V2-Qwen3-4B
Qwen-3 (Instruct)	8B	Skywork/Skywork-Reward-V2-Qwen3-8B
Gemma-2 (Instruct)	27B	Skywork/Skywork-Reward-Gemma-2-27B
QRM	Llama-3.1 (Instruct)	8B	nicolinho/QRM-Llama3.1-8B-v2
Gemma-2 (Instruct)	27B	nicolinho/QRM-Gemma-2-27B
Ai2	Llama-3.1 (Base)	8B	allenai/Llama-3.1-8B-Base-RM-RB2
Llama-3.1 (Instruct)	8B	allenai/Llama-3.1-8B-Instruct-RM-RB2
Llama-3.1 (Instruct)	70B	allenai/Llama-3.1-70B-Instruct-RM-RB2

Figure 7: Hidden-state similarity ／⇒ downstream accuracy. On the ReDial dataset, we plot each transformation’s final-layer hidden-state cosine similarity to SAE against its change in answer accuracy. Among semantic-meaning-preserving transformations, SAE-to-AAVE shows the highest similarity to the original SAE representation but the lowest downstream accuracy.

Table 8: Translation transformation details.
Resource Level	Language	Google Translate Code
High	Chinese	zh-CN
French	fr
Mid	Hindi	hi
Polish	pl
Low	Khmer	km
Yoruba	yo

Table 9: Paired per-token perplexity ratio vs SAE. We compare the perplexity ratios for character perturbations, translations, and dialects (italicized). Each row pairs the transformed text to its matched SAE text by unique_id within a model (ratio =exp⁡(CEcond−CEsae)), aggregated as the median of per-model medians across the models listed in Table 6. The mean fraction of paired items the model finds harder than SAE is indicated under “frac>1”, where a ratio above 1 means the model is more surprised by the surface form than by SAE despite preserved meaning.
	MultiVALUE	ParallelAAVE	ReDial
Condition	Ratio	frac>1	Ratio	frac>1	Ratio	frac>1
Drop (ℙ=0.15)	12.53	1.00	9.64	1.00	2.06	1.00
Singapore	4.92	1.00	–	–	–	–
Insert (ℙ=0.05)	4.41	1.00	3.74	0.98	1.58	1.00
Indian	3.99	1.00	–	–	–	–
Drop (ℙ=0.05)	3.52	1.00	3.06	0.98	1.43	1.00
AAVE	3.13	1.00	2.89	0.92	1.65	0.96
Swap (ℙ=0.05)	3.08	1.00	2.33	0.94	1.36	1.00
Capitalize (random)	2.36	1.00	1.46	0.73	1.41	0.99
Appalachian	2.10	1.00	–	–	–	–
Chicano	1.46	1.00	–	–	–	–
Translate (Yoruba)	1.38	0.75	0.68	0.36	1.26	0.70
Translate (Chinese)	1.26	0.80	1.03	0.50	1.08	0.74
Capitalize (alternating)	0.73	0.37	0.38	0.14	0.92	0.40
Translate (Polish)	0.71	0.16	0.37	0.10	0.96	0.39
Translate (French)	0.64	0.08	0.34	0.05	0.96	0.34
Translate (Hindi)	0.42	0.21	0.13	0.10	0.68	0.23
Translate (Khmer)	0.24	0.15	0.05	0.11	0.38	0.17

Figure 9: Models understand semantic equivalence yet penalize surface form. We find evidence of semantic invariance under surface-form transformations, as shown on the ParallelAAVE corpus. (9(a)) We visualize the semantic equivalence of various text transformations. The AAVE pairs achieve high cosine similarities, which exceed most perturbation and translation baselines. (9(b)) We plot semantic equivalence (cosine similarity at d=768) against the tokenization tax (Δfertility=𝔼⁡[fertilityAAVE]−𝔼⁡[fertilitySAE]) for each AAVE-SAE pair. Samples in the upper-right exhibit high semantic equivalence but higher tokenization cost, where meaning is preserved while a tax is imposed.

Table 10: Tokenization metrics used to measure bias.
Metric	What does it measure?
Average tokens per word	Average number of tokens corresponding with a single real word without punctuations
Average types per word	Average number of types corresponding with a single real word without punctuations
Character length	Number of characters in the string
Fertility (Ács, 2019)	Average number of tokens corresponding with a single real word
P(in vocabulary)	Proportion of words in the tokenizer vocabulary
Token length	Number of tokens in the string
Types length	Number of types in the string
Word length	Number of words in the string

Table 11: US median household income (USD).
Dialect	Income	Source
AAVE	$56,490	Guzman (2024)
Appalachian	$61,688	Srygley et al. (2024)
Chicano	$65,540	Guzman (2024)
Indian	$166,200	Lee and Batalova (2026)
SAE	$80,610	Guzman (2024)
Singaporean	$134,818	ZipAtlas (2025)

Figure 10: Ratio of dialect to SAE token lengths on MultiVALUE. Dotted lines indicate token-length parity with the paired SAE text. We see similar tokenization bias ranking on every measured metric for the six dialects in the dataset (Appalachian > AAVE > Chicano > SAE > Indian > Singapore). While some dialects (AAVE, Appalachian, Chicano) have increased token lengths compared to SAE, other dialects (Indian, Singapore) have decreased token lengths compared to SAE. We find that the same dialectal token bias remains present across all three tokenization strategies.

Table 13: Point-biserial correlation between paired SAE-AAVE gradient similarity si+ and a binary indicator for both-dialect correctness, by model. Per-model sample size is n=1,200, and pooled sample size is n=10,800. We denote significance by ∗∗∗p<0.001 and p∗⁣∗<0.01. Pooled r is negligible, but the per-model breakdown reveals heterogeneity.
Family	Size	r
Llama-3 Base	1B	−0.119∗∗∗
3B	−0.053
8B	−0.091∗⁣∗
Gemma-3 Base	1B	−0.108∗∗∗
4B	−0.105∗∗∗
12B	−0.066
Qwen-3 Base	1.7B	−0.132∗∗∗
4B	+0.098∗∗∗
8B	+0.192∗∗∗
Pooled	–	−0.013

Table 14: Per-RM dialect gap. Δr¯=𝔼⁡[r⁡(xSAE,y)−r⁡(xAAVE,y)] over ReDial sample-level pairs (n=1,200 per RM, i.e. 300 per task × 4 tasks), and one-sample t-statistic against 𝔼⁡[Δr]=0. Significance is denoted by ∗∗∗p<0.001, p∗⁣∗<0.01, ∗p<0.05.
Reward model	Δr¯	t
Skywork Llama 3B	−0.03	−0.67
Skywork Qwen 4B	+0.32	+7.54∗∗∗
Skywork Llama 8B	−0.22	−3.27∗⁣∗
Skywork Qwen 8B	+0.48	+14.42∗∗∗
Skywork Gemma 27B	−0.03	−0.31
QRM Llama 8B	+0.01	+4.72∗∗∗
QRM Gemma 27B	−0.07	−3.38∗∗∗
Ai2 Llama 8B Base	+0.41	+12.28∗∗∗
Ai2 Llama 8B	−0.19	−10.75∗∗∗
Ai2 Llama 70B	+0.03	+2.06∗

Table 15: Per-task dialect gap, pooled across RMs. One-sample t-test of 𝔼⁡[Δr]=0 within task. Significance is denoted by ∗∗∗p<0.001, p∗⁣∗<0.01.
Task	Δr¯	t
Algorithm	+0.43	+13.73∗∗∗
Math	+0.04	+1.18
Logic	−0.08	−2.92∗⁣∗
Planning	−0.14	−4.92∗∗∗

Table 16: Per-corpus token-level dialect gap. For each corpus, we identify subword tokens that appear exclusively in tokenized SAE vs. dialect text within each (RM, tokenizer) pairing, and score them under a fixed prompt. r¯SAE and r¯dial are the mean reward scores over the dialect-exclusive vocabularies. r¯SAE−r¯dial is the raw gap. d¯RM is Cohen’s d computed within each RM (using that RM’s pooled score standard deviation) and then averaged across the ten RMs. This normalizes for the substantial cross-RM scale differences (per-RM σ∈[0.06,3.97]). The bottom row pools all three corpora together. Independent two-sample t-tests, with significance denoted by ∗∗∗p<0.001.
Corpus	Dialects	nSAE	ndialect	r¯SAE	r¯dialect	r¯SAE−r¯dialect	d¯RM	p
ReDial	{AAVE}	121,780	163,120	−4.47	−4.10	−0.37∗∗∗	−0.17	9×10−31
ParallelAAVE	{AAVE}	91,380	163,420	−4.42	−3.83	−0.58∗∗∗	−0.29	1×10−67
MultiVALUE	{AAVE, Appal., Chic., Indian, Sing.}	141,930	132,130	−5.17	−4.74	−0.43∗∗∗	−0.24	9×10−39
Pooled (all)		355,090	458,670	−4.74	−4.19	−0.55∗∗∗	−0.27	7×10−185

Table 17: Per-RM per-dialect token-level dialect gap. Each cell shows the raw reward gap r¯SAE−r¯dial with Cohen’s d in parentheses. Negative values indicate the RM scores dialect-exclusive tokens higher than SAE-exclusive ones. AAVE-exclusive tokens come from ReDial, ParallelAAVE, and MultiVALUE, and other dialect-exclusive tokens come from MultiVALUE. Cohen’s d uses each RM’s pooled score standard deviation as denominator, so raw gaps are not directly comparable across RMs while d values are. The bolded cell is the only one flipping the dialect-favoring pattern. The “Pooled” column aggregates a given RM across all five dialects, and the “Mean across RMs” row is the unweighted average across the ten RMs. We run independent two-sample t-tests and denote the significance: ∗∗∗p<0.001, p∗⁣∗<0.01, ∗p<0.05.
Reward model	AAVE	Appalachian	Chicano	Indian	Singaporean	Pooled
Skywork Llama 3B	−0.45∗∗∗ (−0.25)	+0.11∗ (+0.06)	−0.71∗∗∗ (−0.43)	−0.06 (−0.03)	−0.88∗∗∗ (−0.49)	−0.51∗∗∗ (−0.28)
Skywork Qwen 4B	−0.69∗∗∗ (−0.31)	−0.32∗∗∗ (−0.15)	−0.71∗∗∗ (−0.37)	−0.13∗ (−0.06)	−1.10∗∗∗ (−0.47)	−0.78∗∗∗ (−0.34)
Skywork Llama 8B	−1.26∗∗∗ (−0.38)	−0.09 (−0.03)	−1.49∗∗∗ (−0.49)	−0.28∗⁣∗ (−0.08)	−1.34∗∗∗ (−0.38)	−1.26∗∗∗ (−0.37)
Skywork Qwen 8B	−0.80∗∗∗ (−0.40)	−0.16∗⁣∗ (−0.09)	−0.67∗∗∗ (−0.35)	+0.00 (+0.00)	−0.61∗∗∗ (−0.29)	−0.73∗∗∗ (−0.36)
Skywork Gemma 27B	−1.04∗∗∗ (−0.27)	−1.74∗∗∗ (−0.45)	−1.97∗∗∗ (−0.54)	+2.15∗∗∗ (+0.57)	−2.08∗∗∗ (−0.52)	−1.17∗∗∗ (−0.30)
QRM Llama 8B	+0.00∗∗∗ (+0.04)	−0.00 (−0.01)	−0.02∗∗∗ (−0.37)	−0.00 (−0.05)	−0.01∗∗∗ (−0.19)	−0.00∗ (−0.02)
QRM Gemma 27B	−0.15∗∗∗ (−0.30)	−0.18∗∗∗ (−0.37)	−0.23∗∗∗ (−0.46)	−0.12∗∗∗ (−0.24)	−0.17∗∗∗ (−0.34)	−0.16∗∗∗ (−0.33)
Ai2 Llama 8B Base	−0.42∗∗∗ (−0.26)	−0.39∗∗∗ (−0.25)	−1.05∗∗∗ (−0.74)	−0.38∗∗∗ (−0.22)	−1.19∗∗∗ (−0.71)	−0.55∗∗∗ (−0.34)
Ai2 Llama 8B	−0.09∗∗∗ (−0.12)	−0.07∗∗∗ (−0.10)	−0.28∗∗∗ (−0.47)	−0.07∗∗∗ (−0.09)	−0.30∗∗∗ (−0.43)	−0.13∗∗∗ (−0.18)
Ai2 Llama 70B	−0.16∗∗∗ (−0.15)	−0.02 (−0.03)	−0.13∗⁣∗ (−0.17)	−0.05∗ (−0.05)	−0.22∗∗∗ (−0.22)	−0.18∗∗∗ (−0.17)
Mean across RMs	−0.50 (−0.24)	−0.29 (−0.14)	−0.73 (−0.44)	+0.11 (−0.03)	−0.79 (−0.40)	−0.55 (−0.27)

Table 18: Per-dialect token-level gap, before and after within-RM standardization. Raw scores: pooled two-sample t-test on raw reward scores. Note that per-RM output scales differ, so the test is heavily influenced by the high-σ Skywork models. Within-RM standardized: each RM’s scores are z-scored using its own mean and standard deviation, then pooled across RMs before the t-test. The d column reports the gap in standard deviations, which is mathematically equivalent to the unweighted mean Cohen’s d across the ten RMs (within-RM effect size).
			Raw scores	Within-RM standardized
Dialect	nSAE	ndial	r¯SAE−r¯dial	p	d	p
AAVE	239,000	351,140	−0.50	6×10−115	−0.24	<10−300
Appalachian	24,980	31,540	−0.29	8×10−5	−0.13	5×10−64
Chicano	8,960	6,640	−0.73	3×10−7	−0.39	2×10−165
Indian	27,200	32,710	+0.11	0.14	−0.03	4×10−4
Singaporean	54,950	36,640	−0.79	6×10−40	−0.40	<10−300
All pooled	355,090	458,670	−0.55	7×10−185	−0.27	<10−300

研究结果

在MultiVALUE数据集上,五种方言配对的嵌入相似度均超过0.98(字符扰动/翻译对照组为0.659至0.978);在ParallelAAVE上,SAE-AAVE相似度为0.92,高于大多数字符扰动/翻译对照组,证实方言配对文本的语义确实保持一致。
在BPE、Unigram、WordPiece三种分词算法中,AAVE文本平均每个单词比配对SAE文本多需要约0.07个token,且'阿巴拉契亚>AAVE>奇卡诺>SAE>印度>新加坡'这一不利排序在七代分词器的发展历程中始终保持稳定。
字符级分词使输入熵差距的缩小并不一致,九个模型的推理准确率差距和输出熵差距基本没有变化;仅凭模型内部状态判断方言的分类准确率在常规分词下超过99%,在字符级分词下仍超过86%。
在九个基础模型上,语义相同的SAE-AAVE配对文档产生的训练梯度比两篇完全无关的SAE文档之间的梯度差异更大(平均z分数为-2.64),且全部九个模型都对AAVE输入给出更高的预测损失(高出0.47至0.66 nats,科恩dz在2.12至2.55之间)。
十个奖励模型中有八个表现出统计显著的方向性方言偏好,且该方向随任务不同而变化(算法、数学任务偏向SAE,逻辑、规划任务偏向AAVE)、也随训练阶段变化(某模型经指令微调后偏好方向发生反转);孤立打分时方言专属词平均比SAE专属词高约0.55分,但完整语境下的AAVE回答却常被扣分。

可应用场景

在面向方言多样的真实用户群体部署聊天机器人、搜索、教育工具等产品前,可参考本研究评估公平性,而不要假设仅修复分词器就足够
研究结果表明分词修正并不充分,可据此设计明确考虑方言差异的预训练或微调方案
可用于审计RLHF流程中使用的奖励模型,发现其偏好并非简单地'一律打压方言token',而是随语境和任务变化的复杂模式
可作为记录AAVE、阿巴拉契亚英语、奇卡诺英语使用者面对现有语言技术已知局限性的参考材料

局限与待验证事项

本研究仅考察英语内部的方言差异(SAE与AAVE、阿巴拉契亚英语、奇卡诺英语、印度英语、新加坡英语的对比),结论未必能推广到其他语言的方言或不同社会语言学背景
仅发现奖励模型的偏好随任务、模型不稳定,并未提出或验证具体的纠正方法
字符级分词实验只分离了推理阶段的分词器效应,并未从零开始用不同分词方式重新训练模型,因此无法在架构层面完全区分分词器原因与训练原因
分词差距与美国少数群体收入差距之间的对应关系(图2b、图11)只是一种松散的类比关系,并非因果论证,研究也未测试缩小这种类比差距的干预措施
测试的模型和奖励模型仅限于特定系列和规模(如Llama-3、Gemma-3、Qwen-3、GPT-5/5.5,以及Ai2、QRM、Skywork的奖励模型),其他架构或未测试的规模上结果可能有所不同

为什么重要

这说明仅仅修复最常被指责的分词器无法弥合AAVE等方言使用者遭遇的性能差距,因为偏差分散在整个流程中,而非集中在某一个可单独修复的环节。任何面向语言多样化真实用户开发或评估语言技术的人,都需要把方言公平性当作贯穿整个流程的问题来对待,而不只是优化分词器设计。

本文术语

AAVE(非裔美国人英语) · 常与美国黑人使用者相关联的英语方言,在本研究中被作为与SAE相对的非标准变体来考察
SAE(标准美国英语) · 在制度上受到优待的标准英语变体,本研究以其作为比较基准
分词(tokenization)/token生成率(fertility) · 将文本切分为模型可处理的子词单元的过程;token生成率衡量每个单词平均需要多少个token,数值越高说明处理效率越低、成本越高
梯度(gradient)/梯度发散 · 一条训练样本在模型内部产生的参数更新信号;两段文本的梯度差异越大,说明模型在学习层面把它们当作差异越大的内容对待
奖励模型(reward model, RM) · 用于给回答质量打分、从而指导语言模型后训练(如基于强化学习的对齐)的模型

论文原文摘要（英文）

Systematic dialectal performance gaps in language models (LMs) are well documented, but the source of these disparities within the modern language modeling pipeline remains unclear. Our study traces this "dialect tax" across the natural language processing pipeline. Using parallel English dialect corpora that hold meaning fixed while varying surface form, we first confirm that LMs recognize matched Standard American English (SAE) and dialectal texts as semantically equivalent. However, we discover further representational gaps corresponding to downstream performance gaps. Across model families and generations, modern LMs still encode dialectal texts unequally during tokenization, pre-training, post-training, and inference. Strikingly, bypassing traditional subword segmentation via a character-level counterfactual tokenizer removes neither input and output asymmetries nor dialectal accuracy gaps. During pre-training, dialect pairs induce more divergent gradient updates than pairs of entirely unrelated SAE documents, indicating that models find semantically equivalent dialectal content harder to learn from than unrelated SAE documents. During post-training, reward models show contextual, unstable dialect preferences, assigning higher values to isolated AAVE-exclusive tokens than to SAE-exclusive tokens, while full reasoning contexts receive task- and model-dependent dialect penalties. Overall, our findings suggest that the dialect tax is encoded and accumulated not by any one step in isolation, but at every step of the language modeling process.

作者 · Elle

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Elle et al., arXiv:2608.24952, CC BY 4.0