The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline

arXiv:2608.249522026-08-27

AI language models still charge a hidden 'dialect tax' on AAVE and other non-standard English at every stage, not just tokenization

Researchers tested whether language models treat dialects like AAVE (African American Vernacular English) unfairly, and if so, where the unfairness comes from. Using paired texts that mean the same thing in Standard American English (SAE) and in dialects, they found models understand the meanings as equivalent but still process, learn from, and reward dialect text worse than SAE, at every pipeline stage: tokenization, pre-training, post-training reward models, and inference. Even forcing models to read text character-by-character instead of using their normal tokenizer removed the input-side gap but not the accuracy or output gap, showing the bias is baked into learned model weights, not just the tokenizer.

METAL LAB explanatory visual

Where the Dialect Tax Accumulates Across the LM Pipeline

Evidence statusMeasured results reported

1. Semantic checkEmbedding model confirms SAE and dialect texts (e.g., AAVE) mean the same thing (similarity above 0.92-0.98), ruling out meaning mismatch as the cause of later gaps
2. TokenizationBPE, Unigram, WordPiece tokenizers all split AAVE/Appalachian/Chicano text into more tokens per word than SAE (fertility gap of about +0.07), a bias stable across 7 years of tokenizer development
3. Character-tokenizer testBypassing subword tokenization narrows some input-side gaps but leaves reasoning accuracy and output entropy gaps unchanged, showing bias also lives in learned model weights
4. Pre-training gradients & lossPaired SAE-AAVE texts produce more divergent training gradients (mean z=-2.64) than unrelated SAE documents, and AAVE gets consistently higher prediction loss (0.47-0.66 nats)
5. Post-training reward modelsReward models show unstable, task-dependent dialect preferences: isolated dialect words score higher than SAE words, but full AAVE reasoning contexts often get penalized

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

Semantic check: An embedding model (EmbeddingGemma) rated SAE-dialect pairs as more similar in meaning (cosine similarity above 0.98 on MultiVALUE, 0.92 on ParallelAAVE) than random character-noise or foreign-language translation controls, confirming the dialect gaps found later aren't just due to different meaning.
Tokenization: Across BPE, Unigram, and WordPiece tokenizers used in models like GPT, Gemma, Llama, Qwen, T5, and BERT, AAVE text needed about 0.07 more tokens per word on average than matched SAE text (higher 'fertility' is worse), and this ranking (Appalachian > AAVE > Chicano > SAE > Indian > Singapore) stayed stable across seven years of tokenizer development.
Character-level tokenizer test: Bypassing subword tokenization to feed models character-by-character narrowed differences in how surprised the model was by input text, but left the actual reasoning-accuracy gap, output randomness gap, and the model's ability to detect dialect from its internal states (above 86% classification accuracy, versus above 99% with normal tokenization) all largely unchanged, meaning the bias also lives inside the model's trained parameters.
Pre-training: When measuring how much a training example shifts a model's internal parameters (gradient), matched SAE-AAVE pairs produced more different updates than two completely unrelated SAE documents (mean z-score of -2.64 across 9 models), and AAVE also got consistently higher prediction loss than SAE (0.47 to 0.66 nats higher, a large effect size), showing models find same-meaning dialect text harder to learn from than unrelated standard text.
Post-training reward models: Ten reward models (from Ai2, QRM, Skywork) showed unstable dialect preferences depending on task -favoring SAE on Algorithm and Math tasks but favoring AAVE on Logic and Planning tasks - and while isolated dialect-only words like 'wanna' or 'lil' actually scored higher than SAE-only words like 'Calculate' (about 0.55 points higher on average), full reasoning-context responses in AAVE got penalized in a way that flipped even after instruction-tuning a single model.

Figure 1: Models understand semantic equivalence yet penalize surface form. We visualize the semantic equivalence of various text transformations on MultiVALUE. All dialect pairs achieve high similarities exceeding every perturbation and translation baseline.

Table 1: Dialects are linearly decodable from hidden states under both tokenizations. We list the five-fold cross-validation accuracy of logistic regression predicting dialect from the answer-step hidden state. Character tokenization slightly reduces separability, but all models remain far above chance (50%).
	Llama	Gemma	Qwen
	1B	3B	8B	1B	4B	12B	1.7B	4B	8B
Canonical (%)	99.0	99.9	99.4	99.8	99.8	100.0	99.9	100.0	100.0
Character (%)	94.5	98.3	99.6	86.4	99.5	99.5	98.5	97.3	97.5
Δ	−4.5	−1.6	+0.2	−13.4	−0.3	−0.5	−1.3	−2.7	−2.5

Figure 2: Modern tokenizers consistently exhibit dialectal biases. (2(a)) The mean fertility of the three tokenization algorithms on ParallelAAVE shows a statistically significant difference between SAE and AAVE dialects. (2(b)) The ratio of BPE token lengths on various dialects to that of SAE on MultiVALUE reveals a consistent dialectal tokenization performance gap that loosely parallels current income gaps of minority groups within the US (§D.1).

Table 2: Dialect gradient divergence sits within the range of character-level perturbations. Across nine base models and four ReDial tasks, we report the grand-mean paired cosine similarity (μ+) and z-score against the unrelated-SAE baseline. Higher μ+ indicates greater similarity to SAE gradients; lower values indicate stronger gradient divergence.
Paired condition with SAE	μ+(↑)	z
Baseline (unrelated SAE)	0.57	0.00
Capitalize (alternating)	0.29	−3.36
Capitalize (random)	0.34	−2.88
Drop (ℙ=0.15)	0.36	−2.58
Insert (ℙ=0.05)	0.47	−1.40
Swap (ℙ=0.05)	0.50	−1.06
Dialect (AAVE)	0.35	−2.64

Table 3: Source datasets used to reconstruct ReDial.
Category	Source	Items
Algorithm (26%)	HumanEval (Chen et al., 2021)	164
MBPP (Sanitized) (Austin et al., 2021)	149
Logic (30%)	FOLIO (Han et al., 2024; Wu et al., 2024)	162
LogicBench (Parmar et al., 2024)	200
Math (25%)	GSM8K (Cobbe et al., 2021)	150
SVAMP (Patel et al., 2021)	150
Planning (19%)	AsyncHow (Lin et al., 2024)	225
Total	-	1,200

Figure 3: Character tokenization does not systematically narrow dialect gaps. We plot per-model SAE − AAVE effect sizes under canonical (blue) and character-level (orange) tokenization, split by reasoning strategy (△=CoT, ○=naïve). Grey lines connect paired points for each model. Left: The accuracy gap does not systematically change under character tokenization. Center: The input entropy gap ΔH changes inconsistently, confirming that character tokenization fails to equalize how the model processes dialect inputs. Right: The output entropy gap ΔH persists under both tokenizations, suggesting generation-time dialect bias is independent of the tokenizer.

Table 4: Datasets for parallel SAE and AAVE texts.
Dataset	Source	Pairs	Dialects
ParallelAAVE	Groenwold et al. (2020)	2019	SAE, AAVE
MultiVALUE	Ziems* et al. (2023)	429	SAE, AAVE, Appalachian, Chicano, Indian, Singapore

Figure 4: Dialectal form outweighs semantic mismatch in gradient geometry. Violin plots of the gradient cosine similarity between paired and unrelated documents on ReDial reveal that matched SAE-AAVE pairs produce lower similarity than unrelated SAE-SAE pairs, implying that a meaning-preserving dialect shift can perturb the training signal more than changing the document content within SAE. In the plot, an LM name is indicated by its first-letter abbreviation and size.

Table 5: Tokenizer names used in our analysis.
Tokenization	Name	Model / Encoding
BPE	GPT-5	o200k_base (tiktoken)
GPT-2	openai-community/gpt2
Gemma	google/gemma-3-27b-it
Llama	meta-llama/Llama-3.3-70B-Instruct
Qwen	Qwen/Qwen3-32B
Unigram	T5	t5-small
WordPiece	BERT	bert-base-uncased

Figure 5: Dialectal text incurs higher prediction loss. We plot the mean cross-entropy loss by dialect. All nine models assign significantly higher loss to AAVE inputs than to their SAE counterparts (one-sided paired Wilcoxon test, n=1,200 per model, Bonferroni-corrected p<0.001), with per-model mean gaps of 0.47 to 0.66 nats (Cohen’s dz∈[2.12,2.55]). An LM name is indicated by its first-letter abbreviation and size.

Table 6: Language models used in our analysis.
Family	Name	Size	Type	Model ID
Llama	Llama 3.2	1B	Base	meta-llama/Llama-3.2-1B
Llama-3.2	1B	Instruct	meta-llama/Llama-3.2-1B-Instruct
Llama-3.2	3B	Base	meta-llama/Llama-3.2-3B
Llama-3.2	3B	Instruct	meta-llama/Llama-3.2-3B-Instruct
Llama-3.1	8B	Base	meta-llama/Llama-3.1-8B
Llama-3.1	8B	Instruct	meta-llama/Llama-3.1-8B-Instruct
	Llama-3.1	70B	Base	meta-llama/Llama-3.1-70B
	Llama-3.1	70B	Instruct	meta-llama/Llama-3.1-70B-Instruct
Gemma	Gemma 3	1B	Base	google/gemma-3-1b-pt
Gemma-3	1B	Instruct	google/gemma-3-1b-it
Gemma-3	4B	Base	google/gemma-3-4b-pt
Gemma-3	4B	Instruct	google/gemma-3-4b-it
Gemma-3	12B	Base	google/gemma-3-12b-pt
Gemma-3	12B	Instruct	google/gemma-3-12b-it
	Gemma-3	27B	Base	google/gemma-3-27b-pt
	Gemma-3	27B	Instruct	google/gemma-3-27b-it
Qwen	Qwen 3	1.7B	Base	Qwen/Qwen3-1.7B-Base
Qwen-3	1.7B	Instruct	Qwen/Qwen3-1.7B
Qwen-3	4B	Base	Qwen/Qwen3-4B-Base
Qwen-3	4B	Instruct	Qwen/Qwen3-4B
Qwen-3	8B	Base	Qwen/Qwen3-8B-Base
Qwen-3	8B	Instruct	Qwen/Qwen3-8B
	Qwen-3	32B	Instruct	Qwen/Qwen3-32B

Figure 6: Transformations result in similar hidden-state similarity curves. Layer-wise cosine similarities between SAE hidden states and those of transformed texts are pooled across MultiVALUE and ParallelAAVE. All text transformations follow similar trajectories across normalized LM layers, with dialectal transformations most similar to SAE.

Table 7: Reward models used in our analysis.
Provider	Base Model	Size	Model ID
Skywork	Llama-3.2 (Instruct)	3B	Skywork/Skywork-Reward-V2-Llama-3.2-3B
Llama-3.1 (Instruct)	8B	Skywork/Skywork-Reward-V2-Llama-3.1-8B
Qwen-3 (Instruct)	4B	Skywork/Skywork-Reward-V2-Qwen3-4B
Qwen-3 (Instruct)	8B	Skywork/Skywork-Reward-V2-Qwen3-8B
Gemma-2 (Instruct)	27B	Skywork/Skywork-Reward-Gemma-2-27B
QRM	Llama-3.1 (Instruct)	8B	nicolinho/QRM-Llama3.1-8B-v2
Gemma-2 (Instruct)	27B	nicolinho/QRM-Gemma-2-27B
Ai2	Llama-3.1 (Base)	8B	allenai/Llama-3.1-8B-Base-RM-RB2
Llama-3.1 (Instruct)	8B	allenai/Llama-3.1-8B-Instruct-RM-RB2
Llama-3.1 (Instruct)	70B	allenai/Llama-3.1-70B-Instruct-RM-RB2

Figure 7: Hidden-state similarity ／⇒ downstream accuracy. On the ReDial dataset, we plot each transformation’s final-layer hidden-state cosine similarity to SAE against its change in answer accuracy. Among semantic-meaning-preserving transformations, SAE-to-AAVE shows the highest similarity to the original SAE representation but the lowest downstream accuracy.

Table 8: Translation transformation details.
Resource Level	Language	Google Translate Code
High	Chinese	zh-CN
French	fr
Mid	Hindi	hi
Polish	pl
Low	Khmer	km
Yoruba	yo

Table 9: Paired per-token perplexity ratio vs SAE. We compare the perplexity ratios for character perturbations, translations, and dialects (italicized). Each row pairs the transformed text to its matched SAE text by unique_id within a model (ratio =exp⁡(CEcond−CEsae)), aggregated as the median of per-model medians across the models listed in Table 6. The mean fraction of paired items the model finds harder than SAE is indicated under “frac>1”, where a ratio above 1 means the model is more surprised by the surface form than by SAE despite preserved meaning.
	MultiVALUE	ParallelAAVE	ReDial
Condition	Ratio	frac>1	Ratio	frac>1	Ratio	frac>1
Drop (ℙ=0.15)	12.53	1.00	9.64	1.00	2.06	1.00
Singapore	4.92	1.00	–	–	–	–
Insert (ℙ=0.05)	4.41	1.00	3.74	0.98	1.58	1.00
Indian	3.99	1.00	–	–	–	–
Drop (ℙ=0.05)	3.52	1.00	3.06	0.98	1.43	1.00
AAVE	3.13	1.00	2.89	0.92	1.65	0.96
Swap (ℙ=0.05)	3.08	1.00	2.33	0.94	1.36	1.00
Capitalize (random)	2.36	1.00	1.46	0.73	1.41	0.99
Appalachian	2.10	1.00	–	–	–	–
Chicano	1.46	1.00	–	–	–	–
Translate (Yoruba)	1.38	0.75	0.68	0.36	1.26	0.70
Translate (Chinese)	1.26	0.80	1.03	0.50	1.08	0.74
Capitalize (alternating)	0.73	0.37	0.38	0.14	0.92	0.40
Translate (Polish)	0.71	0.16	0.37	0.10	0.96	0.39
Translate (French)	0.64	0.08	0.34	0.05	0.96	0.34
Translate (Hindi)	0.42	0.21	0.13	0.10	0.68	0.23
Translate (Khmer)	0.24	0.15	0.05	0.11	0.38	0.17

Figure 9: Models understand semantic equivalence yet penalize surface form. We find evidence of semantic invariance under surface-form transformations, as shown on the ParallelAAVE corpus. (9(a)) We visualize the semantic equivalence of various text transformations. The AAVE pairs achieve high cosine similarities, which exceed most perturbation and translation baselines. (9(b)) We plot semantic equivalence (cosine similarity at d=768) against the tokenization tax (Δfertility=𝔼⁡[fertilityAAVE]−𝔼⁡[fertilitySAE]) for each AAVE-SAE pair. Samples in the upper-right exhibit high semantic equivalence but higher tokenization cost, where meaning is preserved while a tax is imposed.

Table 10: Tokenization metrics used to measure bias.
Metric	What does it measure?
Average tokens per word	Average number of tokens corresponding with a single real word without punctuations
Average types per word	Average number of types corresponding with a single real word without punctuations
Character length	Number of characters in the string
Fertility (Ács, 2019)	Average number of tokens corresponding with a single real word
P(in vocabulary)	Proportion of words in the tokenizer vocabulary
Token length	Number of tokens in the string
Types length	Number of types in the string
Word length	Number of words in the string

Table 11: US median household income (USD).
Dialect	Income	Source
AAVE	$56,490	Guzman (2024)
Appalachian	$61,688	Srygley et al. (2024)
Chicano	$65,540	Guzman (2024)
Indian	$166,200	Lee and Batalova (2026)
SAE	$80,610	Guzman (2024)
Singaporean	$134,818	ZipAtlas (2025)

Figure 10: Ratio of dialect to SAE token lengths on MultiVALUE. Dotted lines indicate token-length parity with the paired SAE text. We see similar tokenization bias ranking on every measured metric for the six dialects in the dataset (Appalachian > AAVE > Chicano > SAE > Indian > Singapore). While some dialects (AAVE, Appalachian, Chicano) have increased token lengths compared to SAE, other dialects (Indian, Singapore) have decreased token lengths compared to SAE. We find that the same dialectal token bias remains present across all three tokenization strategies.

Table 13: Point-biserial correlation between paired SAE-AAVE gradient similarity si+ and a binary indicator for both-dialect correctness, by model. Per-model sample size is n=1,200, and pooled sample size is n=10,800. We denote significance by ∗∗∗p<0.001 and p∗⁣∗<0.01. Pooled r is negligible, but the per-model breakdown reveals heterogeneity.
Family	Size	r
Llama-3 Base	1B	−0.119∗∗∗
3B	−0.053
8B	−0.091∗⁣∗
Gemma-3 Base	1B	−0.108∗∗∗
4B	−0.105∗∗∗
12B	−0.066
Qwen-3 Base	1.7B	−0.132∗∗∗
4B	+0.098∗∗∗
8B	+0.192∗∗∗
Pooled	–	−0.013

Table 14: Per-RM dialect gap. Δr¯=𝔼⁡[r⁡(xSAE,y)−r⁡(xAAVE,y)] over ReDial sample-level pairs (n=1,200 per RM, i.e. 300 per task × 4 tasks), and one-sample t-statistic against 𝔼⁡[Δr]=0. Significance is denoted by ∗∗∗p<0.001, p∗⁣∗<0.01, ∗p<0.05.
Reward model	Δr¯	t
Skywork Llama 3B	−0.03	−0.67
Skywork Qwen 4B	+0.32	+7.54∗∗∗
Skywork Llama 8B	−0.22	−3.27∗⁣∗
Skywork Qwen 8B	+0.48	+14.42∗∗∗
Skywork Gemma 27B	−0.03	−0.31
QRM Llama 8B	+0.01	+4.72∗∗∗
QRM Gemma 27B	−0.07	−3.38∗∗∗
Ai2 Llama 8B Base	+0.41	+12.28∗∗∗
Ai2 Llama 8B	−0.19	−10.75∗∗∗
Ai2 Llama 70B	+0.03	+2.06∗

Table 15: Per-task dialect gap, pooled across RMs. One-sample t-test of 𝔼⁡[Δr]=0 within task. Significance is denoted by ∗∗∗p<0.001, p∗⁣∗<0.01.
Task	Δr¯	t
Algorithm	+0.43	+13.73∗∗∗
Math	+0.04	+1.18
Logic	−0.08	−2.92∗⁣∗
Planning	−0.14	−4.92∗∗∗

Table 16: Per-corpus token-level dialect gap. For each corpus, we identify subword tokens that appear exclusively in tokenized SAE vs. dialect text within each (RM, tokenizer) pairing, and score them under a fixed prompt. r¯SAE and r¯dial are the mean reward scores over the dialect-exclusive vocabularies. r¯SAE−r¯dial is the raw gap. d¯RM is Cohen’s d computed within each RM (using that RM’s pooled score standard deviation) and then averaged across the ten RMs. This normalizes for the substantial cross-RM scale differences (per-RM σ∈[0.06,3.97]). The bottom row pools all three corpora together. Independent two-sample t-tests, with significance denoted by ∗∗∗p<0.001.
Corpus	Dialects	nSAE	ndialect	r¯SAE	r¯dialect	r¯SAE−r¯dialect	d¯RM	p
ReDial	{AAVE}	121,780	163,120	−4.47	−4.10	−0.37∗∗∗	−0.17	9×10−31
ParallelAAVE	{AAVE}	91,380	163,420	−4.42	−3.83	−0.58∗∗∗	−0.29	1×10−67
MultiVALUE	{AAVE, Appal., Chic., Indian, Sing.}	141,930	132,130	−5.17	−4.74	−0.43∗∗∗	−0.24	9×10−39
Pooled (all)		355,090	458,670	−4.74	−4.19	−0.55∗∗∗	−0.27	7×10−185

Table 17: Per-RM per-dialect token-level dialect gap. Each cell shows the raw reward gap r¯SAE−r¯dial with Cohen’s d in parentheses. Negative values indicate the RM scores dialect-exclusive tokens higher than SAE-exclusive ones. AAVE-exclusive tokens come from ReDial, ParallelAAVE, and MultiVALUE, and other dialect-exclusive tokens come from MultiVALUE. Cohen’s d uses each RM’s pooled score standard deviation as denominator, so raw gaps are not directly comparable across RMs while d values are. The bolded cell is the only one flipping the dialect-favoring pattern. The “Pooled” column aggregates a given RM across all five dialects, and the “Mean across RMs” row is the unweighted average across the ten RMs. We run independent two-sample t-tests and denote the significance: ∗∗∗p<0.001, p∗⁣∗<0.01, ∗p<0.05.
Reward model	AAVE	Appalachian	Chicano	Indian	Singaporean	Pooled
Skywork Llama 3B	−0.45∗∗∗ (−0.25)	+0.11∗ (+0.06)	−0.71∗∗∗ (−0.43)	−0.06 (−0.03)	−0.88∗∗∗ (−0.49)	−0.51∗∗∗ (−0.28)
Skywork Qwen 4B	−0.69∗∗∗ (−0.31)	−0.32∗∗∗ (−0.15)	−0.71∗∗∗ (−0.37)	−0.13∗ (−0.06)	−1.10∗∗∗ (−0.47)	−0.78∗∗∗ (−0.34)
Skywork Llama 8B	−1.26∗∗∗ (−0.38)	−0.09 (−0.03)	−1.49∗∗∗ (−0.49)	−0.28∗⁣∗ (−0.08)	−1.34∗∗∗ (−0.38)	−1.26∗∗∗ (−0.37)
Skywork Qwen 8B	−0.80∗∗∗ (−0.40)	−0.16∗⁣∗ (−0.09)	−0.67∗∗∗ (−0.35)	+0.00 (+0.00)	−0.61∗∗∗ (−0.29)	−0.73∗∗∗ (−0.36)
Skywork Gemma 27B	−1.04∗∗∗ (−0.27)	−1.74∗∗∗ (−0.45)	−1.97∗∗∗ (−0.54)	+2.15∗∗∗ (+0.57)	−2.08∗∗∗ (−0.52)	−1.17∗∗∗ (−0.30)
QRM Llama 8B	+0.00∗∗∗ (+0.04)	−0.00 (−0.01)	−0.02∗∗∗ (−0.37)	−0.00 (−0.05)	−0.01∗∗∗ (−0.19)	−0.00∗ (−0.02)
QRM Gemma 27B	−0.15∗∗∗ (−0.30)	−0.18∗∗∗ (−0.37)	−0.23∗∗∗ (−0.46)	−0.12∗∗∗ (−0.24)	−0.17∗∗∗ (−0.34)	−0.16∗∗∗ (−0.33)
Ai2 Llama 8B Base	−0.42∗∗∗ (−0.26)	−0.39∗∗∗ (−0.25)	−1.05∗∗∗ (−0.74)	−0.38∗∗∗ (−0.22)	−1.19∗∗∗ (−0.71)	−0.55∗∗∗ (−0.34)
Ai2 Llama 8B	−0.09∗∗∗ (−0.12)	−0.07∗∗∗ (−0.10)	−0.28∗∗∗ (−0.47)	−0.07∗∗∗ (−0.09)	−0.30∗∗∗ (−0.43)	−0.13∗∗∗ (−0.18)
Ai2 Llama 70B	−0.16∗∗∗ (−0.15)	−0.02 (−0.03)	−0.13∗⁣∗ (−0.17)	−0.05∗ (−0.05)	−0.22∗∗∗ (−0.22)	−0.18∗∗∗ (−0.17)
Mean across RMs	−0.50 (−0.24)	−0.29 (−0.14)	−0.73 (−0.44)	+0.11 (−0.03)	−0.79 (−0.40)	−0.55 (−0.27)

Table 18: Per-dialect token-level gap, before and after within-RM standardization. Raw scores: pooled two-sample t-test on raw reward scores. Note that per-RM output scales differ, so the test is heavily influenced by the high-σ Skywork models. Within-RM standardized: each RM’s scores are z-scored using its own mean and standard deviation, then pooled across RMs before the t-test. The d column reports the gap in standard deviations, which is mathematically equivalent to the unweighted mean Cohen’s d across the ten RMs (within-RM effect size).
			Raw scores	Within-RM standardized
Dialect	nSAE	ndial	r¯SAE−r¯dial	p	d	p
AAVE	239,000	351,140	−0.50	6×10−115	−0.24	<10−300
Appalachian	24,980	31,540	−0.29	8×10−5	−0.13	5×10−64
Chicano	8,960	6,640	−0.73	3×10−7	−0.39	2×10−165
Indian	27,200	32,710	+0.11	0.14	−0.03	4×10−4
Singaporean	54,950	36,640	−0.79	6×10−40	−0.40	<10−300
All pooled	355,090	458,670	−0.55	7×10−185	−0.27	<10−300

Findings

On MultiVALUE, all five dialect pairs showed embedding similarity above 0.98 (vs 0.659-0.978 for perturbation/translation controls); on ParallelAAVE, SAE-AAVE similarity was 0.92, exceeding most character/translation controls but slightly below character swap/insert controls, confirming meaning is preserved across dialect pairs.
AAVE texts had on average 0.07 more tokens per word than matched SAE texts across BPE, Unigram, and WordPiece tokenizers, with a stable bias ranking (Appalachian > AAVE > Chicano > SAE > Indian > Singapore) across seven tokenizer generations and years of development.
Character-level tokenization narrowed the input-entropy gap inconsistently and left the reasoning-accuracy gap, output-entropy gap unchanged across nine models; dialect remained decodable from hidden states with over 86% accuracy under character tokenization versus over 99% under normal tokenization.
Across nine base models, paired SAE-AAVE documents produced more divergent training gradients (mean z=-2.64) than pairs of unrelated SAE documents, and all nine models assigned higher cross-entropy loss to AAVE inputs (0.47-0.66 nats higher, Cohen's dz between 2.12 and 2.55).
Eight of ten reward models showed statistically significant directional dialect bias in sample-level scoring that flipped by task (SAE-favoring on Algorithm/Math, AAVE-favoring on Logic/Planning) and by training stage (flipped sign after instruction-tuning one model), while isolated dialect-exclusive tokens scored higher than SAE-exclusive tokens by about 0.55 points on average, even though full-context AAVE responses were often penalized.

Where it can be used

Evaluating language model deployments (chatbots, search, education tools) for fairness across dialect-speaking user populations before assuming tokenizer fixes alone will solve disparities
Designing new pre-training or fine-tuning recipes that explicitly account for dialectal variation, since the study suggests gradient-level and reward-model-level interventions are needed, not just tokenizer changes
Auditing reward models used in RLHF pipelines for context-dependent, task-dependent dialect bias rather than assuming a single fixed 'penalize dialect tokens' pattern
Informing policy or documentation about known limitations of language technologies for AAVE, Appalachian, and Chicano English speakers

Limits and open work

The study only examines English dialects (SAE vs AAVE, Appalachian, Chicano, Indian, Singaporean English), so findings may not generalize to non-English languages or dialects with different sociolinguistic contexts.
Reward model bias was found to be unstable and task/model-dependent, so no single corrective mechanism is proposed or validated for fixing RM-level dialect bias.
The character-tokenization intervention isolates inference-time tokenizer effects but does not test retraining a model from scratch with different tokenization, so it cannot fully separate tokenizer-caused versus training-caused bias at the architecture level.
The income-gap comparison (Figure 2b, Figure 11) is presented as a loose correspondence, not a causal claim, and the study does not test interventions to reduce this parallel.
Models and reward models tested are limited to specific families and sizes (e.g., Llama-3, Gemma-3, Qwen-3, GPT-5/5.5, and RMs from Ai2, QRM, Skywork); results may differ for other architectures or larger/smaller scales not tested.

Why it matters

This shows that fixing tokenizers alone-the most commonly blamed culprit-will not close the performance gap for speakers of AAVE and other dialects, because the bias is distributed across training and inference, not concentrated in one fixable component. Anyone building or evaluating language technology for real-world, linguistically diverse users needs to treat dialect fairness as a pipeline-wide problem requiring new training approaches, not just better tokenizer design.

Terms in this paper

AAVE (African American Vernacular English) · A widely used English dialect associated with Black American speakers, treated in this study as a non-standard variety compared with SAE
SAE (Standard American English) · The 'standard', institutionally favored English variety used as the comparison baseline in this study
Tokenization / fertility · The process of splitting text into subword units for a model; fertility measures how many tokens are needed per word, with more tokens meaning less efficient, costlier processing
Gradient / gradient divergence · The internal update signal a model computes from a training example; more divergent gradients between paired texts mean the model treats them as more different, learning-wise
Reward model (RM) · A model trained to score how good a response is, used to guide a language model's post-training alignment (e.g., via reinforcement learning)

Original abstract (English)

Systematic dialectal performance gaps in language models (LMs) are well documented, but the source of these disparities within the modern language modeling pipeline remains unclear. Our study traces this "dialect tax" across the natural language processing pipeline. Using parallel English dialect corpora that hold meaning fixed while varying surface form, we first confirm that LMs recognize matched Standard American English (SAE) and dialectal texts as semantically equivalent. However, we discover further representational gaps corresponding to downstream performance gaps. Across model families and generations, modern LMs still encode dialectal texts unequally during tokenization, pre-training, post-training, and inference. Strikingly, bypassing traditional subword segmentation via a character-level counterfactual tokenizer removes neither input and output asymmetries nor dialectal accuracy gaps. During pre-training, dialect pairs induce more divergent gradient updates than pairs of entirely unrelated SAE documents, indicating that models find semantically equivalent dialectal content harder to learn from than unrelated SAE documents. During post-training, reward models show contextual, unstable dialect preferences, assigning higher values to isolated AAVE-exclusive tokens than to SAE-exclusive tokens, while full reasoning contexts receive task- and model-dependent dialect penalties. Overall, our findings suggest that the dialect tax is encoded and accumulated not by any one step in isolation, but at every step of the language modeling process.

Authors · Elle

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Elle et al., arXiv:2608.24952, CC BY 4.0