The Asymmetric Harms of LLM Compression

arXiv:2608.196702026-08-21

给AI模型瘦身压缩后,平均分数看起来没变,但常见知识反而流失更多,而且偏见会悄悄在特定群体中反向放大

研究团队对Llama-3.1-8B-Instruct、Qwen-3-8B和Gemma-2-9B-it三个语言模型应用了11种压缩方法(量化与剪枝),细致分析了知识保留能力、模型自信程度和社会偏见的变化。结果发现,压缩会让原本广为人知的头部知识相对损失更多,模型在给出错误答案时往往依然保持较高自信,而整体偏见分数看似稳定,实际上不同人口群体之间可能出现方向相反的巨大偏见变化。这说明仅凭准确率或困惑度这类平均指标,无法判断压缩后的模型是否真的安全可靠。

他们做了什么

对三个开源模型Llama-3.1-8B-Instruct、Qwen-3-8B、Gemma-2-9B-it应用了11种压缩方法,包括量化方法GPTQ、AWQ、OmniQuant、AQLM,以及剪枝方法WANDA、SparseGPT、ShortGPT等
利用按知识流行度分为头部、中部、尾部的PopQA和Head-to-Tail数据集测试发现,压缩后头部知识虽然绝对准确率仍最高,但相对于模型自身基准表现而言,头部知识的保留比例反而比尾部知识损失更严重,出现了反转
对于压缩后由正确变为错误的答案,模型常常仍保持中等偏高的自信程度(中位数约0.4到0.6),这种自信通常只在压缩强度极高、准确率已大幅崩溃之后才会明显下降
在WinoBias和BBQ偏见测试基准上,整体偏见分数在许多设置下几乎没有变化,但具体人口子群体却出现了巨大且有时方向相反的变化,例如某一职业群体的变化达到负53.1个百分点,而整体变化却接近零
这些隐藏效应在轻度到中度压缩下就已出现,当时常规指标看起来完全正常,因此作者主张在部署压缩模型前必须进行更细粒度的子群体级别评估

Figure 1: Accuracy across different popularity groups on PopQA under different compression settings. Results for all methods and models are provided in Figures 8, 9, and 10 in Appendix A.2.1.

Figure 2: Relative retention shift across PopQA popularity groups under different compression settings, reported in percentage points. Negative and positive RSg indicate lower and higher relative retention than the dataset overall, respectively. Results for all methods and models are provided in Figures 14, 15, and 16 in Appendix A.2.1.

Table 1: Quantization configurations.
Parameter	GPTQ	AWQ	OmniQuant	AQLM
Bit width	2, 3, 4	2, 3, 4	2, 3, 4	2, 3, 4
Calibration dataset	C4	C4	C4	C4
Calibration samples	128	128	128	128
Calibration seed	42	42	42	42
Calibration sequence length	512	512	512	512
Group size	128	128	128	–
Calibration split	validation	validation	–	–
Calibration source records	4096	–	–	–
Calibration batch size	1	1	–	–
Symmetric quantization	True	False	–	–
Activation ordering	True	–	–	–
Sequential quantization	True	–	–	–
Target modules	–	Linear	–	–
Ignored modules	–	lm_head	–	–
Activation bit width	–	–	16	–
Optimization epochs	–	–	40 / 20 / 20	–
Learnable weight clipping	–	–	True	–
Learnable equivalent transformation	–	–	False	–
Input group size	–	–	–	8
Output group size	–	–	–	1
Relative MSE tolerance	–	–	–	0.01
Maximum fine-tuning epochs	–	–	–	10
Activation offloading	–	–	–	True
Resume enabled	–	–	–	True

Figure 3: Knowledge-loss rate across popularity groups on PopQA under different compression settings. Higher values indicate that a larger proportion of the knowledge correctly answered by the base model is incorrect after compression. Results for all methods and models are provided in Figures 20, 21, and 22 in Appendix A.2.2.

Figure 4: Confidence on lost knowledge across popularity groups on PopQA under different compression settings. Each dot represents the median confidence; the thick vertical bar shows the 25th–75th percentile range, and the thin vertical bar shows the 5th–95th percentile range. Colors distinguish tail, middle, and head knowledge. Results for all methods and models are provided in Figures 26, 27, and 28 in Appendix A.2.2.

Table 2: Pruning configurations.
Parameter	Magnitude	WANDA	SparseGPT	ShortGPT	Layer Dropping
Compression granularity	weights	weights	weights	decoder blocks	decoder blocks
Compression levels	30/50/70%	30/50/70%	30/50/70%	5/10/15/20/25%	5/10/15/20/25%
Semi-structured patterns	–	4:8, 2:4	4:8, 2:4	–	–
Calibration dataset	–	C4	C4	C4	C4
Calibration samples	–	32	32	32	32
Calibration sequence length	–	512	512	512	256
Calibration split	–	validation	validation	validation	validation
Target modules	attn./MLP linear	attn./MLP linear	attn./MLP linear	decoder blocks	decoder blocks
Excluded components	lm_head	lm_head	lm_head	first/last block	first/last block
Selection criterion	weight magnitude	weight–activation product	Hessian-based reconstruction	block influence	importance + position
Activation-importance weight	–	–	–	–	0.7
Position-prior weight	–	–	–	–	0.35
Random seed	–	–	–	–	13

Figure 5: Bias changes on WinoBias under different compression settings, reported in percentage points. Blue, orange, and black bars denote changes for male, female, and all examples, respectively, and error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference, negative values indicate decreased preference, and the dashed line marks no change from the base model. Results for all methods and models are provided in Figures 38, 39, and 40 in Appendix A.2.3.

Figure 6: Overall accuracy on PopQA under different compression methods and settings.

Table 3: Perplexity of the three full-precision models and their compressed variants on WikiText-2.
Compression	Method	Setting	Perplexity (PPL)
			Llama-3.1- 8B-Instruct	Qwen-3- 8B	Gemma-2- 9B-it
None	Full precision	FP16	7.1253	9.5888	10.2117
Quantization	GPTQ	2-bit	1612.6539	141.3107	137.1019
		3-bit	10.0354	11.2652	12.3045
		4-bit	8.4459	9.9511	10.4903
	AWQ	2-bit	94354.1597	16508.5254	9407.2547
		3-bit	9.4931	11.3750	11.8154
		4-bit	7.5192	9.9949	10.6473
	OmniQuant	2-bit	671.2557	39.6147	35.2010
		3-bit	9.4949	11.7079	12.2236
		4-bit	7.5728	10.0737	10.5624
	AQLM	2-bit	11.5659	13.0687	14.1902
		3-bit	11.3204	12.3484	12.2632
		4-bit	8.4457	10.2807	10.8105
Unstructured pruning	Magnitude	30% sparsity	14.5756	11.5522	16.6274
		50% sparsity	177.7208	28.9189	65.7596
		70% sparsity	127104.4841	138372.7513	117527.1945
	Wanda	30% sparsity	9.2901	11.0325	12.9266
		50% sparsity	12.7384	12.7984	17.2316
		70% sparsity	236.8802	153.0605	106.8877
	SparseGPT	30% sparsity	9.5870	11.0395	13.9585
		50% sparsity	15.4637	13.9930	20.1264
		70% sparsity	272.6379	862.9209	129.0303
Semi-structured pruning	Wanda (N:M)	4:8	18.0268	14.7886	19.6716
		2:4	30.6551	18.4150	24.4520
	SparseGPT (N:M)	4:8	23.8732	16.9133	22.8913
		2:4	43.4478	21.1767	33.7740
Structured pruning	ShortGPT	5% blocks removed	9.8872	15.1473	13.5355
		10% blocks removed	11.1683	35.1417	14.7864
		15% blocks removed	19.4141	43.5930	21.1945

Figure 7: Overall accuracy on Head-to-Tail under different compression methods and settings.

Figure 8: Accuracy across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.

Table 4: The ten largest absolute occupation-level bias changes on WinoBias among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes ΔBg and overall changes ΔB are reported in percentage points.
Rank	Gender	Occupation	Compression Configuration	Base 𝑩𝒈 (%)	Comp. 𝑩𝒈 (%)	𝚫𝑩𝒈 (pp)	𝚫𝑩 (pp)
Llama-3.1-8B-Instruct
1	Female	Secretary	SparseGPT N:M (2:4)	71.9	18.8	−53.1	−2.2
3	Female	Auditor	SparseGPT N:M (4:8)	82.1	32.1	−50.0	+2.3
8	Male	Analyst	SparseGPT N:M (2:4)	50.0	2.5	−47.5	−2.2
9	Female	Clerk	SparseGPT N:M (2:4)	78.6	32.1	−46.4	−2.2
Qwen3-8B
2	Male	Sheriff	Magnitude (50%)	59.6	7.7	−51.9	−8.2
4	Female	Clerk	Magnitude (50%)	67.9	17.9	−50.0	−8.2
5	Female	Nurse	ShortGPT (20%)	77.8	27.8	−50.0	−4.2
6	Male	Developer	SparseGPT N:M (2:4)	71.9	21.9	−50.0	−4.0
7	Female	Attendant	Magnitude (50%)	61.8	12.7	−49.1	−8.2
10	Female	Counselor	Magnitude (50%)	66.1	19.6	−46.4	−8.2

Figure 9: Accuracy across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.

Figure 10: Accuracy across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.

Table 5: The ten largest absolute subgroup-level bias changes on BBQ among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes ΔBg and overall changes ΔB are reported in percentage points.
Rank	Category	Subgroup	Compression Configuration	Base 𝑩𝒈 (%)	Comp. 𝑩𝒈 (%)	𝚫𝑩𝒈 (pp)	𝚫𝑩 (pp)
Llama-3.1-8B-Instruct
8	Disability	People with cognitive disabilities or mental illness	SparseGPT N:M (4:8)	40.6	59.4	+18.8	−1.7
9	Disability	People with cognitive disabilities or mental illness	WANDA N:M (2:4)	40.6	59.4	+18.8	−3.1
Qwen3-8B
1	Disability	Down’s syndrome	ShortGPT (10%)	37.5	75.0	+37.5	−1.2
2	Disability	Down’s syndrome	ShortGPT (15%)	37.5	75.0	+37.5	−1.4
3	Disability	Down’s syndrome	ShortGPT (5%)	37.5	62.5	+25.0	−1.1
4	Disability	People with cerebral palsy	SparseGPT (50%)	56.3	81.3	+25.0	−1.0
10	Disability	Down’s syndrome	Magnitude (30%)	37.5	56.3	+18.8	−1.1
Gemma-2-9B-it
5	Disability	Down’s syndrome	ShortGPT (15%)	50.0	75.0	+25.0	−0.5
6	Disability	Down’s syndrome	ShortGPT (25%)	50.0	75.0	+25.0	−1.0
7	Nationality	Italian	WANDA N:M (2:4)	57.5	37.5	−20.0	+0.1

Figure 11: Accuracy across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.

Figure 12: Accuracy across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.

为什么重要

许多企业为降低部署成本而对大模型进行压缩,但这项研究表明,压缩模型可能在平均分数上看起来完好无损,实际上却悄悄丢失常见知识、对错误答案依然过度自信,并在性别、种族、残障等特定群体上产生不均衡的偏见变化。这提示实际部署压缩模型的团队,不能只看整体准确率或困惑度,还必须检查子群体层面的表现才能确保可靠性。

本文术语

量化(Quantization) · 将模型权重或计算数值用更低精度(如从32位降到4位)存储,以节省内存和计算量的压缩方法
剪枝(Pruning) · 删除模型中对输出贡献较小的权重、神经元或整层结构,使模型变小变快的压缩方法
困惑度(Perplexity) · 衡量语言模型预测下一个词能力的常用整体指标,数值越低通常表示模型表现越好
相对保留偏移(Relative Retention Shift) · 以百分点表示某一群体的准确率保留程度相对于模型整体保留程度高出或低出多少的指标
期望校准误差(ECE) · 衡量模型所表达的自信程度与实际正确率是否吻合的指标,数值越低说明自信与准确性越一致

论文原文摘要（英文）

Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts. In this work, we systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias. We find that compression disproportionately reduces the relative retention of head knowledge compared to tail knowledge. Furthermore, compressed models often remain substantially confident in their incorrect answers on newly lost knowledge. Finally, we demonstrate that stable aggregate bias scores can conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. Together, these findings reveal asymmetric behavioral changes that aggregate performance measures fail to capture, highlighting the need for granular evaluation of compressed models before deployment.

作者 · Yuan Wu, Mairui Li, Lesia Semenova, Chudi Zhong

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Yuan Wu et al., arXiv:2608.19670, CC BY 4.0

The Asymmetric Harms of LLM Compression

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道