每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment

arXiv:2608.198002026-08-21

在正式微调前先偷看几步训练的梯度,让LoRA的初始化更聪明

LoRA能用很少的显存微调大模型,但效果一直不如全参数微调。这篇论文提出LoRA-GA2,先用一个轻量探针跑几步早期训练获取梯度信息,再据此决定每层该分配多少低秩容量、以及如何初始化LoRA权重。在GLUE、GSM8K和HumanEval等基准上,该方法持续优于现有的LoRA变体。

他们做了什么

  1. 以往基于梯度的LoRA方法只看训练最开始那一瞬间的梯度(单步梯度),无法反映真实训练过程中优化路径的变化。
  2. 另一类方法在每一步训练中都尝试对齐梯度,虽然更精确,但需要改动优化器本身,大幅增加显存占用和训练时间。
  3. LoRA-GA2改用一个轻量优化器AdaLomo临时跑几步,把梯度累积保存在CPU上,再恢复原始预训练权重,用这份累积信号来分配各层的秩并初始化LoRA矩阵,这一过程不额外占用GPU显存,只增加很小的时间开销。
  4. 在秩分配上,论文把敏感度分数(该层对任务有多重要)和有效秩分数(该层梯度方向有多分散)相乘使用,避免只依赖单一指标造成的容量浪费。
  5. 在T5-Base的GLUE基准(秩为8)上,LoRA-GA2平均分比最强基线高0.66分;在Llama3.1-8B-Base上,GSM8K比最强基线高1.03分,HumanEval高0.87分;在CLIP-ViT-B/16图像分类任务上也比最强基线高0.63分。
Figure 1: Illustration of (a) LoRA; (b) One-step gradients based LoRA Variants; (c) Multi-step gradients based LoRA Variants; and (d) Ours (LoRA-GA2), which introduces an efficient probe for estimating multi-step gradients of pre-trained weights and utilizing the directions and intrinsic dimensionalities of the gradients to initialize low-rank weights.
Figure 1: Illustration of (a) LoRA; (b) One-step gradients based LoRA Variants; (c) Multi-step gradients based LoRA Variants; and (d) Ours (LoRA-GA2), which introduces an efficient probe for estimating multi-step gradients of pre-trained weights and utilizing the directions and intrinsic dimensionalities of the gradients to initialize low-rank weights.
Table 1: Performance of fine-tuning T5-Base on 5 sub-tasks of the GLUE benchmark. Bold and underline indicate the highest and second-highest scores of low-rank methods with r=8 or rref=8.
MethodMNLISST-2CoLAQNLIMRPCAverage
Full86.33±0.0094.75±0.2180.70±0.2493.19±0.2284.56±0.7387.91
LoRA (13)85.30±0.0494.04±0.1169.35±0.0593.19±0.2284.56±0.7385.29
Convergence Optimization Methods for LoRA
rsLoRA (15)85.73±0.1094.19±0.2372.32±1.1293.12±0.0952.86±2.2779.64
DoRA (19)85.67±0.0994.04±0.5372.04±0.9493.04±0.0668.08±0.5182.57
LoRA+ (9)85.81±0.0993.85±0.2477.53±0.2093.14±0.0374.43±1.3984.95
Initialization Optimization Methods for LoRA
PiSSA (22)85.75±0.0794.07±0.0674.27±0.3993.15±0.1476.31±0.5184.71
LoRA-GA (30)85.70±0.0994.11±0.1880.57±0.2093.18±0.0685.29±0.2487.77
Adaptive Methods for LoRA
AdaLoRA (37)85.45±0.1193.69±0.2069.16±0.2491.66±0.0568.14±0.2881.62
RaLoRA (34)85.76±0.0394.22±0.2978.11±0.4593.36±0.1484.74±0.2787.24
GoRA (10)85.91±0.2294.68±0.4379.86±0.3593.27±0.0886.10±0.2087.96
LoRA-GA2 (Ours)85.91±0.0194.72±0.3782.39±0.2493.19±0.0586.88±0.1488.62
Figure 2: Framework of LoRA-GA2. A temporary multi-step probe collects trajectory gradients, restores the pretrained weights, and uses the accumulated signal for both rank allocation and SVD initialization before standard LoRA training.
Figure 2: Framework of LoRA-GA2. A temporary multi-step probe collects trajectory gradients, restores the pretrained weights, and uses the accumulated signal for both rank allocation and SVD initialization before standard LoRA training.
Table 2: Performance of fine-tuning Llama3.1-8B-Base on GSM8K and HumanEval.
MethodGSM8KHumanEval
Full73.69±0.2851.63±1.27
LoRA (13)67.78±1.2543.09±0.35
rsLoRA (15)68.36±0.7445.78±2.80
DoRA (19)69.17±1.0043.70±1.54
LoRA+ (9)71.29±0.9344.51±2.11
OLoRA (2)68.54±0.4243.29±2.44
PiSSA (22)68.56±1.0344.10±1.54
LoRA-GA (30)71.39±0.9043.29±0.61
AdaLoRA (37)70.63±0.7741.46±3.66
RaLoRA (34)72.25±0.5948.78±1.61
GoRA (10)72.91±0.7648.98±2.14
LoRA-GA2 (Ours)73.94±0.4849.85±0.33
Figure 3: Loss comparison of fine-tuning Llama3.1-8B-Base on MetamathQA using LoRA and LoRA-GA2
Figure 3: Loss comparison of fine-tuning Llama3.1-8B-Base on MetamathQA using LoRA and LoRA-GA2
Table 3: Performance of fine-tuning CLIP-ViT-B/16 on seven image classification tasks.
MethodCarsDTDEuroSATGTSRBRESISC45SUN397SVHNAverage
Zero-shot63.7544.3942.2235.2256.4662.5615.5345.73
LoRA (13)82.31±0.0876.97±0.5198.38±0.2097.10±0.0694.99±0.1177.19±0.1996.62±0.0689.08±0.10
MELoRA (26)82.65±0.3875.16±0.5998.64±0.0598.88±0.0595.78±0.1674.69±0.2296.95±0.0988.96±0.15
MoRA (14)84.61±0.2177.34±0.1498.65±0.1698.68±0.1896.33±0.1978.12±0.0697.17±0.1590.13±0.16
AdaLoRA (37)73.58±0.0973.79±0.4896.96±0.1258.87±0.3889.07±0.6072.00±0.1094.26±0.1379.79±0.27
DoRA (19)82.44±0.2676.86±0.8498.43±0.1797.25±0.1295.10±0.1677.30±0.1796.63±0.0489.14±0.07
rsLoRA (15)83.94±0.2277.64±0.3398.51±0.1798.69±0.1795.90±0.2077.96±0.2196.94±0.0689.94±0.06
LoRA+ (9)86.61±0.3673.33±1.3098.54±0.1498.99±0.2096.06±0.3876.80±0.3496.98±0.0889.62±0.19
PiSSA (22)83.36±0.3877.38±0.5798.54±0.0998.32±0.0995.92±0.4077.46±0.1397.00±0.0989.71±0.25
OLoRA (2)83.85±0.1378.60±0.2598.62±0.0398.49±0.1496.01±0.2877.30±0.0897.15±0.1490.00±0.15
RaLoRA (34)86.63±0.3077.75±0.2098.66±0.2798.98±0.1196.62±0.2877.86±0.0597.24±0.1190.53±0.03
LoRA-GA2 (Ours)87.82±0.1379.49±0.1198.83±0.1199.01±0.0896.48±0.2479.05±0.1497.42±0.0791.16±0.05
Figure 4: Rank allocation results of fine-tuning Llama3.1-8B-Base-base on MetamathQA.
Figure 4: Rank allocation results of fine-tuning Llama3.1-8B-Base-base on MetamathQA.
Table 4: Key ablations with standard deviations on Llama3.1-8B-Base.
VariantGSM8KHumanEval
One-step gradient72.13±0.3449.65±1.01
No SVD init.68.92±0.1248.37±2.50
No rank alloc.72.40±1.3248.88±0.63
Sensitivity only72.98±0.6548.17±2.28
Effective rank only73.51±0.5148.78±1.00
Full LoRA-GA273.94±0.4849.85±0.33
Figure 5: Ablation of the hyperparameters N and γ.
Figure 5: Ablation of the hyperparameters N and γ.

为什么重要

LoRA已经是广泛使用的微调方法,这类改进意味着在同样的计算资源下能得到更好的模型效果。由于几乎不增加显存、时间开销也很小,这种方法很容易直接接入现有的LoRA训练流程。

本文术语

  • LoRA(低秩适配) · 不直接修改大模型全部权重,只训练两个小矩阵来完成微调,从而节省显存的方法
  • 单步/多步梯度 · 梯度指示模型该往哪个方向调整才能降低损失;单步只在训练最开始测一次,多步则在训练早期连续多个步骤中观察
  • SVD(奇异值分解)初始化 · 把累积梯度分解成几个主要方向和强度,并让LoRA权重的初始值对齐这些方向
  • 有效秩 · 衡量某一层梯度分布在多少个独立方向上的指标
  • 敏感度 · 衡量某一层权重变化对最终损失影响大小的分数,反映该层对任务的重要程度

论文原文摘要（英文）

Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA$^2$, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA$^2$ consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA$^2$ surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.

作者 · Haonan He, Xinyue Fan

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Haonan He et al., arXiv:2608.19800, CC BY 4.0