每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models

arXiv:2608.196802026-08-21

智能合约漏洞检测AI能不断学新漏洞、不忘旧知识,还能合并成一个模型上线

智能合约的漏洞类型在不断增加,但每次都把大语言模型全部重新训练一遍成本太高。研究团队设计了一套三阶段方案,先用极少参数完成轻量适配,再通过有针对性的复习防止遗忘旧知识,最后把多个阶段学到的模块合并成一个统一模型。在真实智能合约数据集DIVE上,合并后的模型性能与逐任务单独训练的理论上限相差仅2.7%,而合并过程只需156毫秒。

他们做了什么

  1. 问题:已部署的智能合约无法事后修改,漏洞必须提前发现,但新的攻击手法不断出现;每次都全量重训大模型代价太高,而给每类漏洞单独训练一个模型又会造成存储和运行开销过大。
  2. 方法一(FA-LoRA):保持大语言模型主体参数冻结,只训练极小一部分参数(仅占总参数的0.4%),并且把这部分适配放到傅里叶(频率)域中进行,用一个可学习的门控来决定哪些频率成分更重要。
  3. 方法二(FAR):依次学习多个任务时,模型容易发生对旧知识的'灾难性遗忘',该方法通过跟踪每个训练样本的损失(预测误差)变化,识别出最容易被遗忘的样本,并在后续训练中优先复习这些样本。
  4. 方法三(APPM):多个按任务顺序训练出的小模块最终需要合并成一个统一模型才能部署,该方法先选出泛化能力最强的模块作为受保护的'锚点',再对其余模块做加权平均,并让各模块的频率门控相互竞争来完成合并。
  5. 结果:FA-LoRA仅用0.4%的可训练参数就超过了标准LoRA和QLoRA;FAR在连续任务学习中取得平均Micro-F1 0.8022;APPM合并后的模型与逐任务独立训练的性能上限相差仅2.7%(Micro-F1为0.8085),合并耗时156毫秒且不增加运行时内存。
Fig. 1: Overview of the proposed continual learning framework for LLM-based smart contract vulnerability detection.
Fig. 1: Overview of the proposed continual learning framework for LLM-based smart contract vulnerability detection.
Fig. 2: Architecture of FA-LoRA.
Fig. 2: Architecture of FA-LoRA.
TABLE I: DIVE benchmark statistics.
Splittask_Atask_Btask_Ctask_D
Train5,2625,2625,2625,262
Validation542504536651
Test530542513648
Total6,3346,3086,3116,561
Fig. 3: The FAR mechanism performing forget-aware prioritization via per-sample loss recomputation and temperature-scaled softmax sampling.
Fig. 3: The FAR mechanism performing forget-aware prioritization via per-sample loss recomputation and temperature-scaled softmax sampling.
Fig. 4: The two-stage APPM workflow.
Fig. 4: The two-stage APPM workflow.
TABLE II: Hyperparameter Configuration
HyperparameterValue
FA-LoRA rank r [16]16
Frequency mode [5]High-frequency retention
Retain fraction γ [5]0.2
Optimizer [30]AdamW
Learning rate (CL) [16]5×10−5
Learning rate (PEFT) [16]3×10−5
Batch size B [16]8
Max sequence length1,024
Epochs per task (CL)3–5
Epochs (PEFT)3
Replay buffer capacity [6]2,000
Replay batch ratio [6]0.25
FAR temperature τ2.0
APPM protection ρ1.0
Fig. 5: Continual learning algorithm comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each method, with dashed lines indicating the Independent single-task upper bound.
Fig. 5: Continual learning algorithm comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each method, with dashed lines indicating the Independent single-task upper bound.
Fig. 6: APPM merging comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each merging method, with dashed lines indicating the Independent single-task upper bound. APPM achieves the best average Micro-F1 of 0.8085, closest to the Independent upper bound.
Fig. 6: APPM merging comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each merging method, with dashed lines indicating the Independent single-task upper bound. APPM achieves the best average Micro-F1 of 0.8085, closest to the Independent upper bound.
TABLE III: Parameter breakdown of FA-LoRA.
ComponentParams (M)Fraction
Total1,241.0100.0%
Frozen (LLaMA base)1,235.899.6%
Trainable (FA-LoRA)5.20.4%
— LoRA (𝑼,𝑽)5.00.40%
— Gates (𝒈)0.20.02%
Per-task storage∼10 MB
Fig. 7: APPM component ablation. All variants merge 4 FAR adapters. The full APPM configuration achieves the best balance across all four tasks, while removing either anchor protection or frequency competition degrades performance, so both mechanisms are necessary.
Fig. 7: APPM component ablation. All variants merge 4 FAR adapters. The full APPM configuration achieves the best balance across all four tasks, while removing either anchor protection or frequency competition degrades performance, so both mechanisms are necessary.
Fig. 8: Sensitivity analysis across four hyperparameter dimensions. (a) LoRA rank r is robust with ΔF1=0.0041. (b) Frequency retention γ is similarly insensitive with ΔF1=0.0022. (c) APPM protection ρ is the only impactful parameter, yielding +3.07% monotonic improvement. (d) FAR temperature τ has negligible impact with ΔF1=0.0021.
Fig. 8: Sensitivity analysis across four hyperparameter dimensions. (a) LoRA rank r is robust with ΔF1=0.0041. (b) Frequency retention γ is similarly insensitive with ΔF1=0.0022. (c) APPM protection ρ is the only impactful parameter, yielding +3.07% monotonic improvement. (d) FAR temperature τ has negligible impact with ΔF1=0.0021.
TABLE IV: Comparison of PEFT methods on the full DIVE dataset without task splitting.
MethodQuant.Trainable (M)Storage (MB)Micro-F1Macro-F1Subset Acc.
1B3B1B3B1B3B
WaRA [15]✓35.77136.50.83980.85150.65290.71330.55440.5840
QLoRA [9]✓1.726.60.81850.83650.61330.64070.51810.5298
SLoRA [17]✓6.8526.20.81380.83050.59450.62630.49660.5262
LoRA [16]bf163.4213.00.80940.83700.56130.63560.48450.5428
FourierFT [13]✓0.160.60.74490.78880.47240.55850.34710.4331
FouRA [5]✓0.552.10.76350.80200.49980.59300.40040.4648
FA-LoRA✓2.6210.00.81850.84240.59040.66160.50740.5544
(b) Sensitivity to frequency retention γ.
(b) Sensitivity to frequency retention γ.
(c) Sensitivity to APPM protection ρ.
(c) Sensitivity to APPM protection ρ.
TABLE V: Resource metrics for adapter merging methods. All methods produce identical model architectures.
MethodMerge (ms)CPU Δ(MB)SpeedupΔInd
Simple-Mean1020.070×+5.5%
TIES [40]2,1230.03.4×+5.5%
DARE [44]4,2040.01.7×+7.9%
HAM g=27,16468.11.0×+11.0%
SFA a=0.5720.0100×+33.0%
APPM (ours)1560.046×+2.7%
(d) Sensitivity to FAR temperature τ.
(d) Sensitivity to FAR temperature τ.
TABLE VI: Backward evaluation matrix for FAR. Bold entries denote in-task F1, entries below the diagonal denote backward transfer, and entries above the diagonal denote forward transfer.
After trainingtask_Atask_Btask_Ctask_D
task_A0.74950.61210.63310.6028
task_B0.72530.75030.73030.7065
task_C0.71620.72260.89740.8864
task_D0.71280.71710.89350.8854
Forgetting−0.0367−0.0332−0.0039—

为什么重要

对区块链安全团队而言,这意味着可以低成本地把新发现的漏洞类型纳入检测模型,而不必整体重训,也不用同时维护多个模型,最终仍能以单一模型形式部署上线。这为漏洞知识随时间不断更新的真实审计场景提供了一条实用路径。

本文术语

  • LoRA(低秩适配) · 不重新训练整个大模型,只训练少量额外参数矩阵来让模型适应新任务的技术
  • 灾难性遗忘 · 模型在学习新任务过程中丢失此前学到的知识的现象
  • 傅里叶变换 · 将数据分解为不同频率成分的一种数学变换方法
  • 适配器(Adapter) · 在主模型保持冻结的情况下,为特定任务额外训练的小模块
  • Micro-F1 · 综合精确率和召回率的评价指标,适用于一个样本可能同时属于多个标签(漏洞类型)的情况

论文原文摘要（英文）

Smart contract vulnerability detection with Large Language Models (LLMs) faces three causally linked challenges. First, new vulnerability categories demand parameter-efficient adaptation, since full retraining is prohibitive for sequentially arriving tasks. Second, training per-task adapters on a shared backbone causes catastrophic forgetting of previously learned vulnerabilities. Third, the resulting multiplicity of adapters must be consolidated into a single model, since task identity is unknown at inference time. Each challenge arises directly from the solution to its predecessor, making an integrated framework essential. We propose a three-stage pipeline in which each stage addresses one challenge and feeds into the next. The adaptation stage uses Frequency-Aware Low-Rank Adaptation (FA-LoRA), which performs adaptation in the Fourier domain with per-frequency importance gates, requiring only 0.4% trainable parameters while outperforming standard LoRA and QLoRA. The continual learning stage applies Forget-Aware Replay (FAR), which uses these frequency gates to estimate per-sample forgetting risk via loss dynamics and prioritizes vulnerable knowledge for rehearsal, achieving an average Micro-F1 of 0.8022 across sequential tasks. The deployment stage employs Anchor-Protected Progressive Merging (APPM), which exploits the asymmetric generalization produced by FAR training to identify the strongest-generalizing adapter as an anchor and consolidates all adapters into a single model via anchor-protected weighted merging with frequency-domain gate competition. APPM achieves a Micro-F1 of 0.8085, within 2.7% of the independent per-task upper bound, at a merge cost of 156 ms and no additional runtime memory. Experiments on DIVE confirm the framework effectively addresses all three challenges for evolving blockchain ecosystems.

作者 · Tenghui Huang, Jiawen Kang, Dongning Liu, Changyan Yi, Chengjun Cai, Anjia Yang, Li Li, Dong In Kim

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Tenghui Huang et al., arXiv:2608.19680, arxiv-nonexclusive