매일 아침, 어제의 AI를 한 통으로 정리해 보내드립니다메일로 받아보기

METAL LAB

Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models

arXiv:2608.196802026-08-21

새 취약점이 계속 나와도 잊지 않고 하나의 모델로 합치는 스마트컨트랙트 취약점 탐지 AI

스마트컨트랙트에 새로운 취약점 유형이 계속 생겨나는데, 매번 LLM을 통째로 재훈련하기는 너무 비싸다는 문제를 다뤘다. 연구팀은 적은 파라미터로 학습하고, 이전 지식을 안 잊게 복습시키고, 여러 버전을 하나로 합치는 3단계 파이프라인을 제안했다. DIVE라는 실제 스마트컨트랙트 벤치마크에서 개별적으로 각각 학습한 상한선 성능에 2.7% 차이까지 근접하면서도 병합에 156밀리초밖에 안 걸렸다.

무엇을 했나

  1. 문제: 스마트컨트랙트는 배포되면 고칠 수 없어서 사전에 취약점을 잡아야 하는데, 새로운 공격 유형이 계속 등장해 기존 모델이 낡아진다. 전체 재훈련은 너무 비싸고, 작업별로 따로 모델을 만들면 저장·운영 비용이 커진다.
  2. 방법1(FA-LoRA): LLM 본체는 그대로 얼려두고, 아주 적은 파라미터(전체의 0.4%)만 학습하는 방식인데, 이걸 일반적인 공간이 아니라 푸리에(주파수) 영역에서 수행하고 어떤 주파수 성분이 중요한지 학습 가능한 게이트로 조절한다.
  3. 방법2(FAR): 여러 작업을 순서대로 학습하면 예전에 배운 내용을 까먹는 '치명적 망각' 문제가 생기는데, 각 데이터 샘플의 손실(예측이 얼마나 틀렸는지) 변화를 추적해 잊혀질 위험이 큰 샘플을 더 자주 복습시키는 방식으로 이를 완화했다.
  4. 방법3(APPM): 순서대로 학습된 여러 개의 어댑터(작업별 소형 모델 조각)를 실제 서비스에서는 하나로 합쳐야 하는데, 가장 일반화가 잘 된 어댑터를 기준점(앵커)으로 삼아 보호하면서 나머지를 가중 평균하고, 주파수 게이트끼리 경쟁시켜 병합했다.
  5. 결과: FA-LoRA는 표준 LoRA·QLoRA보다 성능이 좋으면서 파라미터는 0.4%만 사용했고, FAR은 순차 학습에서 평균 Micro-F1 0.8022를 기록했으며, APPM은 병합 후에도 개별 학습 상한선 대비 2.7% 이내(Micro-F1 0.8085)로 성능을 유지하면서 병합 시간은 156밀리초, 추가 메모리는 없었다.
Fig. 1: Overview of the proposed continual learning framework for LLM-based smart contract vulnerability detection.
Fig. 1: Overview of the proposed continual learning framework for LLM-based smart contract vulnerability detection.
Fig. 2: Architecture of FA-LoRA.
Fig. 2: Architecture of FA-LoRA.
TABLE I: DIVE benchmark statistics.
Splittask_Atask_Btask_Ctask_D
Train5,2625,2625,2625,262
Validation542504536651
Test530542513648
Total6,3346,3086,3116,561
Fig. 3: The FAR mechanism performing forget-aware prioritization via per-sample loss recomputation and temperature-scaled softmax sampling.
Fig. 3: The FAR mechanism performing forget-aware prioritization via per-sample loss recomputation and temperature-scaled softmax sampling.
Fig. 4: The two-stage APPM workflow.
Fig. 4: The two-stage APPM workflow.
TABLE II: Hyperparameter Configuration
HyperparameterValue
FA-LoRA rank r [16]16
Frequency mode [5]High-frequency retention
Retain fraction γ [5]0.2
Optimizer [30]AdamW
Learning rate (CL) [16]5×10−5
Learning rate (PEFT) [16]3×10−5
Batch size B [16]8
Max sequence length1,024
Epochs per task (CL)3–5
Epochs (PEFT)3
Replay buffer capacity [6]2,000
Replay batch ratio [6]0.25
FAR temperature τ2.0
APPM protection ρ1.0
Fig. 5: Continual learning algorithm comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each method, with dashed lines indicating the Independent single-task upper bound.
Fig. 5: Continual learning algorithm comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each method, with dashed lines indicating the Independent single-task upper bound.
Fig. 6: APPM merging comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each merging method, with dashed lines indicating the Independent single-task upper bound. APPM achieves the best average Micro-F1 of 0.8085, closest to the Independent upper bound.
Fig. 6: APPM merging comparison on the DIVE benchmark. Grouped bars show per-task Micro-F1 for each merging method, with dashed lines indicating the Independent single-task upper bound. APPM achieves the best average Micro-F1 of 0.8085, closest to the Independent upper bound.
TABLE III: Parameter breakdown of FA-LoRA.
ComponentParams (M)Fraction
Total1,241.0100.0%
Frozen (LLaMA base)1,235.899.6%
Trainable (FA-LoRA)5.20.4%
— LoRA (𝑼,𝑽)5.00.40%
— Gates (𝒈)0.20.02%
Per-task storage∼10 MB
Fig. 7: APPM component ablation. All variants merge 4 FAR adapters. The full APPM configuration achieves the best balance across all four tasks, while removing either anchor protection or frequency competition degrades performance, so both mechanisms are necessary.
Fig. 7: APPM component ablation. All variants merge 4 FAR adapters. The full APPM configuration achieves the best balance across all four tasks, while removing either anchor protection or frequency competition degrades performance, so both mechanisms are necessary.
Fig. 8: Sensitivity analysis across four hyperparameter dimensions. (a) LoRA rank r is robust with ΔF1=0.0041. (b) Frequency retention γ is similarly insensitive with ΔF1=0.0022. (c) APPM protection ρ is the only impactful parameter, yielding +3.07% monotonic improvement. (d) FAR temperature τ has negligible impact with ΔF1=0.0021.
Fig. 8: Sensitivity analysis across four hyperparameter dimensions. (a) LoRA rank r is robust with ΔF1=0.0041. (b) Frequency retention γ is similarly insensitive with ΔF1=0.0022. (c) APPM protection ρ is the only impactful parameter, yielding +3.07% monotonic improvement. (d) FAR temperature τ has negligible impact with ΔF1=0.0021.
TABLE IV: Comparison of PEFT methods on the full DIVE dataset without task splitting.
MethodQuant.Trainable (M)Storage (MB)Micro-F1Macro-F1Subset Acc.
1B3B1B3B1B3B
WaRA [15]✓35.77136.50.83980.85150.65290.71330.55440.5840
QLoRA [9]✓1.726.60.81850.83650.61330.64070.51810.5298
SLoRA [17]✓6.8526.20.81380.83050.59450.62630.49660.5262
LoRA [16]bf163.4213.00.80940.83700.56130.63560.48450.5428
FourierFT [13]✓0.160.60.74490.78880.47240.55850.34710.4331
FouRA [5]✓0.552.10.76350.80200.49980.59300.40040.4648
FA-LoRA✓2.6210.00.81850.84240.59040.66160.50740.5544
(b) Sensitivity to frequency retention γ.
(b) Sensitivity to frequency retention γ.
(c) Sensitivity to APPM protection ρ.
(c) Sensitivity to APPM protection ρ.
TABLE V: Resource metrics for adapter merging methods. All methods produce identical model architectures.
MethodMerge (ms)CPU Δ(MB)SpeedupΔInd
Simple-Mean1020.070×+5.5%
TIES [40]2,1230.03.4×+5.5%
DARE [44]4,2040.01.7×+7.9%
HAM g=27,16468.11.0×+11.0%
SFA a=0.5720.0100×+33.0%
APPM (ours)1560.046×+2.7%
(d) Sensitivity to FAR temperature τ.
(d) Sensitivity to FAR temperature τ.
TABLE VI: Backward evaluation matrix for FAR. Bold entries denote in-task F1, entries below the diagonal denote backward transfer, and entries above the diagonal denote forward transfer.
After trainingtask_Atask_Btask_Ctask_D
task_A0.74950.61210.63310.6028
task_B0.72530.75030.73030.7065
task_C0.71620.72260.89740.8864
task_D0.71280.71710.89350.8854
Forgetting−0.0367−0.0332−0.0039—

왜 중요한가

블록체인 보안팀 입장에서는 새로운 취약점이 발견될 때마다 탐지 모델을 통째로 다시 만들 필요 없이 가볍게 업데이트하고 하나의 배포 모델로 유지할 수 있다는 뜻이다. 이는 실제 감사(auditing) 환경처럼 취약점 정보가 시간에 따라 계속 갱신되는 상황에 실용적인 접근을 제시한다.

이 논문의 용어

  • LoRA(Low-Rank Adaptation) · 큰 모델 전체를 재훈련하지 않고 작은 추가 행렬만 학습해 적은 비용으로 모델을 새 작업에 맞추는 기법
  • 치명적 망각(Catastrophic Forgetting) · 모델이 새 작업을 배우면서 이전에 배운 내용을 잃어버리는 현상
  • 푸리에 변환(Fourier Transform) · 신호나 데이터를 다양한 주파수 성분으로 분해해서 표현하는 수학적 변환
  • 어댑터(Adapter) · 큰 모델은 그대로 두고 특정 작업에 맞게 소량만 추가로 학습시킨 작은 모듈
  • Micro-F1 · 여러 개의 라벨(취약점 종류)을 동시에 맞혀야 하는 상황에서 정확도와 재현율을 종합한 평가 지표

논문 원문 초록 (영문)

Smart contract vulnerability detection with Large Language Models (LLMs) faces three causally linked challenges. First, new vulnerability categories demand parameter-efficient adaptation, since full retraining is prohibitive for sequentially arriving tasks. Second, training per-task adapters on a shared backbone causes catastrophic forgetting of previously learned vulnerabilities. Third, the resulting multiplicity of adapters must be consolidated into a single model, since task identity is unknown at inference time. Each challenge arises directly from the solution to its predecessor, making an integrated framework essential. We propose a three-stage pipeline in which each stage addresses one challenge and feeds into the next. The adaptation stage uses Frequency-Aware Low-Rank Adaptation (FA-LoRA), which performs adaptation in the Fourier domain with per-frequency importance gates, requiring only 0.4% trainable parameters while outperforming standard LoRA and QLoRA. The continual learning stage applies Forget-Aware Replay (FAR), which uses these frequency gates to estimate per-sample forgetting risk via loss dynamics and prioritizes vulnerable knowledge for rehearsal, achieving an average Micro-F1 of 0.8022 across sequential tasks. The deployment stage employs Anchor-Protected Progressive Merging (APPM), which exploits the asymmetric generalization produced by FAR training to identify the strongest-generalizing adapter as an anchor and consolidates all adapters into a single model via anchor-protected weighted merging with frequency-domain gate competition. APPM achieves a Micro-F1 of 0.8085, within 2.7% of the independent per-task upper bound, at a merge cost of 156 ms and no additional runtime memory. Experiments on DIVE confirm the framework effectively addresses all three challenges for evolving blockchain ecosystems.

저자 · Tenghui Huang, Jiawen Kang, Dongning Liu, Changyan Yi, Chengjun Cai, Anjia Yang, Li Li, Dong In Kim

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL LAB 최신 기사

그림 출처: Tenghui Huang et al., arXiv:2608.19680, arxiv-nonexclusive