每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG

arXiv:2608.195352026-08-21

在边缘设备上跑RAG时,检索到的文本压缩多少必须实时决定,否则白白浪费电

RAG通过检索外部文档来让语言模型的回答更准确,但检索到的文本越长,在边缘设备上花费的时间和电量就越多。这篇论文在NVIDIA Jetson AGX Thor边缘设备上实测了这种权衡,发现压缩存在一个合适区间:压缩太轻会浪费节能机会,压缩太狠又会损害回答质量。中等程度的压缩能让GPU能耗最多降低53.2%,整机能耗最多降低48.2%,而回答质量几乎不受影响。

他们做了什么

  1. 对于7B到8B参数规模的大模型,生成回答这一步(而非检索或压缩)占据了每次查询约90%的时间和91%的GPU能耗
  2. 用LLMLingua-2压缩工具把压缩率从1.0调到0.15发现,压缩率在1.0到0.3之间时回答质量(F1分数)基本保持不变,但降到0.15时质量骤降4到10分
  3. 压缩率0.9这种轻度压缩反而会增加总能耗,因为压缩本身有固定开销(每次查询130到310毫秒),抵消了压缩少量文本带来的节省;只有压缩率降到0.7以下才能真正省电
  4. 压缩率0.3被认为是'安全又激进'的最佳设定点,节能效果随模型规模和检索文档数量增大,从30%到53%不等
  5. 作者主张需要一个运行时控制器,根据延迟、能耗、温度等设备实时信号来决定是否压缩、压缩多少,而不是固定使用一个压缩比例
Figure 1. Telemetry-informed adaptive compression for edge RAG. Retrieved passages are compressed before generation, but the compression ratio is selected at runtime by a controller that observes edge-SoC telemetry and optimizes for latency, energy, thermal, memory, and accuracy constraints.
Figure 1. Telemetry-informed adaptive compression for edge RAG. Retrieved passages are compressed before generation, but the compression ratio is selected at runtime by a controller that observes edge-SoC telemetry and optimizes for latency, energy, thermal, memory, and accuracy constraints.
Table 1. Experimental Setup
ItemConfiguration
PlatformJetson AGX Thor: Blackwell GPU, 128 GB LPDDR5x, 273 GB/s, 130 W.
Corpus / indexEnglish Wikipedia 2018, sentence-split via FlashRAG (∼9.4 M passages) (16); e5-base-v2 encoder (50); FAISS GPU IndexFlatL2 ( 28.2 GB).
QA DatasetsNatural Questions (20) and HotpotQA (54); 100 seed-paired queries per config.
ModelsLlama-3.2 1B/3B, Llama-3.1 8B (9); Qwen-2.5 1.5B/3B/7B (53). All fp16.
CompressionNone, or LLMLingua-2 (34)
SweepsExp. 1 uses k∈{1,5,10} with compression off. Exp. 2 uses HotpotQA, k∈{5,10}, and LLMLingua-2 rates 1.0, 0.9, 0.7, 0.5, 0.3, and 0.15.
ControlsSingle-query mode, reranker off, standard pipeline, randomized config order, first 3 queries dropped as warm-up.
Metrics/telemetryEM, token-level F1, retrieval recall; end-to-end and per-stage latency; GPU/SoC energy, power, memory, and temperature from tegrastats at 100 ms cadence.
Figure 2. Per-query share of latency (left) and GPU energy (right) by stage, on AGX Thor, fp16, no compression.
Figure 2. Per-query share of latency (left) and GPU energy (right) by stage, on AGX Thor, fp16, no compression.
Table 2. Net savings at rate=0.3 vs. the in-session uncompressed baseline (positive Δ = saving on the cost columns, improvement on the quality columns).
ModelkΔlat.Δ​EGPUΔ​ESoCΔEMΔF1
Llama-3B5+8.5%+30.2%+28.2%+0.000−0.005
Llama-3B10+15.6%+40.6%+35.1%+0.010+0.012
Llama-8B5+25.2%+44.9%+38.5%+0.052+0.016
Llama-8B10+32.6%+53.2%+48.2%+0.000−0.011
Figure 3. Answer F1 (left axis) and per-query GPU energy (right axis) vs. LLMLingua-2 rate on HotpotQA. Shaded band: adaptive operating room between the two dotted-line knees.
Figure 3. Answer F1 (left axis) and per-query GPU energy (right axis) vs. LLMLingua-2 rate on HotpotQA. Shaded band: adaptive operating room between the two dotted-line knees.

为什么重要

对于在手机、机器人、AR设备等电池和散热受限的场景中部署RAG的开发者来说,这项研究说明一味加大压缩力度可能适得其反,并给出了一个较安全的压缩参考点(压缩率0.3)。它把压缩从一次性的预处理步骤,重新定义为需要根据设备实时状态动态调整的资源管理问题。

Figure 4. Net GPU energy delta (top) and ΔF1 (bottom) at rate=0.9 (mild) and rate=0.3 (safe-aggressive), vs. the in-session uncompressed baseline.
Figure 4. Net GPU energy delta (top) and ΔF1 (bottom) at rate=0.9 (mild) and rate=0.3 (safe-aggressive), vs. the in-session uncompressed baseline.

本文术语

  • RAG(检索增强生成) · 语言模型在回答前先检索外部文档,再依据检索内容生成答案的方法
  • 上下文压缩 · 在把检索到的文档喂给模型之前,先削减或精简其内容以缩短输入长度
  • 压缩率(rate) · 压缩后保留的原文比例,例如rate=0.3表示只保留原文的30%
  • LLMLingua-2 · 一种通过剔除不重要词元来压缩文本的工具,同时保留原文其余部分
  • 边缘SoC · 用于手机、机器人等设备、功耗和散热都受限的片上系统芯片,本文实测所用的Jetson AGX Thor即属此类

论文原文摘要（英文）

Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.

作者 · Zlatan Feric, Amir Taherin, Yanzhi Wang, David Kaeli

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Zlatan Feric et al., arXiv:2608.19535, arxiv-nonexclusive