One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG

arXiv:2608.195352026-08-21

Edge devices running RAG need to decide on the fly how much to compress retrieved text, or they waste energy

RAG systems make answers more accurate by pulling in outside documents, but longer retrieved text costs edge devices extra time and power. This paper measures that tradeoff on an NVIDIA Jetson AGX Thor edge device and finds a sweet-spot compression range: too little compression wastes energy, too much hurts answer quality. A middle-ground compression level cut GPU energy use by up to 53.2% and total device energy by up to 48.2%, with almost no drop in answer quality.

What they did

  1. For 7B-8B parameter language models, the answer-generation step alone (not retrieval or compression) eats up about 90% of per-query time and 91% of GPU energy
  2. Sweeping the compression rate of LLMLingua-2 from 1.0 down to 0.15 showed answer quality (F1 score) staying stable between rate 1.0 and 0.3, then dropping sharply by 4-10 points at rate 0.15
  3. Mild compression at rate 0.9 actually wastes energy overall because the compressor itself has a fixed cost (130-310 milliseconds per query) that outweighs the small savings; real energy savings only kick in from rate 0.7 downward
  4. Rate 0.3 emerges as the 'safe but aggressive' sweet spot, with energy savings growing from 30% to 53% depending on model size and how many documents are retrieved
  5. The authors argue for a runtime controller that watches live device signals like latency, energy, and temperature to decide when and how much to compress, instead of using one fixed compression setting
Figure 1. Telemetry-informed adaptive compression for edge RAG. Retrieved passages are compressed before generation, but the compression ratio is selected at runtime by a controller that observes edge-SoC telemetry and optimizes for latency, energy, thermal, memory, and accuracy constraints.
Figure 1. Telemetry-informed adaptive compression for edge RAG. Retrieved passages are compressed before generation, but the compression ratio is selected at runtime by a controller that observes edge-SoC telemetry and optimizes for latency, energy, thermal, memory, and accuracy constraints.
Table 1. Experimental Setup
ItemConfiguration
PlatformJetson AGX Thor: Blackwell GPU, 128 GB LPDDR5x, 273 GB/s, 130 W.
Corpus / indexEnglish Wikipedia 2018, sentence-split via FlashRAG (∼9.4 M passages) (16); e5-base-v2 encoder (50); FAISS GPU IndexFlatL2 ( 28.2 GB).
QA DatasetsNatural Questions (20) and HotpotQA (54); 100 seed-paired queries per config.
ModelsLlama-3.2 1B/3B, Llama-3.1 8B (9); Qwen-2.5 1.5B/3B/7B (53). All fp16.
CompressionNone, or LLMLingua-2 (34)
SweepsExp. 1 uses k∈{1,5,10} with compression off. Exp. 2 uses HotpotQA, k∈{5,10}, and LLMLingua-2 rates 1.0, 0.9, 0.7, 0.5, 0.3, and 0.15.
ControlsSingle-query mode, reranker off, standard pipeline, randomized config order, first 3 queries dropped as warm-up.
Metrics/telemetryEM, token-level F1, retrieval recall; end-to-end and per-stage latency; GPU/SoC energy, power, memory, and temperature from tegrastats at 100 ms cadence.
Figure 2. Per-query share of latency (left) and GPU energy (right) by stage, on AGX Thor, fp16, no compression.
Figure 2. Per-query share of latency (left) and GPU energy (right) by stage, on AGX Thor, fp16, no compression.
Table 2. Net savings at rate=0.3 vs. the in-session uncompressed baseline (positive Δ = saving on the cost columns, improvement on the quality columns).
ModelkΔlat.Δ​EGPUΔ​ESoCΔEMΔF1
Llama-3B5+8.5%+30.2%+28.2%+0.000−0.005
Llama-3B10+15.6%+40.6%+35.1%+0.010+0.012
Llama-8B5+25.2%+44.9%+38.5%+0.052+0.016
Llama-8B10+32.6%+53.2%+48.2%+0.000−0.011
Figure 3. Answer F1 (left axis) and per-query GPU energy (right axis) vs. LLMLingua-2 rate on HotpotQA. Shaded band: adaptive operating room between the two dotted-line knees.
Figure 3. Answer F1 (left axis) and per-query GPU energy (right axis) vs. LLMLingua-2 rate on HotpotQA. Shaded band: adaptive operating room between the two dotted-line knees.

Why it matters

For developers building RAG on phones, robots, or AR devices with limited battery and heat budgets, this shows that aggressive fixed compression can backfire and identifies a safe operating point (rate 0.3). It reframes compression as a live resource-management decision tied to device state, not a one-time preprocessing choice.

Figure 4. Net GPU energy delta (top) and ΔF1 (bottom) at rate=0.9 (mild) and rate=0.3 (safe-aggressive), vs. the in-session uncompressed baseline.
Figure 4. Net GPU energy delta (top) and ΔF1 (bottom) at rate=0.9 (mild) and rate=0.3 (safe-aggressive), vs. the in-session uncompressed baseline.

Terms in this paper

  • RAG (Retrieval-Augmented Generation) · A method where a language model retrieves outside documents before answering, to ground its response in real information
  • context compression · Trimming or shortening retrieved documents before feeding them to the model, to reduce input length
  • compression rate · The fraction of original text kept after compression; rate=0.3 means only 30% of the text remains
  • LLMLingua-2 · A text compression tool that removes less-important tokens while keeping the rest of the text intact
  • edge SoC · A power- and heat-constrained system-on-chip used in devices like phones or robots, such as the Jetson AGX Thor tested here

Original abstract (English)

Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.

Authors · Zlatan Feric, Amir Taherin, Yanzhi Wang, David Kaeli

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Zlatan Feric et al., arXiv:2608.19535, arxiv-nonexclusive