AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
arXiv:2608.045692026-08-04
Prompt compressors that shrink long documents often keep the answer sentence but delete the fact needed to understand it
Hard prompt compressors shorten long context by scoring sentences or chunks independently and keeping only the top-scoring ones under a budget, but this can leave an answer sentence in place while deleting the earlier definition or bridge fact needed to interpret it. The paper names this failure 'referential dangling,' measures how often it occurs across compressors and datasets, and tests whether restoring the missing content or training a small classifier to auto-restore it recovers accuracy. At compression ratio 0.30 the failure appeared in up to 60% of tested cases, and restoring the missing paragraph raised accuracy by up to 34 percentage points.
METAL LAB explanatory visual
How referential dangling arises and gets repaired
Evidence statusMeasured results reported
1. Independent scoring compressionThe compressor scores sentences or chunks separately and keeps only top-scoring ones under budget, which can split an answer sentence from its defining context
2. Dangling diagnosisMeasured across six compressors, three multi-hop QA datasets, and LongBench-v2 documents; dangling rates ranged from 32% to 60%
3. Fixed-budget reselectionReinserting the missing supporting paragraph while removing non-supporting paragraphs of equal token count, keeping the budget fixed, and measuring accuracy change
4. Automatic restoration classifierA small trained classifier ranks omitted sentences by whether they're needed for interpretation and restores top candidates at inference without support annotations
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
Using the Beaver compressor (Qwen3-0.6B embeddings) at compression ratio 0.30, 34-54% of bridge examples across HotpotQA, 2WikiMultiHopQA, and MuSiQue retained an answer but lost the definition paragraph needed to interpret it.
On a shared set of 184 HotpotQA bridge examples, all six tested compressors (using embedding similarity, self-information, a trained token classifier, syntactic parsing, perplexity, and attention-derived signals) showed the failure, at rates from 32% to 60%, regardless of whether the compressor used the query.
Every one of 80 documents in LongBench-v2 Single-Document QA had at least one dangling reference, with rates of 25-37% consistent across all seven subdomains.
Reinserting the missing supporting paragraph while removing an equal number of tokens from non-supporting paragraphs (keeping the budget fixed) improved Qwen3-8B accuracy by 29-34 percentage points (p<0.0001), recovering at least 88% of the gap to contexts with both supporting paragraphs; on MuSiQue, even GPT-5.5 was 8.8 points less accurate on compressed contexts than on fully supported ones, showing stronger models don't fix this on their own.
A compact bert-base-uncased classifier trained to rank omitted sentences by whether they're needed to interpret retained text, and used to auto-restore top candidates without support annotations at inference, improved HotpotQA accuracy by 4.7 points with Qwen3-8B while the compression ratio changed only from 0.30 to 0.31.
Figure 1: Referential dangling with a missing bridge. Independent scoring retains the query subject and the answer string but removes the fact that Tim DuBois was born in Southwest City, leaving the inference chain incomplete.
Table 1: Referential dangling and complete evidence retention under Beaver at r=0.30.
Dataset
Hops
ρd (%)
ρe (%)
HotpotQA
2
34.2
61.0
2WikiMultiHopQA
2
53.5
30.7
MuSiQue
2 to 4
54.2
27.0
Figure 2: Referential dangling under Beaver. Panel (a) reports ρd across compression ratios on HotpotQA (n=269 to 300 per point, including partial paragraph retention). Panel (b) reports ρd by annotated hop count on HotpotQA (n=234), 2WikiMultiHopQA (n=241), and MuSiQue (n=286). Panel (c) reports dangling rates for 4,649 reference pairs in LongBench-v2 Single-Document QA by sentence distance from first mention to later reference. Error bars are bootstrap 95% confidence intervals.
Table 2: Pairwise Jaccard similarities between dangling case sets on the shared HotpotQA bridge set (n=184) at compression ratio 0.30. The first row reports the dangling rate of each compressor. Abbreviations match Figure 3.
BEAVER
PartPr.
Sel.-Ctx
LLML-2
DAC
LongLL
Dangling rate (%)
32.1
47.8
51.6
56.0
58.7
59.8
BEAVER
N/A
0.36
0.23
0.30
0.29
0.32
PartPr.
0.36
N/A
0.36
0.44
0.44
0.37
Sel.-Ctx
0.23
0.36
N/A
0.39
0.35
0.51
LLML-2
0.30
0.44
0.39
N/A
0.47
0.45
DAC
0.29
0.44
0.35
0.47
N/A
0.48
LongLL
0.32
0.37
0.51
0.45
0.48
N/A
Figure 3: Dangling rates for six compressors on the shared HotpotQA bridge set (n=184) at compression ratio 0.30. PartPr. denotes PartPrompt, Sel.-Ctx denotes Selective-Context, LLML-2 denotes LLMLingua-2, and LongLL denotes LongLLMLingua. All outputs are evaluated using the content-word overlap criterion with threshold 0.5. Light bars denote methods that use the query, and darker bars denote methods that do not.
Table 3: Answer accuracy with base contexts produced by Beaver at target compression ratio 0.30. Panel 1 uses the dangling subsets of HotpotQA (n=80), 2WikiMultiHopQA (n=72), and MuSiQue (n=102), with McNemar p values comparing Base and Reselected. Panel 2 uses a separate set of 200 HotpotQA examples, with McNemar p values comparing Base and Full support.
Panel 1: dangling subsets evaluated with Qwen3-8B
Dataset
Downstream LLM
Base
Reselected
Full support
McNemar p
HotpotQA
Qwen3-8B
0.287
0.575
0.600
1.6×10−6
2WikiMultiHopQA
Qwen3-8B
0.097
0.403
0.444
1.1×10−5
MuSiQue
Qwen3-8B
0.147
0.490
0.471
3.1×10−8
Panel 2: a 200 example HotpotQA evaluation set with four downstream LLMs
Dataset
Downstream LLM
Base
Full support
McNemar p
HotpotQA
Qwen3-8B
0.535
0.615
0.001
Qwen3-4B
0.500
0.585
0.002
Llama-3.1-8B
0.575
0.660
0.004
Mistral-7B
0.455
0.545
0.0005
Figure 4: Mean salience percentiles (%) for answer and definition sentences among all sentences in 180 bridge examples. Beaver similarity is query-aware; self-information is not.
Table 4: Answer accuracy of proprietary models under Base and Full support, with base contexts produced by Beaver at target compression ratio 0.30. HotpotQA uses the full shared bridge set, while MuSiQue uses the dangling subset. GLM-5.2 returned answers for 95 of the 102 MuSiQue contexts because of API timeouts.
Model
Dataset
n
Base
Full support
McNemar p
GPT-5.5
HotpotQA
184
0.913
0.913
1.0
GPT-5.5
MuSiQue
102
0.775
0.863
0.011
GLM-5.2
MuSiQue
95
0.695
0.937
2.4×10−7
Figure 5: Dangling rate across content-word-overlap thresholds for 184 bridge examples (Figure 3; ratio 0.30).
Table 5: Changes in answer accuracy, in percentage points relative to Base, for candidate sources with a fixed classifier and Qwen3-8B (K=3). Hybrid augments first-mention candidates with embedding retrieval, and the final row includes the annotated supporting sentence in the candidate set.
Candidate source
HotpotQA
2WikiMultiHopQA
First mention
+4.7 (p=0.022)
+0.5 (not significant)
All mentions
+4.5 (p=0.15)
+4.0 (p=0.20)
Hybrid
+4.5 (p=0.12)
+5.5 (p=0.063)
Annotated support included
+8.0 (p=0.008)
N/A
Figure 6: Referential dangling examples from HotpotQA, 2WikiMultiHopQA, MuSiQue, and LongBench-v2 Single-Document QA. Each panel shows the original context and the compressed output.
Table 6: Dangling rate (%) across content-word overlap retention thresholds on the same 184 bridge examples as Figure 3 at compression ratio 0.30. The 0.5 column matches Figure 3.
Overlap threshold
0.3
0.4
0.5
0.6
0.7
LLMLingua-2 (token)
28.3
43.5
56.0
57.6
36.4
Beaver (chunk)
19.6
25.0
32.1
36.4
40.2
Figure 7: Accuracy gains from full support and first-mention automatic restoration on HotpotQA (Beaver at ratio 0.30, K=3). Full support uses 200 examples; restoration uses 300 for Qwen3-8B and Llama-3.1-8B and 200 for Mistral-7B.
Table 7: Robustness of the dangling diagnostic to its three main free choices (HotpotQA, n=300, ratio 0.30 unless swept). Embedding shifts are measured in percentage points relative to the released Qwen3-0.6B embedding setup.
Check
Variation
Outcome
Embedding scorer
Qwen3-0.6B embeddings → GPT-2
+0.9 points
Overlap threshold
0.3 to 0.7
substantial throughout
Compression ratio
0.70 to 0.20
monotonic increase
Table 8: Referential dangling on LongBench-v2 Single-Document QA by subdomain (Beaver, ratio 0.30, n=80 documents). “Mean rate” is the per-document average fraction of retained sentences that are dangling, macro-averaged over documents. “Affected docs” is the fraction of documents with at least one dangling reference.
Subdomain
n
Mean rate
Affected docs
Academic
13
36.5%
100%
Literary
12
34.4%
100%
Financial
12
32.3%
100%
Legal
8
29.1%
100%
Detective
15
27.3%
100%
Event ordering
11
27.0%
100%
Governmental
9
25.1%
100%
All
80
30.5%
𝟏𝟎𝟎%
Table 9: Official checkpoint and API identifiers. Display names are the shorthand used in the paper; exact identifiers are shown for reproducibility.
Table 10: Automatic restoration results with the classifier fixed at K=3. The evaluation uses 300 HotpotQA examples, except for Mistral-7B, which uses 200. Base is Beaver at compression ratio 0.30, and Restored has an average ratio of 0.31. The reported p values use paired McNemar tests.
Downstream LLM
Candidate source
Base
Restored
p
Qwen3-8B
First mention
0.567
0.613
0.022
Mistral-7B
First mention
0.455
0.520
0.012
Llama-3.1-8B
First mention
0.587
0.600
0.60
Llama-3.1-8B
Hybrid
0.587
0.610
0.17
Table 11: Restoration statistics when the Beaver baseline was incorrect (HotpotQA, n=300, compression ratio 0.30; downstream Qwen3-8B). SD denotes standard deviation.
Feature
Fixed (23)
Failed (107)
Sentences added, mean ± SD
2.13 ± 1.08
1.79 ± 1.17
Sentences added, median
3.0
2.0
McNemar: 23 fixes, 9 breaks, p=0.022
Table 12: Matched addition control on HotpotQA with Qwen3-8B (n=300). Random insertion and targeted restoration add the same number of sentences per example (m: mean 1.81, median 2, interquartile range [1,3]; approximately 40 tokens; compression ratio 0.30 to 0.31; K=3). Brackets report bootstrap 95% confidence intervals.
Condition
Accuracy [95% CI]
Δ
Base compressor
0.567 [.51,.62]
N/A
Random insertion, m sentences
0.587 [.53,.64]
+2.0
Targeted restoration, m sentences
0.613 [.55,.67]
+4.7
Table 13: Transfer of one restoration configuration across four compressor outputs on HotpotQA with downstream Qwen3-8B (n≈150 to 300).
Compressor (output type)
Gain
McNemar p
Beaver (coherent chunks)
+4.7
p=0.022
PartPrompt (parse tree spans)
+3.2
0.30
Selective-Context (self-information)
+1.0
0.80
LLMLingua-2 (token fragments)
+1.0
0.75
Findings
Under Beaver at ratio 0.30, the answer-path incompleteness rate (rho_d) ranged from 34.2% to 54.2% across three multi-hop QA datasets.
Across six compressors on the same 184 HotpotQA bridge examples, dangling rates ranged from 32.1% (Beaver) to 59.8% (LongLLMLingua), independent of query use.
All 80 documents in LongBench-v2 Single-Document QA contained at least one dangling reference.
Reselecting content (reinserting the missing supporting paragraph, removing non-supporting paragraphs) improved accuracy by 28.8-34.3 points over the base compressed context (p<0.0001), recovering 88-92% of the gap to full-support contexts.
The automatic restoration classifier improved Qwen3-8B accuracy on HotpotQA by 4.7 points (p=0.022) while the compression ratio changed only from 0.30 to 0.31.
Where it can be used
Teams using prompt compression for multi-hop or evidence-fragmented QA tasks can adopt this dangling diagnostic to check whether supporting content survives compression.
A lightweight classifier-based post-processing step could be added to existing compression pipelines to restore a small number of missing supporting sentences.
When designing or benchmarking compression methods, referential completeness metrics could be reported alongside relevance scores.
Limits and open work
Most measurements center on Beaver, and the automatic restoration classifier is tuned to Beaver's paragraph/sentence-block structure; transfer to other compressor outputs did not reach statistical significance.
In the matched-addition control, random sentence insertion improved accuracy by 2.0 points versus 4.7 points for targeted restoration, but the difference was not statistically significant at this sample size (p=0.15).
The dangling diagnostic only covers dependencies stated explicitly in the source text, excluding commonsense or externally required inferences.
The LongBench-v2 diagnostic uses a different first-mention method lacking support annotations, so its rate is not directly comparable to rho_d on the other datasets.
The study is a single-author paper limited to six compressors, three to four datasets, and one hardware setup (A100 80GB); generalization to other embeddings, ratios, or downstream models remains untested.
Why it matters
Prompt compression is becoming a standard way to cut long-context inference costs, but this work shows that ranking fragments by relevance alone is not enough to guarantee a usable compressed prompt. Anyone building or deploying a compression pipeline gets a concrete design criterion: score for referential completeness, not just relevance.
Terms in this paper
hard prompt compression · Compression that scores tokens, sentences, or chunks and keeps only the highest-scoring ones verbatim under a token budget
referential dangling · When retained text contains an answer but the deleted text held the definition or bridge fact needed to interpret it, breaking the inference chain
bridge example · A multi-hop QA question that requires passing through an intermediate entity to reach the answer
compression ratio (r) · The fraction of original tokens kept after compression
McNemar test · A statistical test comparing correct/incorrect answer changes for the same examples under two conditions
Original abstract (English)
Hard prompt compression reduces long-context inference cost by independently scoring tokens, sentences, or chunks and retaining the highest-scoring units under a budget. We identify a structural failure in this procedure: independent selection can split dependent evidence pairs, retaining one member while deleting the other. When retained text contains an answer but deleted text defines the entity needed to interpret it, we call the result referential dangling. At a compression ratio of 0.30, Beaver, which ranks coherent chunks using Qwen3-0.6B embeddings, leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop question answering datasets. On a shared HotpotQA bridge set, all six hard compressors we test exhibit dangling at rates up to 60%, and every document in LongBench-v2 Single-Document QA contains at least one dangling reference. On dangling examples evaluated with Qwen3-8B, reinserting the missing supporting paragraph while removing nonsupporting paragraphs to maintain the token budget improves accuracy by 29-34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts retaining both supporting paragraphs. Stronger answer models do not absorb the loss: on MuSiQue, GPT-5.5 is 8.8 points less accurate on compressed contexts than on contexts retaining both supporting paragraphs. Finally, we train a compact classifier to rank omitted sentences by whether they are needed to interpret retained text and reinsert the top-ranked candidates without support annotations at inference. On HotpotQA with Qwen3-8B, this automatic restoration improves accuracy by 4.7 points while changing the compression ratio only from 0.30 to 0.31. Hard compressors should optimize both relevance and referential completeness.
