Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

What they did

1. Unlike prior work that added machinery such as segmentation masks, edge maps, or LLM-filtered captions, this paper targets a flaw in the training objective itself, the loss function that tells the model how wrong it is and which way to adjust.
2. On datasets with long, detail-rich captions, many captions are near-duplicates of each other; within the first training epoch, 80% of batches already had loss near zero, and in 47% of measurements the gradient (the signal telling the model how to update) was exactly zero, meaning the model stopped learning from hard cases early.
3. Their fix, HN-CLIP, computes a text-to-text similarity matrix between captions in the batch, detaches it so it doesn't get updated by training, and adds it to the score gap between correct and incorrect matches, so more similar (harder) negative captions require a bigger score margin before the loss goes quiet.
4. The method adds no extra data, no new model components, and no inference-time cost, just one matrix multiplication and addition during training, yet it achieved the best top-1 retrieval accuracy in all eight tested retrieval directions across four benchmarks, and using only 20% of training data it still beat the strongest prior method trained on 100% of the data.
5. Simply swapping this loss into six different existing fine-tuning frameworks (Long-CLIP, FineLIP, GOAL, StructXLIP, LoRA, DoRA) improved in-domain performance in every case, showing the fix is broadly applicable rather than tied to one specific architecture.

Why it matters

Boosting image-text search accuracy just by fixing the training objective, without added preprocessing or model complexity, offers a cheap and fast way to improve real-world retrieval systems. The finding that a seemingly converged loss can hide a model that has stopped learning the hardest, most important cases is also a useful diagnostic for other contrastive learning projects.

Terms in this paper

• InfoNCE · a contrastive loss function that pulls matching pairs together and pushes non-matching pairs apart
• near-duplicate captions · descriptions that are so similar in content that they're hard to tell apart
• gradient · the signal that tells a model's parameters which direction and how much to update
• detach / stop-gradient · a technique that uses a computed value as a reference without letting it be updated by backpropagation
• R@1 (Recall@1) · the percentage of queries where the correct match was ranked first

Figures we cannot republish

• Figure 1: Dense-caption benchmarks are dominated by hard negatives. Distributions of (a) all pairwise caption-caption cosine similarities and (b) each caption’s hardest-negative similarity, measured with the pre-trained Long-CLIP-L text encoder on the test sets. Dotted lines mark the means. The pairs in (b) receive the strongest boosts.
• Figure 2: Overview of HN-CLIP. For a batch of B image–text pairs, HN-CLIP encodes images and captions with the dual encoder being fine-tuned, and forms the image–text similarity matrix S=v​t⊤ and a detached, diagonal-masked caption-similarity matrix G¯ from G=t​t⊤. Their combination yields boosted logits s⁡(S+γ​G¯), so near-twin negatives receive larger margins. The token-level term of Eq. 3 acts alongside this objective, improving supervision on hard negatives during training while keeping inference unchanged.
• Figure 3: Illustration of HN-CLIP. A real DOCCI query and its hardest in-batch negative (cos 0.89). The strongest baseline ranks the ground truth 52nd (its top-1 (pink) is the near-twin’s own image); HN-CLIP ranks it first.
• Figure 4: Empirical gradient analysis (Long-CLIP-L, DOCCI, 10 epochs; losses and full-parameter gradients measured every 5 optimizer steps, 148 measurements). Standard InfoNCE declares 80% of batches solved within the first epoch and its gradient is exactly zero in fp32 in 47% of measurements (median zero at five epochs; plotted clamped to 10−10). The boosted loss keeps at least 26% of batches active in every epoch and, where the standard gradient is nonzero, exceeds it by 102–106× in per-epoch median, with interquartile bands disjoint at nine epochs, and the retrieval error it buys keeps falling.
• Figure 5: Convergence comparison. Average R@1 (T→I, I→T) per fine-tuning epoch. HN-CLIP’s first epoch matches or exceeds most baselines’ final accuracy on all four benchmarks.
• Figure 6: Sample efficiency (avg R@1, identical subsets). HN-CLIP at 20% of the data already clears the strongest baseline trained on 100% (dotted line, from Table 1); full numbers in Table A9.
• Figure A1: Near-duplicate caption pairs on DOCCI. Each row: a test image, its caption, the three nearest other captions’ images (cosine printed underneath), and the caption texts with shared content words highlighted.
• Figure A2: Near-duplicate caption pairs on DCI.
• Figure A3: Near-duplicate caption pairs on Long-DCI (full-length captions).
• Figure A4: Near-duplicate caption pairs on Urban-1K.
• Figure A5: One real training batch per benchmark. Text–text similarity matrices G¯ (pre-trained encoder; diagonal masked, max off-diagonal entry annotated). Uniformly dark = every negative is hard; this matrix, detached and scaled by γ, is the entire mechanism of HN-CLIP.
• Figure A6: Statistical analysis of the caption geometry across the four benchmarks (pre-trained Long-CLIP-L text encoder, full test splits). (A) Mean pairwise caption similarity. (B) Mean similarity of each caption’s hardest companion. (C) Composition of in-batch negatives by hardness bucket.
• Figure A8: Gradient dynamics across data scales. Rows: fraction of the training set; columns: dataset. Per cell: (a) raw loss traces (thin) with running medians (thick), (b) gradient norms and their ratio (dashed), (c) gradient cosine. Log axes in (a,b); values clamped at 10−10 (exact zeros at fp32).
• Figure A9: Per-direction convergence. Recall@1 per epoch; top: Text→Image, bottom: Image→Text.
• Figure A10: Qualitative T→I retrieval on DOCCI (green = ground truth).

Latest papers

• FM-Bench: A Benchmark for Long-Horizon Management with Competing AgentsHanding an AI a football club to run for 20 years reveals that winning comes from management habits, not raw model powerarXiv:2608.184232026-08-20
• FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI SystemsAI often names the right cause of a financial mismatch without ever finding the proof for itarXiv:2608.185342026-08-20
• FACET: Preserving Source Intent and Executable State in Terminal Task SynthesisFACET builds internally consistent terminal-task 'exam sets' to train command-line AI agentsarXiv:2608.185802026-08-20
• Alignment Is All You Need: Instruction-Free Training for General Audio-Language ModelsA frozen language model plus one lightweight connector is enough to build a capable audio-understanding AIarXiv:2608.181322026-08-20
• Adversarial Review: Structured Disagreement for Grounded Agentic Code ReviewFor AI code review, one reviewer plus one critic beats piling on more agentsarXiv:2608.181672026-08-20
• Looped Language Models Improve Compositional Tool CallingAI models that rethink their own answers multiple times get better at chaining tools togetherarXiv:2608.181712026-08-20
• Bridging Search and CRM: Productionizing AI Product Research Agents for Customer Re-EngagementAI agents win back window-shopping customers by chasing them down on WhatsApparXiv:2608.185432026-08-20
• GenEx: A Graph-Based Representational Paradigm for SARS-CoV-2 Variant Detection via Codon Co-occurrence NetworksTurning viral gene sequences into codon relationship maps to tell coronavirus variants apartarXiv:2608.182382026-08-20

Latest from METAL LAB

• NVIDIA's 300 Verified Skills Lift Correctness by 41 Points
• Wave your hand at a webcam, hear a theremin: browser instrument released
• Meta AI launches desktop app for Mac, can read an entire app window
• Factory Commits $100M to Partner Network, Pushes to Scale Software Factories
• SpaceX approached Cognition for acquisition four days after closing Cursor deal