AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
Vision-Language Grounding as Bidirectional Concept Correspondence
arXiv:2608.078862026-08-07
ConCor-1 makes a model figure out which words in a caption actually point to something in the image, not just localize a given phrase
Traditional vision-language grounding assumes you already know which phrase to look for and just finds where it is in the image. This paper reframes the task as recovering all correspondences between visually referential text spans and image object masks simultaneously, without being told which spans matter, via a model called ConCor-1 built on learnable 'bridge tokens'. The authors report large improvements over baselines on a long-caption benchmark and on zero-shot LVIS category grounding. Author: Jieyu Zhang, arXiv 2608.07886.
METAL LAB explanatory visual
How ConCor-1 finds text-image correspondences
Evidence statusMeasured results reported
InputAn image paired with text of any form: a full caption, a referring expression, or a list of category names.
Shared backbone encodingPretrained Qwen3.5 jointly processes image tokens and text tokens as one sequence.
Bridge tokens insertedLearnable bridge tokens attend to both image and text tokens to form candidate correspondence hypotheses.
Three-way predictionEach bridge token predicts a text mask, an image mask, and a presence score for whether the pairing is valid.
Matching and scoringHungarian matching compares predicted correspondences to ground truth, evaluated with JointF1, mJS, and related metrics.
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
Reframing the problem: instead of localizing a prespecified phrase or category name in an image (the standard setup), the model must recover the full set of correspondences between text spans that refer to something visual and instance-level image masks, without knowing in advance which text spans are relevant.
Architecture: built on a pretrained vision-language backbone (Qwen3.5, 0.8B), ConCor-1 adds learnable 'bridge tokens' that attend jointly to image and text tokens; each bridge token then predicts a text mask, an image mask, and a presence score indicating whether that hypothesized correspondence is actually valid.
Training data pipeline: existing grounding and segmentation datasets (GoldG, COCONut-PanCap, COCO, EntitySeg, etc.) were converted into a unified correspondence format using an LLM rewriting pipeline that merges co-referring mentions into a single text mask and removes mentions with no matching image mask.
Results: on the long-caption COCONut-PanCap benchmark, ConCor-1 improved JointF1 from 59.9 to 88.8 and mJS from 63.3 to 89.5 versus the strongest baseline; on zero-shot LVIS-minival (a large category list as text input), it improved JointF1 from 23.1 to 29.9, a 29% relative gain, while processing the entire category list in a single forward pass unlike baselines that needed chunking.
Internal analysis: probing the attention between bridge tokens and text/vision tokens across language-model layers showed grounding quality improving progressively with depth, with the final trained prediction heads adding a further large boost over attention alone.
Figure 2: Model overview. ConCor-1 builds on a pretrained vision-language backbone and introduces learnable bridge tokens to represent candidate text–image correspondences. Given an image and paired text, vision tokens, language tokens, and bridge tokens are jointly processed by the language model. For each bridge token, the model predicts an image mask with a vision segmentation head, a text mask with a text segmentation head, and a presence score indicating whether the token corresponds to a valid grounded concept.
Table 1: Image-caption results on COCONut-PanCap, GroundedRef, and Flickr30k validation sets.
COCONut-PanCap
GroundedRef
Flickr30k
Method
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
GDINO+SAM
34.1
64.8
39.8
48.0
67.9
44.2
45.0
56.4
50.9
37.8
44.2
39.7
81.6
85.2
86.7
81.9
86.9
81.5
MM-GDINO+SAM
32.2
59.5
37.4
51.1
70.8
47.4
39.8
58.1
44.3
31.9
44.2
33.3
84.6
87.7
89.6
82.6
87.3
82.3
LLMDet+SAM
31.5
52.8
37.9
59.1
75.4
55.7
54.6
69.0
62.8
68.1
77.9
70.1
76.7
79.6
85.4
75.3
78.7
77.1
GPT-5.4 (medium)+SAM
43.8
47.4
88.9
48.6
43.5
82.7
25.5
37.5
68.2
33.9
34.3
70.8
37.3
43.4
86.6
44.7
41.4
87.0
Florence-2
36.7
53.4
53.6
47.9
55.8
56.7
52.6
61.2
79.2
56.4
55.2
77.7
68.7
70.1
92.8
69.4
61.6
88.9
GLaMM
2.5
39.1
3.0
11.9
27.4
8.0
6.1
47.3
10.0
18.4
32.8
16.9
53.1
69.1
59.5
55.4
60.3
58.2
Qwen3.5-FT
59.9
63.4
93.1
63.3
55.6
87.9
51.5
56.3
82.1
49.7
45.8
77.5
70.2
72.6
95.2
69.1
60.3
95.1
ConCor-1 (random init.)
51.8
59.4
67.6
63.2
63.6
77.9
32.4
43.4
56.9
44.6
48.1
70.5
55.0
61.1
72.7
64.4
62.3
83.7
ConCor-1
88.8
91.9
93.4
89.5
87.0
95.7
70.3
76.4
78.7
69.6
68.8
78.7
91.4
92.4
95.1
91.8
88.6
97.4
Figure 3: Grounding emerges progressively across language model layers. We probe bridge-to-text and bridge-to-vision attention across language-model layers and evaluate the induced masks with mSpanIoU, mMaskIoU, and mJS. Grounding quality improves progressively across layers, and the final prediction heads provide a further substantial boost.
Table 2: Image-category results on COCO validation set, LVIS-minival, and EntitySeg validation set.
COCO
LVIS-minival
EntitySeg
Method
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
GDINO+SAM
63.0
66.5
73.3
76.0
71.3
90.7
15.3
22.4
17.7
33.1
47.8
35.7
21.5
30.6
27.0
36.9
45.5
43.3
MM-GDINO+SAM
65.7
68.7
75.6
77.1
72.0
91.6
9.4
18.4
12.4
53.6
75.7
56.8
29.4
36.9
37.0
45.5
51.7
55.7
LLMDet+SAM
62.4
65.9
72.1
77.2
72.2
91.9
4.3
8.7
5.8
53.2
76.0
56.5
22.2
31.7
29.2
44.5
56.2
55.2
GPT-5.4 (medium)+SAM
34.2
35.1
81.7
39.7
32.5
80.0
17.2
20.9
43.0
25.8
26.3
53.7
20.2
30.1
42.4
30.7
34.4
51.9
Florence-2
9.8
14.8
11.9
7.6
10.1
9.1
6.2
17.7
8.2
17.0
38.0
19.6
11.8
20.9
23.5
21.2
28.8
32.1
GLaMM
38.5
40.8
45.7
33.1
31.5
38.9
23.1
24.0
29.7
19.4
18.4
24.1
22.2
23.1
26.0
18.3
17.4
21.2
Qwen3.5-FT
39.4
41.2
51.4
37.3
37.4
48.3
3.9
26.9
5.2
4.4
28.0
5.2
31.1
38.6
47.6
33.1
38.0
43.6
ConCor-1 (random init.)
15.1
32.8
22.9
25.1
49.5
35.6
0.1
18.5
0.2
0.2
32.5
0.3
2.8
20.4
5.4
4.3
26.6
6.5
ConCor-1
72.4
74.3
81.0
79.8
74.2
92.7
29.9
41.4
33.8
39.8
51.4
45.0
49.8
59.9
59.1
48.9
53.7
57.4
Figure 4: Qualitative comparison on COCONut-PanCap. We visualize ground truth, GDINO+SAM, and ConCor-1 predictions, using matching colors for grounded text spans and image masks. ConCor-1 better disambiguates referred instances and recovers more complete text–image correspondences, while GDINO+SAM can confuse different people or objects mentioned in the caption and may miss some grounded entities. Additional examples are provided in Appendix 8.
Table 3: Ablation study. We train each variant for 15K steps and report the average score for each metric across tasks. Ablations are performed sequentially; the vision encoder is frozen for the scoring-function, segmentation-head, and attention ablations, and fine-tuned for the subsequent vision-feature and bridge-assignment ablations. Bidirectional attention is used for the vision-feature and bridge-assignment ablations.
Component
Variation
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
Scoring function
Dot product
48.7
55.4
60.3
59.9
59.7
73.3
Bilinear
49.3
56.0
61.2
59.4
59.0
72.9
Image segmentation head
Bilinear-up (4x4)
47.8
54.3
60.3
59.1
58.0
73.3
Two Conv-up (4x4)
49.3
56.0
61.2
59.4
59.0
72.9
Three Conv-up (2x2)
48.8
56.3
60.1
59.0
59.5
71.6
Four Conv-up (1x1)
48.8
56.4
60.6
58.5
59.3
70.4
Attention
Causal
49.3
56.0
61.2
59.4
59.0
72.9
Bidirectional
51.3
57.2
62.6
61.4
60.4
74.7
Vision feature
Vision encoder feature + VLM vision tokens
52.4
59.4
62.5
62.2
62.7
73.9
Vision encoder feature only
51.8
59.1
61.9
61.7
62.7
73.4
VLM vision tokens only
52.5
59.5
62.8
61.0
61.4
72.8
Bridge token assignment
Raster order (64 tokens)
43.4
53.3
58.4
48.3
51.6
60.6
Spatial assignment (8x8 tokens)
51.7
59.2
62.5
60.8
61.7
72.4
Spatial assignment (11x11 tokens)
51.3
59.1
61.4
61.3
62.5
72.6
Multi-scale spatial assignment
52.4
59.4
62.5
62.2
62.7
73.9
Figure 5: Text-to-image attention visualization. We visualize layer-20 attention from selected text tokens (highlighted in red) to image tokens for the original Qwen3.5, Qwen3.5 fine-tuned on our dataset, and our model. Compared with the diffuse or noisy baselines, ConCor-1 produces more localized and semantically aligned attention. Additional visualizations are provided in Appendix 9.
Table 4: Quantitative text-to-image attention analysis across full-attention layers, averaged over the image-caption evaluation datasets. Attn-IoU measures the overlap between the highest-attention visual tokens and the phrase-specific ground-truth mask, using the ground-truth mask area to determine the number of selected tokens. Top-1 Hit measures whether the visual token receiving the maximum attention lies inside the ground-truth mask. Entropy measures the normalized spatial dispersion of attention, where lower values indicate more concentrated attention.
Metric
Model
L4
L8
L12
L16
L20
Attn-IoU ↑
Qwen3.5
0.16
0.20
0.22
0.22
0.21
Qwen3.5-FT
0.16
0.27
0.23
0.19
0.15
ConCor-1
0.29
0.42
0.48
0.33
0.57
Top-1 Hit ↑
Qwen3.5
0.32
0.43
0.43
0.43
0.40
Qwen3.5-FT
0.41
0.56
0.47
0.36
0.27
ConCor-1
0.56
0.70
0.70
0.55
0.75
Entropy ↓
Qwen3.5
0.82
0.69
0.73
0.74
0.73
Qwen3.5-FT
0.78
0.73
0.71
0.70
0.63
ConCor-1
0.73
0.66
0.65
0.68
0.54
Figure 6: Bridge-token attention and prediction. We visualize two bridge tokens at layer 20 for one test sample, including bridge-to-image attention, bridge-to-text attention, spatial prior (red bounding box), and final prediction. Different bridge tokens specialize to distinct object-level correspondence hypotheses, such as the dog and the seated person. Additional examples are provided in Appendix 10.
Table 5: Effect of the loss weights (λtext,λbce,λdice,λpres).
λtext:λbce:λdice:λpres
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
1 : 0.5 : 0.5 : 1
60.5
66.8
70.7
66.8
66.5
78.4
1 : 2 : 0.5 : 1
61.3
67.2
71.3
67.3
66.8
78.9
1 : 5 : 5 : 1
59.9
66.1
69.9
66.0
66.0
77.4
1 : 2.5 : 0 : 2
59.7
65.5
70.8
66.1
65.6
79.3
1 : 2 : 0.5 : 2
62.0
67.7
72.1
68.0
67.3
79.5
Figure 7: Spatial specialization of bridge tokens across token groups. We visualize the spatial activation patterns of bridge tokens from different scale groups under our multi-scale bridge-token design. Each group corresponds to a spatial partition of the image, ranging from a global 1×1 group to increasingly fine grids. Bridge tokens in coarse groups respond to broad image regions, while tokens in finer groups activate on increasingly localized areas aligned with their assigned spatial cells. This shows that bridge tokens learn structured spatial priors and progressively cover the image at multiple resolutions, supporting correspondence prediction for objects of different sizes and locations.
Table 6: Model size and computational profile. FLOPs, peak GPU memory, and latency are measured for one forward pass with a 512×512 image, 512 text tokens, batch size 1, bf16 precision, and an A100-80GB GPU.
Model
Seq. Len.
FLOPs
Peak Mem.
Latency
Params
Qwen3.5-0.8B backbone
770
1.061 T
1.687 GiB
54.0 ms
852.99 M
ConCor-1
1,155
1.549 T
1.846 GiB
55.4 ms
866.79 M
Δ
+385
+46.0%
+9.5%
+2.6%
+1.62%
Figure 8: Qualitative visualization examples.
Table 7: Component-level parameter and FLOPs breakdown. The computation induced by the bridge tokens is included in the language-model row.
Component
Added Params
GFLOPs
(M)
Backbone
ConCor-1
𝚫
Vision tower
–
221.5
221.5
–
Language model
–
839.3
1,279.2
+439.9
New bridge embeddings
0.145
–
–
–
Visual segmentation head
12.01
–
46.3
+46.3
Text segmentation head
1.11
–
1.1
+1.1
Presence head
0.52
–
0.4
+0.4
Total
13.80
1,060.8
1,548.5
+487.7
Figure 10: Qualitative visualization examples.
Table 8: Training data mixture used for ConCor-1. We report the source-level mixture weight in the final training setup. The image augmentation column indicates the fraction of each source group sampled with collage or crop augmentation; the remaining fraction is sampled without image augmentation.
Source dataset
Mix weight
Image aug.
Instance Segmentation Data
COCO 24
15%
80% collage
EntitySeg 34
10%
10% crop
PixMo Points 6
10%
10% crop
COCONut 7
5%
70% collage, 10% crop
SA-1B 18
5%
10% crop
ADE20K 57
5%
50% collage, 20% crop
Roboflow-VL-100 37
5%
—
Caption Grounding Data
GoldG 25
18%
—
COCONut-PanCap 7
15%
—
GroundedRef
12%
—
Total
100%
—
Table 9: Statistics of training data. Image counts are unique images per source and the total is their sum, so COCO images shared by COCO, COCONut-PanCap and GroundedRef are counted once per source.
Source dataset
Num. of images
Num. of instances
Annotation Type
Role
Instance Segmentation Data
COCO 24
117.3K
860.0K
Human annotated
Standard benchmark
COCONut (Objects365 subset) 8
242.6K
3.6M
Human annotated
Standard benchmark
EntitySeg 34
8.1K
79.1K
Human annotated
Standard benchmark
ADE20K 57
25.6K
639.5K
Human annotated
Scene-centric objects
Roboflow-VL-100 (81 datasets) 37
92.9K
812.3K
Converted benchmark data
Diverse domains
PixMo Points 6
56.8K
1.7M
Auto-constructed
Dense scenes + diverse text queries
SA-1B 18
71.1K
249.5K
Auto-constructed
Dense scenes + diverse text queries
Caption Grounding Data
GoldG 25
75.2K
2.6M
Rewritten + pseudo masks
Human-caption grounding
COCONut-PanCap 7
100.1K
1.3M
Rewritten + GT masks
Human-caption grounding
GroundedRef
60.2K
230.5K
Synthetic captions + GT masks
Referring and compositional grounding
Total
849.8K
12.0M
–
–
Table 10: Statistics of the three image-caption benchmarks we curated or repurposed. Caption length is measured by whitespace-tokenized words per caption. Span multiplicity denotes the number of distinct character spans associated with each mask instance.
Dataset
Images
Captions
Masks
Caption length
Span multiplicity
words/caption
spans/mask
Flickr30k
779
2,002
6,717
11.01 ± 4.36
1.06 ± 0.26
GroundedRef
1,318
2,025
6,193
10.34 ± 4.20
1.04 ± 0.19
COCONut-PanCap
2,213
2,213
10,821
50.84 ± 16.89
1.68 ± 0.90
Table 11: Effects of annotation standardization and model architecture. Image-caption JointF1 is averaged over the three caption benchmarks, while image-category JointF1 is averaged over COCO and LVIS-minival.
Training annotations
Method
Image-caption
Image-category
JointF1
JointF1
Original source
MM-GDINO-FT + SAM
57.0
33.3
Original source
ConCor-1
60.8
44.9
Standardized
MM-GDINO-FT + SAM
66.4
33.6
Standardized
ConCor-1
70.6
45.2
Findings
On COCONut-PanCap (long-caption benchmark), ConCor-1 improved JointF1 from 59.9 to 88.8 and mJS from 63.3 to 89.5 compared with the strongest baseline.
On zero-shot LVIS-minival, ConCor-1 improved JointF1 from 23.1 to 29.9 (29% relative gain) over the strongest baseline, and processed the full LVIS category vocabulary in a single forward pass while other baselines required chunking.
Qwen3.5-FT, which generates text spans and polygon masks autoregressively, scored well on text-only metrics but lagged behind ConCor-1 on mask and joint correspondence metrics.
Layer-wise attention probing showed early language-model layers carried little usable grounding signal, with mMaskIoU, mSpanIoU, and mJS improving progressively in deeper layers, and the final trained heads providing a substantial further boost over attention alone.
Standardizing annotations (via the LLM rewriting pipeline) improved image-caption JointF1 by 9.4 points for MM-GDINO-FT and by 9.8 points for ConCor-1.
Where it can be used
Automatically checking how faithfully an image caption actually describes the objects present in an image
Verifying whether a text-to-image generation model actually rendered each element mentioned in its prompt
Producing object-level text-to-image alignments for long, complex descriptions with multiple referring expressions
Open-vocabulary detection or segmentation pipelines that need to handle large category lists without running multiple chunked inference passes
Limits and open work
Results are reported only on specific benchmarks (COCONut-PanCap, GroundedRef, Flickr30k for captions; COCO, LVIS-minival, EntitySeg for categories), so generalization to other domains is untested.
The backbone used is a relatively small 0.8B Qwen3.5 model; behavior with larger backbones or different VLM families is not reported.
The number of bridge tokens is fixed, and the paper does not report how the model behaves when an image-text pair contains far more correspondences than the token budget allows.
The annotation-standardization ablation was run only with 15K training steps, a reduced setting, so its effect at full training scale is not separately reported.
Extensions like Winograd-style referential ambiguity resolution or visual coreference resolution are discussed conceptually but not evaluated with quantitative experiments in this paper.
Why it matters
Because the model doesn't need to be told which phrase to localize, it can directly check whether a caption or a text-to-image generation actually corresponds to what's in the picture, which is useful for automatically auditing caption faithfulness or generation quality. It also handles large open-vocabulary category lists in one pass instead of requiring multiple chunked inference calls, which matters for latency and cost in real detection pipelines.
Terms in this paper
grounding · linking a piece of text to the specific part of an image it refers to
bridge token · a learnable token trained to represent one candidate correspondence between a text span and an image region
JointF1 / mJS · metrics that count a prediction correct only if both the text span and image mask match the ground truth well enough (thresholded F1 and a softer geometric-mean score)
open-vocabulary detection · finding objects in an image using arbitrary category names not fixed in advance
presence score · a predicted value indicating whether a bridge token's hypothesized text-image pairing is actually a valid correspondence
Figures we cannot republish
Figure 1: Bidirectional concept correspondence. Given an image paired with different forms of text, including a full caption, a referring expression, or a list of category names, the concept correspondence model predicts a complete set of correspondences between text segments and image segments. For each correspondence, the model identifies the visually grounded text span, including repeated or coreferential mentions, and localizes the corresponding instance-level mask in the image.
Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as bidirectional concept correspondence over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce ConCor-1, a grounding model built on top of a pretrained vision-language model. It uses learnable bridge tokens to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that ConCor-1 consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.
