Figure 2: Model overview. ConCor-1 builds on a pretrained vision-language backbone and introduces learnable bridge tokens to represent candidate text–image correspondences. Given an image and paired text, vision tokens, language tokens, and bridge tokens are jointly processed by the language model. For each bridge token, the model predicts an image mask with a vision segmentation head, a text mask with a text segmentation head, and a presence score indicating whether the token corresponds to a valid grounded concept.
Table 1: Image-caption results on COCONut-PanCap, GroundedRef, and Flickr30k validation sets.
COCONut-PanCap
GroundedRef
Flickr30k
Method
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
GDINO+SAM
34.1
64.8
39.8
48.0
67.9
44.2
45.0
56.4
50.9
37.8
44.2
39.7
81.6
85.2
86.7
81.9
86.9
81.5
MM-GDINO+SAM
32.2
59.5
37.4
51.1
70.8
47.4
39.8
58.1
44.3
31.9
44.2
33.3
84.6
87.7
89.6
82.6
87.3
82.3
LLMDet+SAM
31.5
52.8
37.9
59.1
75.4
55.7
54.6
69.0
62.8
68.1
77.9
70.1
76.7
79.6
85.4
75.3
78.7
77.1
GPT-5.4 (medium)+SAM
43.8
47.4
88.9
48.6
43.5
82.7
25.5
37.5
68.2
33.9
34.3
70.8
37.3
43.4
86.6
44.7
41.4
87.0
Florence-2
36.7
53.4
53.6
47.9
55.8
56.7
52.6
61.2
79.2
56.4
55.2
77.7
68.7
70.1
92.8
69.4
61.6
88.9
GLaMM
2.5
39.1
3.0
11.9
27.4
8.0
6.1
47.3
10.0
18.4
32.8
16.9
53.1
69.1
59.5
55.4
60.3
58.2
Qwen3.5-FT
59.9
63.4
93.1
63.3
55.6
87.9
51.5
56.3
82.1
49.7
45.8
77.5
70.2
72.6
95.2
69.1
60.3
95.1
ConCor-1 (random init.)
51.8
59.4
67.6
63.2
63.6
77.9
32.4
43.4
56.9
44.6
48.1
70.5
55.0
61.1
72.7
64.4
62.3
83.7
ConCor-1
88.8
91.9
93.4
89.5
87.0
95.7
70.3
76.4
78.7
69.6
68.8
78.7
91.4
92.4
95.1
91.8
88.6
97.4
Figure 3: Grounding emerges progressively across language model layers. We probe bridge-to-text and bridge-to-vision attention across language-model layers and evaluate the induced masks with mSpanIoU, mMaskIoU, and mJS. Grounding quality improves progressively across layers, and the final prediction heads provide a further substantial boost.
Table 2: Image-category results on COCO validation set, LVIS-minival, and EntitySeg validation set.
COCO
LVIS-minival
EntitySeg
Method
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
GDINO+SAM
63.0
66.5
73.3
76.0
71.3
90.7
15.3
22.4
17.7
33.1
47.8
35.7
21.5
30.6
27.0
36.9
45.5
43.3
MM-GDINO+SAM
65.7
68.7
75.6
77.1
72.0
91.6
9.4
18.4
12.4
53.6
75.7
56.8
29.4
36.9
37.0
45.5
51.7
55.7
LLMDet+SAM
62.4
65.9
72.1
77.2
72.2
91.9
4.3
8.7
5.8
53.2
76.0
56.5
22.2
31.7
29.2
44.5
56.2
55.2
GPT-5.4 (medium)+SAM
34.2
35.1
81.7
39.7
32.5
80.0
17.2
20.9
43.0
25.8
26.3
53.7
20.2
30.1
42.4
30.7
34.4
51.9
Florence-2
9.8
14.8
11.9
7.6
10.1
9.1
6.2
17.7
8.2
17.0
38.0
19.6
11.8
20.9
23.5
21.2
28.8
32.1
GLaMM
38.5
40.8
45.7
33.1
31.5
38.9
23.1
24.0
29.7
19.4
18.4
24.1
22.2
23.1
26.0
18.3
17.4
21.2
Qwen3.5-FT
39.4
41.2
51.4
37.3
37.4
48.3
3.9
26.9
5.2
4.4
28.0
5.2
31.1
38.6
47.6
33.1
38.0
43.6
ConCor-1 (random init.)
15.1
32.8
22.9
25.1
49.5
35.6
0.1
18.5
0.2
0.2
32.5
0.3
2.8
20.4
5.4
4.3
26.6
6.5
ConCor-1
72.4
74.3
81.0
79.8
74.2
92.7
29.9
41.4
33.8
39.8
51.4
45.0
49.8
59.9
59.1
48.9
53.7
57.4
Figure 4: Qualitative comparison on COCONut-PanCap. We visualize ground truth, GDINO+SAM, and ConCor-1 predictions, using matching colors for grounded text spans and image masks. ConCor-1 better disambiguates referred instances and recovers more complete text–image correspondences, while GDINO+SAM can confuse different people or objects mentioned in the caption and may miss some grounded entities. Additional examples are provided in Appendix 8.
Table 3: Ablation study. We train each variant for 15K steps and report the average score for each metric across tasks. Ablations are performed sequentially; the vision encoder is frozen for the scoring-function, segmentation-head, and attention ablations, and fine-tuned for the subsequent vision-feature and bridge-assignment ablations. Bidirectional attention is used for the vision-feature and bridge-assignment ablations.
Component
Variation
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
Scoring function
Dot product
48.7
55.4
60.3
59.9
59.7
73.3
Bilinear
49.3
56.0
61.2
59.4
59.0
72.9
Image segmentation head
Bilinear-up (4x4)
47.8
54.3
60.3
59.1
58.0
73.3
Two Conv-up (4x4)
49.3
56.0
61.2
59.4
59.0
72.9
Three Conv-up (2x2)
48.8
56.3
60.1
59.0
59.5
71.6
Four Conv-up (1x1)
48.8
56.4
60.6
58.5
59.3
70.4
Attention
Causal
49.3
56.0
61.2
59.4
59.0
72.9
Bidirectional
51.3
57.2
62.6
61.4
60.4
74.7
Vision feature
Vision encoder feature + VLM vision tokens
52.4
59.4
62.5
62.2
62.7
73.9
Vision encoder feature only
51.8
59.1
61.9
61.7
62.7
73.4
VLM vision tokens only
52.5
59.5
62.8
61.0
61.4
72.8
Bridge token assignment
Raster order (64 tokens)
43.4
53.3
58.4
48.3
51.6
60.6
Spatial assignment (8x8 tokens)
51.7
59.2
62.5
60.8
61.7
72.4
Spatial assignment (11x11 tokens)
51.3
59.1
61.4
61.3
62.5
72.6
Multi-scale spatial assignment
52.4
59.4
62.5
62.2
62.7
73.9
Figure 5: Text-to-image attention visualization. We visualize layer-20 attention from selected text tokens (highlighted in red) to image tokens for the original Qwen3.5, Qwen3.5 fine-tuned on our dataset, and our model. Compared with the diffuse or noisy baselines, ConCor-1 produces more localized and semantically aligned attention. Additional visualizations are provided in Appendix 9.
Table 4: Quantitative text-to-image attention analysis across full-attention layers, averaged over the image-caption evaluation datasets. Attn-IoU measures the overlap between the highest-attention visual tokens and the phrase-specific ground-truth mask, using the ground-truth mask area to determine the number of selected tokens. Top-1 Hit measures whether the visual token receiving the maximum attention lies inside the ground-truth mask. Entropy measures the normalized spatial dispersion of attention, where lower values indicate more concentrated attention.
Metric
Model
L4
L8
L12
L16
L20
Attn-IoU ↑
Qwen3.5
0.16
0.20
0.22
0.22
0.21
Qwen3.5-FT
0.16
0.27
0.23
0.19
0.15
ConCor-1
0.29
0.42
0.48
0.33
0.57
Top-1 Hit ↑
Qwen3.5
0.32
0.43
0.43
0.43
0.40
Qwen3.5-FT
0.41
0.56
0.47
0.36
0.27
ConCor-1
0.56
0.70
0.70
0.55
0.75
Entropy ↓
Qwen3.5
0.82
0.69
0.73
0.74
0.73
Qwen3.5-FT
0.78
0.73
0.71
0.70
0.63
ConCor-1
0.73
0.66
0.65
0.68
0.54
Figure 6: Bridge-token attention and prediction. We visualize two bridge tokens at layer 20 for one test sample, including bridge-to-image attention, bridge-to-text attention, spatial prior (red bounding box), and final prediction. Different bridge tokens specialize to distinct object-level correspondence hypotheses, such as the dog and the seated person. Additional examples are provided in Appendix 10.
Table 5: Effect of the loss weights (λtext,λbce,λdice,λpres).
λtext:λbce:λdice:λpres
JointF1
MaskF1
TextF1
mJS
mMaskIoU
mSpanIoU
1 : 0.5 : 0.5 : 1
60.5
66.8
70.7
66.8
66.5
78.4
1 : 2 : 0.5 : 1
61.3
67.2
71.3
67.3
66.8
78.9
1 : 5 : 5 : 1
59.9
66.1
69.9
66.0
66.0
77.4
1 : 2.5 : 0 : 2
59.7
65.5
70.8
66.1
65.6
79.3
1 : 2 : 0.5 : 2
62.0
67.7
72.1
68.0
67.3
79.5
Figure 7: Spatial specialization of bridge tokens across token groups. We visualize the spatial activation patterns of bridge tokens from different scale groups under our multi-scale bridge-token design. Each group corresponds to a spatial partition of the image, ranging from a global 1×1 group to increasingly fine grids. Bridge tokens in coarse groups respond to broad image regions, while tokens in finer groups activate on increasingly localized areas aligned with their assigned spatial cells. This shows that bridge tokens learn structured spatial priors and progressively cover the image at multiple resolutions, supporting correspondence prediction for objects of different sizes and locations.
Table 6: Model size and computational profile. FLOPs, peak GPU memory, and latency are measured for one forward pass with a 512×512 image, 512 text tokens, batch size 1, bf16 precision, and an A100-80GB GPU.
Model
Seq. Len.
FLOPs
Peak Mem.
Latency
Params
Qwen3.5-0.8B backbone
770
1.061 T
1.687 GiB
54.0 ms
852.99 M
ConCor-1
1,155
1.549 T
1.846 GiB
55.4 ms
866.79 M
Δ
+385
+46.0%
+9.5%
+2.6%
+1.62%
Figure 8: Qualitative visualization examples.
Table 7: Component-level parameter and FLOPs breakdown. The computation induced by the bridge tokens is included in the language-model row.
Component
Added Params
GFLOPs
(M)
Backbone
ConCor-1
𝚫
Vision tower
–
221.5
221.5
–
Language model
–
839.3
1,279.2
+439.9
New bridge embeddings
0.145
–
–
–
Visual segmentation head
12.01
–
46.3
+46.3
Text segmentation head
1.11
–
1.1
+1.1
Presence head
0.52
–
0.4
+0.4
Total
13.80
1,060.8
1,548.5
+487.7
Figure 10: Qualitative visualization examples.
Table 8: Training data mixture used for ConCor-1. We report the source-level mixture weight in the final training setup. The image augmentation column indicates the fraction of each source group sampled with collage or crop augmentation; the remaining fraction is sampled without image augmentation.
Source dataset
Mix weight
Image aug.
Instance Segmentation Data
COCO 24
15%
80% collage
EntitySeg 34
10%
10% crop
PixMo Points 6
10%
10% crop
COCONut 7
5%
70% collage, 10% crop
SA-1B 18
5%
10% crop
ADE20K 57
5%
50% collage, 20% crop
Roboflow-VL-100 37
5%
—
Caption Grounding Data
GoldG 25
18%
—
COCONut-PanCap 7
15%
—
GroundedRef
12%
—
Total
100%
—
Table 9: Statistics of training data. Image counts are unique images per source and the total is their sum, so COCO images shared by COCO, COCONut-PanCap and GroundedRef are counted once per source.
Source dataset
Num. of images
Num. of instances
Annotation Type
Role
Instance Segmentation Data
COCO 24
117.3K
860.0K
Human annotated
Standard benchmark
COCONut (Objects365 subset) 8
242.6K
3.6M
Human annotated
Standard benchmark
EntitySeg 34
8.1K
79.1K
Human annotated
Standard benchmark
ADE20K 57
25.6K
639.5K
Human annotated
Scene-centric objects
Roboflow-VL-100 (81 datasets) 37
92.9K
812.3K
Converted benchmark data
Diverse domains
PixMo Points 6
56.8K
1.7M
Auto-constructed
Dense scenes + diverse text queries
SA-1B 18
71.1K
249.5K
Auto-constructed
Dense scenes + diverse text queries
Caption Grounding Data
GoldG 25
75.2K
2.6M
Rewritten + pseudo masks
Human-caption grounding
COCONut-PanCap 7
100.1K
1.3M
Rewritten + GT masks
Human-caption grounding
GroundedRef
60.2K
230.5K
Synthetic captions + GT masks
Referring and compositional grounding
Total
849.8K
12.0M
–
–
Table 10: Statistics of the three image-caption benchmarks we curated or repurposed. Caption length is measured by whitespace-tokenized words per caption. Span multiplicity denotes the number of distinct character spans associated with each mask instance.
Dataset
Images
Captions
Masks
Caption length
Span multiplicity
words/caption
spans/mask
Flickr30k
779
2,002
6,717
11.01 ± 4.36
1.06 ± 0.26
GroundedRef
1,318
2,025
6,193
10.34 ± 4.20
1.04 ± 0.19
COCONut-PanCap
2,213
2,213
10,821
50.84 ± 16.89
1.68 ± 0.90
Table 11: Effects of annotation standardization and model architecture. Image-caption JointF1 is averaged over the three caption benchmarks, while image-category JointF1 is averaged over COCO and LVIS-minival.
Figure 1: Bidirectional concept correspondence. Given an image paired with different forms of text, including a full caption, a referring expression, or a list of category names, the concept correspondence model predicts a complete set of correspondences between text segments and image segments. For each correspondence, the model identifies the visually grounded text span, including repeated or coreferential mentions, and localizes the corresponding instance-level mask in the image.
Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as bidirectional concept correspondence over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce ConCor-1, a grounding model built on top of a pretrained vision-language model. It uses learnable bridge tokens to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that ConCor-1 consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.
