CoToGrasp: Contact-Topology-Conditioned Dexterous Grasp Synthesis via Canonical Workspace Learning
arXiv:2608.197762026-08-19
로봇 손이 물건을 '아무렇게나' 잡지 않고, 원하는 방식대로 잡게 만드는 방법
기존 로봇 그리퍼(집게손) 학습 방법들은 물건을 안 떨어뜨리고 안정적으로 쥐는 데만 집중해서, 대부분 손 전체로 감싸 쥐는 방식으로만 수렴하는 문제가 있었다. CoToGrasp는 사람의 손 잡는 방식(예: 두 손가락으로 집기, 손 전체로 감싸기, 도구 전용 잡기 등 21가지 유형)을 그대로 흉내내도록 학습하되, 물체별 데이터 없이 그리퍼 자체의 접촉 가능 영역만 학습해 처음 보는 물체에도 바로 적용할 수 있게 했다. 대규모 DexGraspNet 데이터셋 실험과 실제 로봇 팔 실험 모두에서 기존 방법보다 원하는 잡기 방식을 더 정확히 재현하면서도 안정성도 더 높았다.
무엇을 했나
로봇이 물체를 쥘 때 '떨어뜨리지 않는지'만 보는 기존 방식은 정밀 집기, 손잡이 잡기 같은 작업에 필요한 다양한 잡기 형태를 만들지 못하고 감싸쥐기로 획일화되는 문제(모드 붕괴)가 있었다.
CoToGrasp는 물체가 아니라 로봇 손(그리퍼) 자체를 기준으로 하는 '공통 작업공간'을 만들어, 손의 어느 부위가 어떤 물체 표면과 닿을 수 있는지를 물체 종류와 무관하게 학습한다.
학습 시에는 오직 그리퍼의 표면 모양과 사람 손 잡기 분류(정밀, 파워, 도구 전용 등 21가지) 정보만 사용하고, 실제 물체 데이터나 물리 시뮬레이션 없이 21만 개의 학습 데이터를 순수 기하학적으로 생성했다.
추론 시에는 물체를 이 공통 좌표계로 옮겨 넣고, DGCNN·트랜스포머·CVAE로 이루어진 신경망이 접촉해야 할 손 부위를 예측한 뒤, 힘 균형 검증과 에너지 최적화를 거쳐 최종 손가락 관절 각도를 정한다.
DexGraspNet 데이터셋 평가에서 기존 분류체계 기반 방법(Dexonomy)보다 안정적 성공률과 의도한 잡기 유형 일치도 모두 더 높았고, 실제 Allegro 로봇 손으로 YCB 물체를 잡는 실험에서도 물리적으로 실행 가능함을 확인했다.
Figure 1: Contact-Topology-Conditioned Grasp Synthesis. Given a desired semantic contact-topology condition (top left) – categorized into Precision, Object-Specific (highly constrained topologies tailored for specific tool use) or Power functional groups – and a novel, unseen object (bottom left), our framework synthesizes functionally diverse and physically stable grasps (right). Rather than learning contact topologies directly on the object geometry, we project local object features into a feature-based canonical workspace. This unified spatial representation effectively decouples the functional intent from the specific object identity. Within this workspace, we learn a latent manifold (center) that models the intrinsic contact capabilities of the gripper, enabling zero-shot generalization to diverse target geometries.Figure 2: CoToGrasp Method Overview. The proposed framework operates in two distinct phases. Top (Object-Agnostic Training): The model learns an intrinsic, gripper-centric contact manifold within a canonical feature-based workspace, independent of object geometry. Bottom (Grasp Synthesis): At inference, a target object is transformed into the canonical frame. The network’s contact-topology-conditioned prediction is strictly filtered through a validation pipeline before energy-based optimization aligns the gripper to yield the final stable grasp (Q∗).
Table 1: Comparison with taxonomy-unaware baselines. CoToGrasp achieves the highest semantic entropy (HTC) and generation speed, overcoming the functional mode collapse typical of unconditioned planners.
Method
Object-Agnostic Training
SR ↑
HTC ↑
Speed (sec. / grasps)
Diversity (avg.) ↑
𝐭 (m)
R (rad)
Q (rad)
DFC [26]
✓
72.15
0.7389
>1800
0.0607
1.424
0.3579
GenDexGrasp [24]
✗
71.15
0.5956
14.65
0.0519
1.416
0.2567
DRO-Grasp [43]
✗
63.30
0.6504
1.72
0.0546
1.515
0.2892
GOAG [32]
✓
77.90
0.6527
0.20
0.0479
1.401
0.3170
CoToGrasp
✓
36.94
0.83
0.11
0.0674
1.4927
0.3458
Figure 3: Semantic Grasp Taxonomy and Contact Mapping. (Left) Correspondences between the classical Feix [11] (F) taxonomy (top), the haptic Gonzalez [12] (M) taxonomy (middle row) and our derived point cloud contact templates 𝒜m (bottom row). We categorized the 21 templates into three distinct functional groups: Precision, Power and Object-Specific (highly constrained topologies tailored for specific tool use). (Right) The 22 anatomical contact zones defined by Gonzalez (top) and the direct surjective mapping (ζ) onto our discrete gripper handprint ℋ (bottom).Figure 4: Functional contact topology distribution across taxonomy-unaware planners. The histogram illustrates the distribution of grasps generated by unconditioned baselines compared to CoToGrasp on the Multidex objects set. The unknown category represents physically stable grasps with unnatural contact patterns that fail to match any contact topology. Notably, unconditioned baselines exhibit a severe generative bias (mode collapse) toward enveloping power grasps (red box).
Table 2: Taxonomy-aware grasp synthesis. CoToGrasp outperforms the baseline in both physical stability (SR) and semantic accuracy (TC), particularly on highly constrained precision grasps.
Method
SR (%)
HSR
TC (%)
HTC
Power
Precision
Obj. Spe.
Avg. Topo.
Avg. Obj.
Dexonomy [5]
27.16
12.36
19.62
21.13
23.80
0.91
14.28
0.77
CoToGrasp
29.75
22.71
25.50
26.72
27.56
0.96
17.18
0.84
CoToGrasp (w/o Label-Consistency)
25.11
14.77
20.85
21.14
22.97
0.94
14.45
0.81
CoToGrasp (w/o Force-Closure)
26.90
14.87
21.08
22.08
23.65
0.95
16.26
0.81
CoToGrasp (No Check)
25.09
14.73
20.58
21.06
23.00
0.94
14.72
0.81
Figure 5: Real-World kinematic validation. CoToGrasp synthesizes diverse, topology-compliant grasps that are physically executable on a physical Allegro Hand using YCB [4] objects. The target contact topologies (indicated above each frame) demonstrate the physical viability of the generated grasps across both precision and power categories.Figure 6: Handprint areas and workspace constitution. Left: Discretized handprints of the Shadow and Allegro hands, illustrating the manually defined anatomical zone divisions. Right: A three-quarter view of the Shadow Hand’s workspace.
Table 3: Per-Topology results: 3 Power grasps (M13, M18, M21), 2 Precision (M2, M6) and 1 Object-Specific (M11). †Indicates artificially inflated scores due to mode collapse, where Dexonomy defaults to unverified enveloping grasps.
Method
SR (%)
M2
M6
M11
M13
M18
M21
Dexonomy [5]
10.5
15.2
60.5†
20.3
29.6
37.2†
CoToGrasp
30.3
21.7
29.8
29.6
31.3
33.5
Figure 7: Taxonomy transfer mapping for anthropomorphic grippers. The handprints of the Shadow Hand (left) and the Allegro Hand (right) are segmented into manually defined, corresponding anatomical zones (𝒜1–𝒜21).Figure 8: t-SNE visualization of features after canonical workspace projection.
Table 4: Implementation Details and Network Hyperparameters for the CoToGrasp framework.
Module / Parameter
Notation
Value / Size
Training & Optimization
Batch Size
–
32
Learning Rate
–
10−5
Training Epochs
–
50
KLD Regularization Weight
β
0.1
Attention Factor
α
3.0
Workspace & Geometric Projection
Workspace Resolution
NW
8,192
Projection kNN
k
5
Aligned Distance Scaling
γ
2.0
Pointwise Feature Extraction
Local Graph kNN
K
16
Hidden Layer Sizes
–
[12,64,64,128,256,512]
Output Feature Dimension
ℱ𝒫,ℱ𝒲
1,024
Conditioning Embeddings
Topology Embedding Dim.
ℱ𝒯
64
Label Embedding (MLP)
ℱΛ
[128,64]
Self-Attention Modules
Transformer Encoder Feature Dim.
Φ
1088
Transformer Encoder Blocks
–
4
Transformer Encoder Heads
–
8
Set Transformer (Pooling) Heads
–
8
Global Latent Descriptor Dim.
𝒵
1,024
CVAE & Latent Space
Latent Encoder Hidden Dim.
–
512
Latent Variable Dimension
ψ
64
AdaLN Decoder Output Classes
N+1
23
Figure 9: Topology-Conditioned Pose Sampling. For specific grasp topologies (such as M4 and M12), the initial object pose is sampled within a restricted kinematic region. Middle: The template’s full active sub-workspace is shown in light blue, while the truncated sub-workspace – filtered for reachability and palm clearance – is highlighted in red. Right: Examples of the initialized object point cloud 𝒪~ (green) successfully placed within this feasible region after the sampled spatial transformation.Figure 10: Illustration of metric-induced misclassification. While CoToGrasp strictly respects the target contact topology, the kinematic optimization may result in incidental contacts where an adjacent phalanx rests against the object surface. Despite the repulsive term in our optimization energy function, these phalanges often cannot be pushed away due to inherent kinematic constraints. For instance, the left grasp illustrates an intended M3 pinch reclassified as M12, while the right shows an M4 grasp reclassified as M15. Although the intended contacts are successfully achieved and the grasps remain physically stable, these incidental contacts trigger a strict reclassification by our automated pipeline.
Table 5: Feature Alignment (Cosine Similarity).
Raw DGCNN
Workspace
+ Attn.
Matched Pairs
0.35
0.61
0.66
Random Pairs
0.23
0.33
0.27
Figure 11: Histogram comparing the frequencies of effective topologies and attempted topologies (as defined Sec. 4.2) among all stable grasps generated by CoToGrasp (top) and Dexonomy [5] (bottom).
Table 6: Topology compliance ablation.
Label
Eval.
HSR
TC (%)
HTC
Consistency
Isaac
✓
✓
0.96
17.18
0.84
✓
✗
21.92
0.53
✗
✓
0.94
14.45
0.81
✗
✗
19.34
0.50
Table 7: Object-Level Analysis. Evaluating performance across geometric complexity (c=Vobj/Vhull). CoToGrasp shows superior retention of both physical stability (SR) and semantic compliance (TC) on challenging non-convex objects.
Object Complexity
Metric
CoToGrasp
Dexonomy [5]
Convex → Non-Convex
SR Retained
58.72%
37.42%
Severe Concavities (c<0.4, ∼4% data)
SR
18.30%
12.05%
TC
19.17%
8.94%
왜 중요한가
고차원 자유도를 가진 다지 로봇 손이 단순히 물체를 놓치지 않는 것을 넘어, 도구 사용이나 손잡이 조작처럼 특정 방식으로 잡아야 하는 작업을 수행하려면 잡는 방식 자체를 지정할 수 있어야 한다. 이 방법은 물체별 학습 데이터 없이도 원하는 잡기 방식을 지정해 새로운 물체에 즉시 적용할 수 있어, 로봇 매니퓰레이션 응용의 데이터 수집 비용 문제를 줄여준다.
이 논문의 용어
그리퍼(Gripper) · 물체를 쥐는 로봇의 손 또는 집게 부분
접촉 위상(Contact Topology) · 손의 어느 부위가 물체의 어느 부위와 닿는지를 정의한 패턴, 사람 손 잡기 방식 분류 체계에서 가져옴
제로샷 일반화(Zero-shot generalization) · 학습 때 본 적 없는 새로운 대상(물체)에도 추가 학습 없이 바로 적용되는 능력
CVAE(Conditional Variational Auto-Encoder) · 조건 정보를 주면 그에 맞는 다양한 결과를 확률적으로 생성하는 신경망 구조
모드 붕괴(Mode collapse) · 생성 모델이 다양한 결과 대신 몇 가지 패턴에만 몰려서 결과를 내는 현상
논문 원문 초록 (영문)
Current dexterous grasp planners primarily optimize for physical stability, focusing on whether an object can be grasped rather than how it should be grasped to support downstream functional tasks. However, conditioning grasp synthesis on specific human grasp taxonomies typically requires prohibitively expensive, object-annotated datasets. To address these limitations, we propose CoToGrasp, a novel generative framework that synthesizes diverse, stable grasps strictly conditioned on specific contact topologies. To bypass the data collection bottleneck, CoToGrasp is trained entirely in an object-agnostic manner. We introduce a feature-based canonical workspace that projects local object features into a unified gripper-centric domain, effectively decoupling the semantic functional intent from the arbitrary object geometry. By learning the intrinsic contact manifold of the gripper within this workspace, our model achieves zero-shot generalization to unseen objects at inference. Extensive evaluations on the large-scale DexGraspNet dataset demonstrate that CoToGrasp achieves state-of-the-art performance, outperforming existing taxonomy-guided planners. Finally, we demonstrate the physical viability and kinematic feasibility of our synthesized contact topologies on a physical robot platform. Code is available on our project website https://cea-list.github.io/cotograspweb/ .
![Figure 3: Semantic Grasp Taxonomy and Contact Mapping. (Left) Correspondences between the classical Feix [11] (F) taxonomy (top), the haptic Gonzalez [12] (M) taxonomy (middle row) and our derived point cloud contact templates 𝒜m (bottom row). We categorized the 21 templates into three distinct functional groups: Precision, Power and Object-Specific (highly constrained topologies tailored for specific tool use). (Right) The 22 anatomical contact zones defined by Gonzalez (top) and the direct surjective mapping (ζ) onto our discrete gripper handprint ℋ (bottom).](https://media.metallab.ai/papers/2608.19776/f2.png)
![Figure 5: Real-World kinematic validation. CoToGrasp synthesizes diverse, topology-compliant grasps that are physically executable on a physical Allegro Hand using YCB [4] objects. The target contact topologies (indicated above each frame) demonstrate the physical viability of the generated grasps across both precision and power categories.](https://media.metallab.ai/papers/2608.19776/f4.png)
![Figure 11: Histogram comparing the frequencies of effective topologies and attempted topologies (as defined Sec. 4.2) among all stable grasps generated by CoToGrasp (top) and Dexonomy [5] (bottom).](https://media.metallab.ai/papers/2608.19776/f10.svg)
