One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation

arXiv:2608.197592026-08-19

A robot hand learns to grasp new objects by only studying its own fingers, never the objects

Robot hands that grip objects with multiple fingers usually need training on object-specific datasets, which makes them struggle with unfamiliar objects. GOAG instead trains only on the gripper's own surface geometry and joint kinematics, never seeing any object during training, and only looks at object shape at inference time. On the MultiDex benchmark it reaches an average success rate of 86.93%, matching top methods trained specifically on that dataset while being much faster when generating many grasps.

What they did

  1. Existing deep-learning grasp planners are trained on limited, object-specific datasets and often fail to generalize to new objects
  2. GOAG generates its training data purely from the gripper's own surface and joint configurations, so the training phase never involves any object geometry
  3. Using six grasp types adapted from a human grasp taxonomy, the team sampled contact regions on the gripper to build a 3-million-sample dataset in about 1 GPU hour, versus 1,400 GPU hours reported by a prior method
  4. At inference, a Conditional Variational Autoencoder (CVAE) takes the new object's shape and predicts plausible contact points on it, which then pass through a force-closure check and an optimization step to produce the final hand pose
  5. GOAG achieved 86.93% average success rate on MultiDex, matching leading dataset-specific methods, ranked second-highest average success rate across five different grasp benchmarks using only a single Shadow Hand model without retraining, and successfully grasped 11 real YCB objects with a physical robot arm
Fig. 1: GOAG Paradigm. A successful grasp on an object induces dual contact zones on both object and gripper, at the intersection of the two geometries. Our method is built on this key observation: these contact zones (𝒞(.)) are closely the same from either perspective. GOAG capitalizes on this by training exclusively on gripper geometry, allowing it to learn a robust and generalizable grasping strategy without ever being exposed to a grasp database with specific objects geometries.
Fig. 1: GOAG Paradigm. A successful grasp on an object induces dual contact zones on both object and gripper, at the intersection of the two geometries. Our method is built on this key observation: these contact zones (𝒞(.)) are closely the same from either perspective. GOAG capitalizes on this by training exclusively on gripper geometry, allowing it to learn a robust and generalizable grasping strategy without ever being exposed to a grasp database with specific objects geometries.
TABLE I: Comparison of Dexterous Grasp Planning Methods.
Grasp RepresentationGraspRepresentationGripper PoseGripperPoseGripper Joint ValuesGripperJoint ValuesForce ClosureForceClosureNon- PenetrationNon-PenetrationTraining SetTrainingSetWorking Reference Frame
Grasp
Representation
Gripper
Pose
Gripper
Joint Values
Force
Closure
Non-
Penetration
Training
Set
Working
Reference Frame
Optional Grasp
Preference Interface
DFC [14]DirectOptimizedOptimized✓✓NoObject✗
UniGrasp [24]IntermediateIK SolvedIK Solved✓✗Objects + GrippersObject✗
GeoMatch [2]IntermediateIK SolvedIK Solved✓✗Objects + GrippersObject✗
GenDexGrasp [12]IntermediateOptimizedOptimized✓✓Objects + GrippersObject✗
ManiFM [37]IntermediateOptimizedOptimized✓✓Objects + GrippersObjectContact Region
DRO-Grasp [32]IntermediateOptimizedOptimized✓✓Objects + GrippersObjectPalm Orientation
DexDiffuser [33]DirectLearnedLearnedA posterioriA posterioriObjects + GrippersObject✗
DexGrasp Anything [39]DirectLearnedLearned✓✓Objects + GrippersObject✗
GOAG (Ours)IntermediateSampledOptimized✓✓Gripper OnlyGripperPalm Full Pose
Fig. 2: Grasp Taxonomy Adaptation and Contact Sampling. (Top) We adapt the human grasp taxonomy from [7] to the Allegro Hand geometry. For each grasp type (e.g., C6, F27), we define a corresponding admissible contact region (black points), distinguishing it from the non-contact surface (blue points). (Bottom) Data generation mechanism: We randomly sample specific contact points (red) strictly within the admissible black regions. This allows the model to learn structured, feasible contact distributions based solely on gripper kinematics, independent of any object.
Fig. 2: Grasp Taxonomy Adaptation and Contact Sampling. (Top) We adapt the human grasp taxonomy from [7] to the Allegro Hand geometry. For each grasp type (e.g., C6, F27), we define a corresponding admissible contact region (black points), distinguishing it from the non-contact surface (blue points). (Bottom) Data generation mechanism: We randomly sample specific contact points (red) strictly within the admissible black regions. This allows the model to learn structured, feasible contact distributions based solely on gripper kinematics, independent of any object.
TABLE II: In-depth grasp performance analysis on the Multidex [12] test set. We report our grasp results, on three classic dexterous grippers in terms of success rate, efficiency and diversity.
MethodData DrivenObject-Agnostic TrainingSuccess Rate (%) ↑Efficiency (sec. / grasps) ↓Diversity (avg.) ↑
BarrettAllegroShadowHandAvg.BarrettAllegroShadowHandT (m)R (rad)Q (rad)
DFC [14]✗✓83.1082.7172.1579.32>1800>1800>18000.06071.4240.3579
GenDexGrasp [12] (full)✓✗70.2671.4871.1570.969.7816.4514.650.05191.4160.2567
DRO-Grasp [32] (pretrain, w/o controller)✓✗78.3075.8063.3072.470.880.421.720.05461.5150.2892
GOAG (w/o FC)✓✓86.3091.2074.7084.070.090.130.150.04801.3960.3162
GOAG✓✓87.4093.2077.9086.930.180.190.200.04791.4010.3170
Fig. 3: Overview of GOAG. Geometrical graspability is learned in an object-agnostic manner by focusing on the gripper’s capabilities. Training: We sample gripper configurations Q to generate ℋ⁡(Q) and corresponding contact points (𝒞⁡(ℋ⁡(Q))). To ensure transferability, we use a Basis Point Set (BPS) encoding tied to the gripper’s workspace. A Conditional Variational Autoencoder (CVAE) is trained to reconstruct these contact distributions, while a Links Mapper (PointNet++) learns to associate contact points with specific gripper links. Inference: A novel object 𝒪, positioned at the inverse gripper pose [R,T]−1, is BPS-encoded. By sampling a latent variable z∈ℝψ, the CVAE Decoder generatively predicts diverse, plausible contact points 𝒞^​(𝒪). The Links Mapper then labels which gripper link should reach each point. Generation: Finally, a Force Closure check ensures the predicted contacts yield a stable grasp, and a Grasp Optimization step outputs the final, refined gripper configuration Q∗.
Fig. 3: Overview of GOAG. Geometrical graspability is learned in an object-agnostic manner by focusing on the gripper’s capabilities. Training: We sample gripper configurations Q to generate ℋ⁡(Q) and corresponding contact points (𝒞⁡(ℋ⁡(Q))). To ensure transferability, we use a Basis Point Set (BPS) encoding tied to the gripper’s workspace. A Conditional Variational Autoencoder (CVAE) is trained to reconstruct these contact distributions, while a Links Mapper (PointNet++) learns to associate contact points with specific gripper links. Inference: A novel object 𝒪, positioned at the inverse gripper pose [R,T]−1, is BPS-encoded. By sampling a latent variable z∈ℝψ, the CVAE Decoder generatively predicts diverse, plausible contact points 𝒞^​(𝒪). The Links Mapper then labels which gripper link should reach each point. Generation: Finally, a Force Closure check ensures the predicted contacts yield a stable grasp, and a Grasp Optimization step outputs the final, refined gripper configuration Q∗.
TABLE III: Grasp generalization assessment across multiple grasp datasets. Following [39] we evaluate our method performances with the Shadow hand on multiple grasp test sets. We report the success rates as defined in IV-C. It is worth noting that state-of-the-art methods are retrained for each dataset. GOAG has only been trained once on Shadow hand kinematics.
MethodPer-Dataset TrainingDexGraspNet ↑UniDexGrasp ↑MultiDex ↑RealDex ↑DexGRAB ↑Avg. (%)
UniDexGrasp [35]✓33.923.721.627.120.825.42
GraspTTA [10]✓18.621.030.313.314.419.52
SceneDiffuser [9]✓26.628.369.821.739.137.10
UGG [16]✓46.946.055.332.742.744.72
DGA [39]✓57.553.179.144.857.958.48
GOAG✗43.0749.5177.9037.3762.1353.97
Fig. 4: GOAG grasp results on Multidex [12] objects. Grasps are shown for the Barrett (green), Allegro (pink), and Shadow Hand (purple) grippers.
Fig. 4: GOAG grasp results on Multidex [12] objects. Grasps are shown for the Barrett (green), Allegro (pink), and Shadow Hand (purple) grippers.

Why it matters

Robots in warehouses or homes encounter an enormous variety of objects, and retraining a grasp model for every new object is costly and impractical. By training only once on the gripper itself, GOAG offers a more practical path to robots that can grip unfamiliar objects without needing new object data or retraining.

Fig. 5: Real-world setup and results with Allegro hand on YCB [5] objects. First row presents the real robot grasps. Second row presents corresponding virtual grasps. Objects have been rotated around the z-axis for a better understanding of the grasp poses.
Fig. 5: Real-world setup and results with Allegro hand on YCB [5] objects. First row presents the real robot grasps. Second row presents corresponding virtual grasps. Objects have been rotated around the z-axis for a better understanding of the grasp poses.

Terms in this paper

  • Gripper · the robotic hand mechanism used to grasp objects
  • CVAE (Conditional Variational Autoencoder) · a generative deep learning model that produces varied outputs conditioned on given input information
  • BPS (Basis Point Set) · a method that encodes a 3D point cloud into a fixed-size numerical representation
  • Force Closure · a physical condition checking whether a set of contact points can hold an object stably without slipping
  • PointNet++ · a deep learning network architecture designed to process 3D point cloud data

Original abstract (English)

Multifingered grasping is a crucial robotic skill, but current deep-learning grasp planners often struggle to generalize to new objects because they are trained on limited, object-specific datasets. We introduce a fundamentally different approach, grounded in the observation that the gripper and the object share identical surface geometry at their mutual contact points. We propose GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation, a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution, enabling the efficient sampling of valid grasp configurations without relying on object-specific training data. We show that by introducing object features only at inference time, our model can effectively retrieve admissible contact areas that are compatible with the gripper's capabilities. We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios, demonstrating its effectiveness with different grippers from the literature. Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%. It offers significantly faster processing when generating numerous grasps, while matching the performance of leading approaches specifically trained on this dataset. Unlike these methods, our approach does not rely on object-specific training data, highlighting the advantages of object-agnostic learning. It effectively addresses the generalization challenges faced by traditional data-driven grasp planners. Code and videos are available on our project website https://cea-list.github.io/goagweb/ .

Authors · Julien Merand

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Julien Merand et al., arXiv:2608.19759, CC BY 4.0