[Paper Notes] HRDexDB: A Paired Human-Robot Dataset for Cross-Embodiment Dexterous Grasping
Published:
HRDexDB is a dataset paper, but its real contribution is a data contract for studying cross-embodiment dexterity. It captures human grasps and multiple dexterous robot grasps on the same objects, in the same calibrated workspace, with reconstructed 3D hand motion, robot states, object 6D trajectories, egocentric views, and tactile signals where the robot hardware supports them.
My read: the paper is useful because it moves the question from “can we retarget a human hand pose to a robot hand?” to “can we learn how the contact strategy itself changes across embodiments?” That distinction matters for dexterous grasping. Different hands can share the same grasp intent while using different fingers, contact patches, timing, and feasible force closure patterns.
Paper Info
The paper is “HRDexDB: A Paired Human-Robot Dataset for Cross-Embodiment Dexterous Grasping” by Jongbin Lim, Taeyun Ha, Mingi Choi, Jisoo Kim, Byungjun Kim, Subin Jeon, and Hanbyul Joo from Seoul National University and RLWRLD. It is available as arXiv:2604.14944, with project page snuvclab.github.io/HRDexDB.
The Problem
Human manipulation is an attractive source of demonstrations because most everyday objects are designed around the human hand. The hard part is that human hands and robot hands differ in morphology, kinematics, actuation, sensing, and feasible contacts. Even two robot hands can require different grasp strategies for the same object.
Existing datasets usually cover one side of the problem. Human-object interaction datasets provide rich human hand motion and object annotations. Robot datasets provide robot trajectories, often with grippers or a single dexterous embodiment. Paired human-robot datasets exist, but the paper argues that they lack the combination HRDexDB targets: markerless RGB observations, multiple dexterous robot hands, shared objects, 3D hand and object annotations, and tactile signals in a unified setup.
The central missing ingredient is paired behavior. For cross-embodiment learning, the dataset should show a human grasp and a robot grasp that are semantically comparable on the same object, while still allowing embodiment-specific differences in motion and contact.
Dataset Contract
HRDexDB contains 2.1K grasping sequences, 24M frames, 100 objects, and five embodiments: one human hand plus four robotic hands, namely Allegro Hand V4, Allegro Hand V5 Plus, Inspire Hand RH56DFTP, and Inspire Hand RH56F1. The capture system uses 21 synchronized exocentric RGB cameras and 2 egocentric views, giving 23 synchronized RGB streams in total. The dataset also includes scanned object meshes, object 6D pose trajectories, success/failure labels, and tactile force signals for tactile-enabled robot fingertips.
The robot trial representation is:
[ T^{robot}=\left{{I_t^{c_i}}{i=1}^{21}, I_t^{ego}, q_t^{robot}, T_t^{object}, F_t^{tactile}, y\right}{t=1}^{T_r}. ]
The human trial representation is:
[ T^{human}=\left{{I_t^{c_i}}{i=1}^{21}, I_t^{ego}, \theta_t^{human}, T_t^{object}, y\right}{t=1}^{T_h}. ]
Here (q_t^{robot}) is the robot state, (\theta_t^{human}\in\mathbb{R}^{51}) is the MANO hand pose, (T_t^{object}\in SE(3)) is the object pose, (F_t^{tactile}) is tactile force data, and (y\in{0,1}) marks grasp success.
This representation is compact but important. It aligns visual, kinematic, object, tactile, and outcome signals in one coordinate system, making the dataset usable for both manipulation learning and perception evaluation.
Capture And Reconstruction
The acquisition protocol has two stages. A human subject first performs a natural grasp on the target object. Then a teleoperator observes that demonstration and performs a semantically corresponding grasp with the target robot hand. The teleoperation setup uses an Xsens inertial motion-capture suit and MANUS gloves to map wrist and finger motion to the robot arm and hand.
This protocol avoids forcing an exact joint-level correspondence. The robot execution preserves the grasp intent while adapting to the robot’s own morphology. That is the point of the dataset: the paired examples expose how successful contact strategies change when the hand changes.
The reconstruction pipeline combines several vision components:
- Human hand motion: HaMeR detects 2D hand keypoints in each calibrated view, triangulation produces 3D joints, and MANO fitting recovers pose parameters. Subject-specific shape is calibrated with SAM3 masks and silhouette alignment.
- Object pose: FoundationStereo estimates dense depth from a calibrated stereo pair, SAM3 segments the object, and FoundationPose estimates and tracks the CAD model pose through time.
- Multi-view consistency: the object mesh is rendered into all calibrated views, and silhouette misalignment is minimized to reduce drift during long grasps.
The practical challenge is occlusion. Dexterous hands and objects hide each other constantly, so the paper’s multi-camera system is not decoration; it is what makes markerless 3D supervision plausible.
Benchmark 1: Contact Map Transfer
The first downstream task asks: given a human contact pattern on an object, can a model predict the robot-specific contact pattern that will actually work for a target hand?
Each grasp is represented on an object point cloud (O\in\mathbb{R}^{N\times3}) with a contact map (C\in[0,1]^N) and a part map (P). The human part map uses six parts, while the robot part map uses (B=6) for Inspire and (B=5) for Allegro. Conditioned on the human representation ([C^h,P^h]) and PointNet++ object features, the model predicts ([C^r,P^r]). Training uses a contact-weighted L1 loss for (C^r) and cross-entropy over contacted points for (P^r).
The predicted robot contact map is then used as the objective for a CEDex-style physics-aware grasp optimizer with contact, penetration, and self-collision terms. The comparison is clean: the optimizer stays fixed, and only the contact objective changes.
The results show why paired data matters. Directly using the human contact map reaches 66.7% real-world success for Inspire and 63.3% for Allegro. Using the transferred robot-specific contact map improves this to 73.3% and 80.0%. In simulation, the same trend appears, with Allegro improving from 60.2% to 65.8%.
Benchmark 2: Latent-Space Grasp Retrieval
The second task learns a shared embedding space across human, Inspire, and Allegro grasps. Given a human hand-object grasp as query, the model retrieves robot grasp candidates from HRDexDB whose geometry and function match the query.
The architecture is CLIP-like: separate point-cloud encoders process the human hand, Inspire hand, Allegro hand, and object, then project query and candidate branches into a shared latent space with a symmetric contrastive loss.
Retrieval is evaluated in two ways. First, the paper measures whether paired robot grasps are ranked highly among 33 candidates. Human-to-Inspire retrieval reaches 36.36% R@1, 81.82% R@3, and 100.00% R@5. Human-to-Allegro is harder, at 24.24% R@1, 63.64% R@3, and 72.73% R@5.
Second, retrieved grasps are used to initialize BODex refinement. This is a useful test because a retrieved prior only matters if it helps downstream optimization. Retrieval-top5 gives the best episode-level success: 75.76% for Inspire-F1 and 93.94% for Allegro-V5, outperforming vanilla BODex and kinematic retargeting. Retrieval-top1 gives higher seed-level precision, while top5 gives better coverage across episodes.
Benchmark 3: 3D Hand Pose Estimation
HRDexDB also functions as a perception benchmark. The paper evaluates WiLoR, HaMeR, Hamba, MeshGraphormer, and FrankMocap on captured human hand-object sequences.
The dataset is harder than FreiHAND for most methods. For example, WiLoR’s PA-MPVPE increases from 5.27 mm on FreiHAND to 6.09 mm on HRDexDB, and MeshGraphormer’s PA-MPJPE increases from 6.64 mm to 8.31 mm. The reason is expected: hand-object grasping causes persistent occlusion and contact-rich hand shapes, which are underrepresented in cleaner single-hand benchmarks.
The paper also tests whether HRDexDB adds useful training signal. Adding 6K HRDexDB samples to a 2.7M-sample finetuning pool slightly improves HaMeR and WiLoR on FreiHAND. The improvement is modest, but it supports the claim that the data is complementary instead of redundant.
Benchmark 4: Object 6D Pose Under Grasping
The object pose benchmark compares FoundPose, GigaPose, PicoPose, and MegaPose-refined variants on paired human-grasp and robot-grasp frames. All methods receive the same RGB image, mask, camera intrinsics, object identity, and CAD model.
Robot grasping is consistently harder than human grasping. FoundPose with MegaPose refinement reaches 3.35 cm ADD and 70.00% ARMSSD on human-grasp frames, but drops to 4.40 cm ADD and 64.10% ARMSSD on robot-grasp frames. The paper’s explanation is plausible: rigid robot links and fingertips introduce visual structures that overlap object boundaries and confuse pose localization.
The authors further fine-tune the MegaPose refiner with 100K GSO synthetic samples plus 5.3K HRDexDB robot-grasp annotations, then evaluate on a held-out OmniRobotHome robot-grasp environment. ADD-S improves from 4.40 cm to 3.95 cm, a 10.2% relative gain. This positions HRDexDB as training data for interaction-centric pose refinement, beyond its role as an evaluation set.
Strengths And Limitations
HRDexDB’s strength is its pairing. The dataset does not merely place human and robot demonstrations in the same folder. It captures comparable human and robot grasps over shared objects, reconstructs both in a calibrated 3D workspace, and includes the signals needed to study contact, retrieval, perception, and outcome.
Another strength is embodiment diversity. Four robot hands are still a small sample of the hand-design space, but they are enough to expose why direct hand-pose retargeting is brittle and why contact-level or latent-space transfer can be more useful.
The limitations are also clear. Tactile sensing is available only for robot hands, and sensor specifications vary across platforms, making unified tactile modeling difficult. The human-robot correspondence is semantic, so defining functionally equivalent trajectories across different hand morphologies remains open. The dataset also focuses on grasping; the authors plan to expand toward 1,000 objects and more complex functional manipulation tasks.
Takeaway
HRDexDB is best read as infrastructure for learning embodiment-aware dexterity. It gives researchers paired examples where a human grasp and a robot grasp aim at the same object-level function while expressing that function through different bodies.
For robot learning, the strongest lesson is that cross-embodiment transfer should be evaluated at the level of contact, object motion, tactile outcome, and perception robustness. A human demonstration is not just a pose sequence to copy. It is evidence about object affordance and grasp intent, and HRDexDB makes that evidence measurable across multiple robot hands.