[Paper Notes] CHORD: Contact Wrench Guidance From Human Demonstration
Published:
CHORD studies a central problem in dexterous robot learning: human demonstrations contain rich manipulation knowledge, yet robot hands cannot directly replay human hand motion. Different morphology, kinematics, fingertip geometry, and contact timing can turn the same-looking motion into a different object effect.
My read: the paper’s useful abstraction is object-centric contact wrench guidance. CHORD asks the robot to reproduce the object-level mechanical effect of a human contact, measured by the force-torque directions that contacts can induce on the object. This gives reinforcement learning a contact-rich reward that is less tied to a specific hand shape than raw contact positions.
Paper Info
The paper is “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration” by Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, Michael Andres Lin, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, and Yan Chang from NVIDIA. The project page is nvidia-isaac.github.io/video_to_data/chord, and the technical report PDF is linked there.
CHORD stands for Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation. The reported benchmark contains 4,739 bimanual dexterous manipulation tasks, and the main evaluation covers 1,831 tasks. The paper reports 82.12% average success on this large-scale simulation evaluation, 90.77% success on whole-body manipulation tasks, and successful transfer to a real Dexmate robot with two Sharpa dexterous hands.
Why Human Motion Is Not Enough
A human demonstration gives at least three signals: hand motion, object motion, and hand-object contact. Purely tracking object motion is often too sparse for reinforcement learning because the robot receives useful feedback only after making the right contact. Purely tracking hand motion is brittle because the robot hand may need a different pose to generate the same effect. Tracking contact locations helps, yet it can still be mechanically wrong: touching the same object patch with a different surface normal or force direction may push, lever, or slide the object in the wrong way.
CHORD’s answer is to move the comparison into wrench space. A contact is judged by the force and torque directions it can apply to an object part. This makes the target more functional: the robot can choose embodiment-specific contacts as long as those contacts support object motion aligned with the human reference.
The Core Formulation
Each reference trajectory is written as
[ \tau^{\mathrm{ref}}={x_t^{\mathrm{human}}, x_t^{\mathrm{object}}}_{t=1}^{H}, ]
where the human term contains 3D hand keypoints and the object term contains the poses of (K) rigid bodies or articulated parts. The human hand motion is first retargeted by inverse kinematics to a robot reference (x_t^{\mathrm{robot}}). The policy then receives robot observations, object observations, and reference motion, and learns actions that make the rollout object poses track the reference.
The reward has three parts:
[ r = r_{\mathrm{task}} + r_{\mathrm{imit}} + r_{\mathrm{contact}}. ]
The task term tracks object-part poses. For multi-object interactions such as insertion, pouring, scooping, and tool use, CHORD also adds a relative object-pose reward that activates during demonstrated object-object interaction phases. The imitation term keeps the robot near the retargeted motion. The contact term is the main contribution.
For object part (k), CHORD extracts human contact positions (p_{h,k}^{i}) and normals (n_{h,k}^{i}) in the object frame. It approximates each Coulomb friction cone with (d) edge forces. Each edge force creates a primitive wrench:
[ w_{h,k}^{i,j} = \begin{bmatrix} f_{h,k}^{i,j}
p_{h,k}^{i} \times f_{h,k}^{i,j} \end{bmatrix} \in \mathbb{R}^{6}. ]
Collecting all primitive wrenches gives the human wrench matrix:
[ \mathcal{W}{h,k}\in \mathbb{R}^{6\times(c{h,k}d)}. ]
Directly comparing human and robot wrench matrices is awkward because the two hands may have different contact counts and different ordering of contact primitives. CHORD therefore compares the geometry of the induced wrench set through a support function. With pre-sampled unit basis directions (\mathcal{B}\in\mathbb{R}^{6\times b}),
[ \sigma_{h,k} = \max_{\mathrm{col}}(\mathcal{B}^{\top}\mathcal{W}_{h,k})\in\mathbb{R}^{b}, ]
and (\sigma_{r,k}) is computed the same way for the robot. The contact wrench-space reward then keeps the robot support close to the human support under a relative tolerance (\beta):
[ r_{\mathrm{cws}}^{k} = \exp\left( -\frac{\lVert\max(0,(1-\beta)\sigma_{h,k}-\sigma_{r,k})\rVert_2^2}{v_{\mathrm{cws}}} -\frac{\lVert\max(0,\sigma_{r,k}-(1+\beta)\sigma_{h,k})\rVert_2^2}{v_{\mathrm{cws}}} \right). ]
The lower-bound term encourages the robot to supply enough support in the same wrench directions as the human contact. The upper-bound term prevents excessive or mechanically unrelated support. The paper also adds penalties for unintended contacts and missed contacts because the exponential kernel alone would still give a positive value for mismatched contact states.
Training Recipe
CHORD still uses practical scaffolding around the reward. It uses a virtual object controller (VOC) during training, resets the simulator to arbitrary frames along the reference trajectory, keeps VOC active for a short stabilization window, and anneals assistance through a curriculum. The policy uses residual actions around the retargeted robot motion, so RL learns corrections instead of rediscovering the whole movement from scratch.
The paper adds two robustness choices. First, object parts are perturbed by wrenches sampled from the human contact wrench matrix, which creates disturbances aligned with the demonstrated contact mechanics. Second, when contact estimates are noisy, CHORD can switch from exact wrench matching to a reduced force-closure-style objective:
[ r_{\mathrm{fc}}^{k}=\frac{1}{B}\sum_{b=1}^{B}\mathbf{1}[\sigma_{r,k,b}>\epsilon]. ]
That fallback is less behavior-specific, but it avoids trusting corrupted contact normals and positions from noisy video reconstruction.
Benchmark and Results
The benchmark is a large part of the contribution. CHORD processes human hand-object interaction data from ARCTIC, TACO, HOT3D, OakInk2, DexYCB, GRAB, and H2O, then imports the retargeted tasks into Isaac Lab. The resulting library includes rigid objects, articulated objects, multi-object interactions, long horizons, and dense contact events.
On 1,831 sampled tasks, CHORD reports an average success rate of 82.12% with one shared training recipe across tasks. Compared with prior methods under their own evaluation protocols, CHORD matches or exceeds DexMachina, ManipTrans, and SPIDER across the tested suites. The contact reward ablations are especially clear:
| Setting | Main finding |
|---|---|
| CHORD contact wrench support | Best performance on box grabbing, mixer use, and whole-body ablations |
| Contact position reward | Weaker when the same object patch does not imply the same force/torque effect |
| No contact reward | Weakest on contact-rich manipulation because kinematic tracking alone gives poor exploration guidance |
The paper also reports that normalized contact wrench reward correlates strongly with task success across 1,831 runs, with Pearson correlation around 0.80. That is an important empirical check: the proposed reward is not just elegant mechanics; it tracks downstream manipulation performance.
For whole-body manipulation, CHORD handles both hand-only references and third-person whole-body references. Hand-only references can be expanded with a motion inpainting module that predicts whole-body motion from end-effector trajectories. For noisy third-person reconstructions, the reduced force-closure objective is used. The whole-body evaluation reports 90.77% success, and a cross-embodiment ablation shows a large gap between wrench guidance and contact position guidance when transferring five-finger human contacts to the three-fingered Dex3 hand on a Unitree G1.
Real-World Deployment
The real-world experiments use a Dexmate robot with two Sharpa dexterous hands. The paper tests both open-loop action-chunk execution and closed-loop inference. Object and robot poses are tracked with a six-camera Vicon setup, and the deployed policy runs as an ONNX model at 20 Hz while lower-level bridges hold commands at higher frequency.
This is a state-based real-world transfer result, not a vision policy. That matters for interpreting the result: the paper shows that the learned contact behavior can leave simulation, while visual perception remains future work.
Strengths and Limitations
The strongest part of CHORD is the abstraction boundary. It does not ask morphology-specific robot hands to copy human contact positions. It asks them to create contacts with similar object-level wrench support. That is a better match for dexterous manipulation because functional contact matters more than visual similarity of the hand pose.
The second strength is scale. The paper moves beyond a small set of hand-picked manipulation tasks and evaluates RL-based dexterous manipulation over thousands of long-horizon bimanual tasks. This makes the reward design feel less like a single-task trick and more like a reusable training signal.
The limitations are also concrete. Real-world deployment currently uses state-based observations from motion capture. Accurate contact guidance still depends on reasonably clean demonstrations; when contact positions or normals become very noisy, the method falls back to a less specific force-closure objective. The evaluation metric is object-pose tracking, which may miss task functionality in both directions: a small pose error can break an insertion task, while a larger pose error may be acceptable for a loose placement task.
Takeaway
CHORD’s reusable idea is: transfer human dexterous demonstrations through the mechanics of contact, not through the appearance of contact. Represent each human and robot contact set by the wrench directions it can induce on the object, compare those supports with tolerance, and use the result as a dense RL reward. For dexterous grasping and manipulation, this is a powerful shift because it respects embodiment differences while preserving the part of the demonstration that actually moves the object.