[Paper Notes] OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A hand can match the joint angles of a human-derived reference and still fail to hold the object. OATH learns the joint offsets needed to turn that reference into a physically feasible grasp. Its most useful design choice is a contact-error representation: for each hand body, the policy sees the displacement between a reference point attached to the object and its corresponding point attached to the hand. A single residual policy then handles new trajectories and held-out object geometries.
On DexYCB, the selected model achieves 58% success on held-out trajectories and 54% on unseen objects, with reported object tracking errors of 15.3 and 13.8 mm. Simulation rollout takes 0.40 seconds per second of reference motion. Hardware experiments replay the synthesized trajectories open-loop; they establish physical plausibility without measuring the same closed-loop success rates on a real robot.
Paper and source version
Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon, Joel Burdick, Dingqi Zhang, and Jiuguang Wang, affiliated with Multiply Labs, Caltech, and Carnegie Mellon University. These notes follow arXiv:2610.14379v1, submitted October 9, 2026. Sources: eight-page PDF and official project page. The manuscript is an arXiv preprint; all experimental numbers below are author-reported.
1. The residual supplies what a pose reference leaves unresolved
OATH starts with a kinematic robot trajectory and an object trajectory. At step $k$, the policy predicts a residual $\delta_k$ conditioned on the reference $r_k$ and current observations $o_k$:
\[\delta_k\sim\pi(\delta_k\mid r_k,o_k),\qquad u_k=\bar q_k^{\mathrm{robot}}+\delta_k.\]The low-level controller tracks the joint command $u_k$. The bar marks a reference quantity. Executing this policy in a physics simulator produces a dynamically retargeted trajectory $q_k^*$. The reference supplies the intended motion; the learned correction adjusts the commanded configuration so contact can support the object through that motion. Joint targets may need to push against a constrained finger to generate force, which geometric pose matching alone leaves unresolved.
The tested embodiment has 26 actuated joints: a six-DoF UR5e arm and a 20-DoF Wuji 2 hand. Arm residuals also change wrist pose, so the correction can coordinate the arm and fingers. The study concentrates on dexterous grasping for pick-and-place motions.
2. Contact correspondence becomes a state-dependent error
The reference pipeline first uses TopoRetarget to transfer interactions from the human MANO mesh to the robot hand. It replays the resulting hand and object poses kinematically in MuJoCo, with dynamics disabled, and reads collisions at zero margin. Touching or penetrating the object activates a contact slot. Multiple contacts on one hand body are reduced to their centroid.
For each of 16 bodies—the palm and three bodies per finger—the pipeline stores a contact-presence flag $\bar m_{k,i}$ and two local coordinates: $\bar g_{k,i}$ in the object frame and $\bar h_{k,i}$ in hand-body $i$’s frame. Inactive slots have a zero flag and zero coordinates. Differential inverse kinematics supplies the arm motion, and Time-Optimal Path Parameterization adjusts timing under joint velocity, acceleration, and torque constraints. The resulting reference contains motion targets and contact correspondences.
At runtime, the policy input includes the following vector, Equation 5 of the paper:
\[e_{k,i}= \left(p_{k,o}+R_{k,o}\bar g_{k,i}\right) -\left(p_{k,i}+R_{k,i}\bar h_{k,i}\right).\]Here $(p_{k,o},R_{k,o})$ is the current world pose of the object, and $(p_{k,i},R_{k,i})$ is the current pose of the corresponding hand body. The two reference points move with their respective bodies. Their world-coordinate difference tells the policy how far, and in which direction, the intended contact pair has separated.
For example, if an object slips while a finger remains near its reference joint angles, the object-attached point moves away from the hand-attached point. The contact-error vector exposes that mismatch even when joint tracking looks satisfactory. This example is an interpretation of the equation, not a separate slip-recovery experiment reported by the authors.
The representation has a fixed size across object geometries. A mug handle and a bottle surface both supply local contact targets for the same hand-body slots. “Object-agnostic” therefore refers to reuse of a policy across objects. The pipeline still needs object pose and geometry to construct the references and evaluate these errors; it does not remove the state-estimation problem.
The network also receives robot state, joint velocity, wrist state, the previous residual, and reference motion. The nominal reference horizon is one second sampled every 0.1 seconds. Table I reports 497 input dimensions and 26 action dimensions, with MLP widths of 512, 256, and 128; the method text describes temporal convolution followed by the MLP. The best-performing configuration later replaces the future object-pose window with the current object reference.
3. Learning contact behavior with a tracking reward
Contact error enters the observation. The stated tracking reward uses five other distances: object position, object orientation, hand joint angles, wrist position, and wrist orientation:
\[R_{k,\mathrm{track}}=\prod_j\exp(-\beta_j d_{j,k}).\]Position distances are Euclidean; orientation error is $\sin^2(\theta/2)$; joint error is the per-joint RMS. The multiplicative form makes a large error in one component suppress the whole tracking reward. Additional terms penalize residual changes and arm residual magnitude:
\[R_{k,\mathrm{rate}}=-w_{\mathrm{rate}}\sum_j s_j(\delta_{k,j}-\delta_{k-1,j})^2, \qquad R_{k,\mathrm{arm}}=-\frac{w_{\mathrm{arm}}}{n_{\mathrm{arm}}} \sum_{j\in\mathrm{arm}}\delta_{k,j}^2.\]The scale factors $s_j$ normalize the action-rate penalty across joints. The arm penalty decays to zero over the first 10,000 iterations, limiting disruptive shoulder exploration early in training. There is no explicit contact-force target in this reward; the policy learns useful actuation through simulated interaction and tracking feedback.
Training uses PPO with MuJoCo through mjlab, 16,384 parallel environments, a 30 Hz policy, and 120 Hz physics. Each iteration gathers 24 steps per environment. The 15,000-iteration run totals approximately 5.9 billion environment steps and ten hours on four L40S GPUs. This cost is paid before the fast retargeting rollout.
Episodes last up to five seconds and terminate when object position error exceeds 15 cm. Reference State Initialization samples a start frame anywhere in a clip, including mid-grasp, places the arm, hand, and object at the corresponding reference poses, and resets velocities and the previous residual to zero. Physical properties are randomized once per environment at startup; observation noise is sampled every control step. Those choices matter when reproducing the training distribution.
4. Read success, speed, and ablations together
The dataset contains 340 DexYCB trajectories across 20 objects. Mustard bottle, mug, bowl, and power drill are held out completely. The remaining 276 trajectories are split 80/20 into training and testing sets. After time parameterization, the average clip lasts 5.82 seconds.
Table II defines success as bringing the object within 10 cm of its goal position. The separate object tracking error measures trajectory accuracy; a sub-2 cm reported error should not be read as a 2 cm success threshold.
| Method / split | Success | Object tracking error | Compute seconds per reference second |
|---|---|---|---|
| SPIDER | 22.0% | 29.9 mm | 25.2 |
| DAID | 100% | 20.4 mm | 138 |
| ReGrind | 5.4% | 43.2 mm | 0.93 |
| CHORD | 2.0% | 7.9 mm | 0.88 |
| OATH training | 90% | 11.7 mm | 0.40 |
| OATH testing | 58% | 15.3 mm | 0.40 |
| OATH unseen objects | 54% | 13.8 mm | 0.40 |
The baselines use 50 randomly selected trajectories from the dataset. DAID generates its own kinematic references, while the other baselines use the supplied references. Compute figures are normalized to RTX 5080 processing speeds because every evaluation could not run serially on the same machine. These conditions limit an interpretation of the table as a controlled comparison of controller architecture alone.
OATH trades some of DAID’s success for much faster rollout. The rounded table values imply roughly 63× and 345× lower compute than SPIDER and DAID. The paper describes rollout as 2.4× faster than real time. End-to-end processing also includes approximately four seconds of kinematic retargeting plus 2.3 seconds of rollout per average clip, reported as about 6.5 seconds overall. A faster-than-real-time rollout does not establish the same throughput for the complete pipeline.
The observation ablation in Table III is more revealing about the mechanism:
| Reference / observation variant | Training success | Testing success | Unseen success |
|---|---|---|---|
| All listed terms | 89% | 53% | 46% |
| Remove contact flag and contact error | 69% | 31% | 38% |
| Remove robot reference window | 89% | 47% | 52% |
| Use current robot reference instead of its window | 89% | 56% | 52% |
| Use current object reference instead of its window | 90% | 58% | 54% |
Removing contact inputs from the all-terms row reduces success by 20, 22, and 8 percentage points. The paper’s prose gives drops of 21, 27, and 16 points, which correspond to comparison against the selected best row. Keeping that distinction avoids attributing the entire best-model gap to one controlled removal. Both comparisons support the usefulness of contact information, while additional future poses offer little benefit in these experiments. Table II uses the selected best configuration.
5. Reference quality sets a practical limit
The paper illustrates fingers penetrating the table, spurious approach contacts, missing grasp contacts, and floating object references that fall when physics starts. These failures expose a dependence on the reference construction: a policy guided toward the wrong contact can receive a precise error signal for an unhelpful target.
Hardware trials use open-loop joint playback on the UR5e and Wuji 2. Figure 7 shows four held-out trajectories, including the unseen mug. The authors identify contact slip and calibration offsets as sources of accumulating error and leave tactile or visual feedback for more reliable execution. The real-robot evidence is qualitative; the 58% and 54% success figures belong to simulation.
My main takeaway is that contact correspondence is a useful interface between demonstration processing and control. For a system that already has object geometry, object state, and reasonable grasp references, I would first test this compact error signal before adding a more elaborate geometry encoder. If contact references are incomplete or wrong, improving their extraction becomes the more urgent experiment. The present evidence covers grasping across a small set of held-out household objects; broader dynamic manipulation remains an open test.