[Paper Notes] Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking
Published:
TL;DR
AdaDex learns an adaptable dexterous skill from one short hand–object motion reference. A shared goal-conditioned policy trains in two environment populations: one rewards accurate tracking, while the other perturbs states and goals so the policy must depart from the reference.
The combination preserves human motion structure while improving generalization. On similar unseen GRAB motions, tracking progress rises to 80.02%, compared with 31.75% for the tracking-only ablation. On random goals, however, AdaDex’s complete ten-goal success rate is lower than SimToolReal’s. The method improves the tradeoff; it does not lead every metric.
Paper and reading version
Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking — Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye, Sungjae Park, Umar Iqbal, Katerina Fragkiadaki, Shubham Tulsiani. These notes follow arXiv:2610.28628v1, an arXiv preprint, checked on October 25, 2026. PDF · Project
AdaDex is the method name for this paper. These notes focus on Sections 3–4, Tables 1–2, and Appendix C. It is distinct from papers using names such as AdaDexTrack.
Replace the reference clock with a goal interface
Training starts with a short trajectory of wrist pose, hand configuration, and object pose, plus a watertight object mesh. The reference may come from motion capture or hand–object reconstruction from RGB(D) video.
The policy receives robot joint positions, wrist pose and velocity, object pose, and a Basis Point Set encoding of object geometry. Its goal has two parts:
\[g_t=(g_t^w,g_t^o)\in SE(3)\times SE(3).\]It outputs a six-dimensional wrist increment and absolute finger-joint targets. The actor does not need the dense reference trajectory at deployment. A sequence of wrist and object goals can come from the original reference, a different trajectory, or a high-level planner.
This interface is important: tracking experience and off-reference practice must train the same actor on the same observation and action spaces. Otherwise, success in the tracking environments would not necessarily become a reusable goal-conditioned skill.
Two environment distributions train the same behavior
Tracking environments initialize near sampled reference states and use a short look-ahead reference pose as the goal. Their dense reward includes object and wrist tracking, hand-link alignment, contact, and regularization:
\[r_t^{\rm track} =\sum_i w_i\exp(-\|e_t^i\|^2/\sigma_i^2) +r_t^{\rm contact}+r_t^{\rm reg}.\]The object terms define task progress; hand and contact terms transmit approach direction, grasp configuration, finger gaiting, and release timing. Strict termination keeps the motion close to the reference. Adaptive reference-state sampling concentrates practice on phases where success remains variable.
Generalization environments perturb initial hand/object states and future goals much more strongly. Their objective combines sparse object success with dense wrist guidance and regularization:
\[r_t^{\rm gen} =\mathbf 1[\|p_t^o-g_t^{o,p}\|<\delta_p \land \angle(R_t^o,g_t^{o,R})<\delta_R] +r_t^{\rm wrist}+r_t^{\rm reg}.\]There is no finger-tracking term in this population. The shared policy must draw on finger behavior acquired from tracking while adapting to goals outside the demonstrated path. Calling the entire generalization reward sparse would omit the wrist guidance.
PPO optimizes rollouts from both populations:
\[J(\theta)=p\,\mathbb E_{\rm track}\!\left[\sum_t\gamma^t r_t^{\rm track}\right] +(1-p)\,\mathbb E_{\rm gen}\!\left[\sum_t\gamma^t r_t^{\rm gen}\right].\]The privileged critic receives simulator information and the reference during training; its reference input is zeroed in generalization environments. The actor uses the common goal interface without an explicit environment-type flag.
The training curriculum matters. Appendix C starts with tracking only, then uses a 50:50 TrackEnv/GenEnv mixture in Stage II. Thus the final hybrid objective follows an initial tracking phase. A single policy here means shared parameters across the two populations for a skill; the quantitative study still trains one policy per reference motion.
Separate tracking progress from complete goal-sequence success
The main simulation benchmark selects twelve GRAB objects with three similar clips per object. One clip trains each policy and the other two test related unseen motions. Evaluation uses 128 rollouts per motion on a floating 22-DoF Arti-MANO hand with wrist control.
Tracking progress is the fraction of the clip completed before the first object-position error above 5 cm or orientation error above 20°. It is not a binary success rate.
| Setting | Tracking-only ablation | AdaDex |
|---|---|---|
| Seen-motion tracking progress | 97.47% | 96.74% |
| Similar unseen-motion tracking progress | 31.75% | 80.02% |
Adding generalization practice sacrifices a small amount of seen-motion tracking and produces a much larger improvement on related unseen motions. It still depends on the object-specific reference and training setup; this is not a universal policy trained once across arbitrary objects.
The random-goal test chains ten procedural goals. AdaDex achieves 51.80% goal progress, versus SimToolReal’s 44.60%, but 27.08% complete-sequence success, versus 31.71%. More average progress does not guarantee more fully completed sequences. On nine reconstructed RGB-D references, reported mean tracking progress is 80.53%, compared with ManipTrans’s 31.70%, using a relaxed failure rule that requires ten consecutive threshold violations.
What transfers to a robot
The paper demonstrates transfer from Isaac Gym to MuJoCo and real hardware. Physical tests include both replaying simulated joint targets at original 30 Hz speed and closed-loop control. In the closed-loop case, FoundationPose estimates object pose using an object mesh, RGB images, and Fast-FoundationStereo depth. The policy also receives proprioceptive history and object geometry.
These demonstrations establish physical feasibility for selected grasps, reorientation, and tool behaviors. The main numerical comparisons remain simulation results; the paper does not provide a matching large real-robot success table. Object models, pose estimation, and system identification remain practical dependencies.
A high-level coding agent can generate wrist/object waypoints for downstream tasks while the skill handles finger control. My preferred follow-up would fix the planner and compare policies with and without generalization environments under the same pose-estimation errors. That would show whether the generalization gain survives the uncertainty introduced by the real perception pipeline.