[Paper Notes] REGRIND: A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
REGRIND asks whether the familiar humanoid-control recipe—retarget a human motion, train an RL policy to track it, then deploy the policy on hardware—also works for contact-rich dexterous manipulation. Its answer is yes, provided that the retargeted reference preserves hand-object interaction, the RL policy stays close to that reference through residual control and reference-state initialization, and the simulator is carefully aligned with the real system.
The hand-retargeting component is the center of the pipeline. REGRIND combines 21 semantic hand keypoints with 50 object keypoints into an interaction mesh, then minimizes the change in their Laplacian coordinates. This objective preserves the local spatial arrangement between fingers and object surfaces across the human-to-robot embodiment gap. Joint limits, velocity bounds, and non-penetration constraints keep the reference robot-feasible. The resulting trajectory provides both a human-like nominal action and a high-value exploration distribution for RL.
The full recipe learns from a single human demonstration, reaches roughly 99% success on four simulated task-hand settings, and transfers successfully on three of them: LEAP-Scissors 9/10, LEAP-Screwdriver 10/10, and WUJI-Screwdriver 9/10. The unsuccessful WUJI-Scissors case is equally informative: inaccurate object geometry and non-backdrivable hand motors create a dynamics gap that good retargeting alone cannot remove.
Paper Info
The paper is “A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation” by Yunhai Feng, Natalie Leung, Jiaxuan Wang, Lujie Yang, Haozhi Qi, and Preston Culbertson, from Cornell University and Amazon FAR. REGRIND stands for REtargeting-Guided ReINforcement learning for Dexterous manipulation.
- Paper: arXiv:2607.11874
- Project page, videos, and code: yunhaifeng.com/REGRIND
1. The Pipeline in One Equation Chain
REGRIND follows a real-to-sim-to-real pipeline:
[ \text{human hand-object motion} \rightarrow \text{interaction-preserving robot reference} \rightarrow \text{residual RL tracking in simulation} \rightarrow \text{zero-shot hardware deployment}. ]
The human demonstration contains MANO hand keypoints and the object configuration at every timestep. For a rigid object, the configuration is a 6D pose; an articulated object additionally includes its joint state. The retargeter converts this demonstration into a wrist-and-finger trajectory for the robot. RL then learns the dynamic corrections needed to execute the kinematic reference under gravity, friction, contact, motor lag, and disturbances.
The division of labor is important:
- Retargeting specifies the hand-object strategy and contact arrangement.
- Residual RL supplies force-producing corrections and robust closed-loop execution.
- System identification and domain randomization narrow the simulation-to-reality gap.
REGRIND is minimalist because it avoids task-specific contact rewards and large collections of robot demonstrations. Most of the task structure is already encoded by the retargeted reference.
2. Why Ordinary Hand IK Is a Weak Reference
A human hand and a robot hand differ in palm proportions, finger lengths, joint axes, ranges of motion, and available contact surfaces. Matching corresponding fingertips can therefore produce a pose that looks kinematically plausible while placing the robot fingers inside the object, around the wrong handle, or on an unstable side of the tool.
This matters twice in a retargeting-guided RL system. First, the reference is added directly to the policy action, so a poor reference biases every control target. Second, reference state initialization resets the simulator to states along that trajectory. A penetrated or semantically wrong grasp then becomes the policy’s exploration distribution. The reference can accelerate learning only when it lies near a useful state-visitation distribution.
REGRIND therefore evaluates retargeting by the structure it preserves between the hand and the object. The target becomes a robot hand-object interaction with the same local geometry as the demonstrated human interaction.
3. Semantic Hand-Object Correspondence
At timestep (t), the demonstration provides human hand keypoints (P_t^h) and object keypoints (P_t^o). Their union forms the source point set:
[ \widetilde P_t = P_t^o \cup P_t^h. ]
Given robot configuration (q_t), forward kinematics produces robot keypoints (P_t^r(q_t)) that semantically correspond to the MANO hand keypoints. The target point set is:
[ P_t(q_t) = P_t^o \cup P_t^r(q_t). ]
The object points appear in both sets. This shared object frame anchors the comparison: the optimizer measures where each robot hand region sits relative to the same task geometry seen in the human demonstration.
The implementation uses 21 hand keypoints following MANO. It samples 50 object keypoints. For scissors, sampling is concentrated on the handle region where contact occurs; for the screwdriver, points are sampled over the full surface. The authors manually define corresponding points on the LEAP and WUJI robot models.
This representation expresses task semantics without prescribing contact forces. A human thumb point, for example, is paired with a robot-thumb point, while nearby object samples specify which side and region of the tool that point should approach.
4. Interaction Mesh and Laplacian Coordinates
The source and target point sets are converted into interaction meshes through Delaunay tetrahedralization. For a mesh (M(P)=(P,E)), the Laplacian coordinate of vertex (i) is
[ L_i(P) = p_i- \frac{1}{|\mathcal N_i|} \sum_{j\in\mathcal N_i}p_j, ]
where (\mathcal N_i) contains the neighbors of vertex (i).
This quantity describes a point relative to the center of its local neighborhood. When that neighborhood contains both hand and object vertices, it encodes local finger-object geometry: which hand region lies near which object region, along with their local relative arrangement. It is translation-invariant and more compatible with embodiment changes than copying absolute human keypoint coordinates.
REGRIND measures deformation between the human and robot interaction meshes by
[ D!\left(M(\widetilde P_t),M(P_t(q_t))\right) = \sum_i \left| L_i(\widetilde P_t)-L_i(P_t(q_t)) \right|_2^2. ]
Minimizing this energy encourages the robot to preserve the demonstrated hand-object spatial and contact relationships. This is the main difference from a hand-only IK objective: object geometry participates directly in the retargeting metric.
5. Trajectory Optimization and Feasibility
The robot configuration is (q_t=(T_t^r,q_t^r)), containing the floating wrist pose and all actuated finger joints. The complete trajectory is optimized as
[ \bar q_{0:T-1} = \arg\min_{q_{0:T-1}} \sum_{t=0}^{T-1} D!\left(M(\widetilde P_t),M(P_t(q_t))\right) + \lambda\sum_{t=1}^{T-1} |q_t-q_{t-1}|2^2, \qquad q{0:T-1}\in\mathcal Q. ]
The deformation term preserves interaction. The temporal term suppresses frame-to-frame jitter. The feasible set (\mathcal Q) imposes hard constraints:
[ \mathcal Q= \left{ q_t: q_{\min}\le q_t\le q_{\max}, \quad v_{\min}\Delta t\le q_t-q_{t-1}\le v_{\max}\Delta t, \quad \phi_j(q_t)\ge 0 \right}. ]
These constraints enforce joint limits, per-frame velocity limits, and non-penetration for robot-object and robot-environment collision pairs. The demonstration object pose stays fixed during optimization. In practice, the authors solve the trajectory sequentially: each frame is warm-started from the previous solution, then refined with a small number of SQP iterations using Drake and MOSEK.
The output remains a kinematic trajectory: smooth, collision-free, and equipped with useful contact geometry. RL supplies dynamic feasibility during execution.
6. How the Retargeted Motion Guides RL
The retargeted trajectory is used in three coupled ways.
6.1 Residual Action Prior
The policy predicts a scaled correction on top of the reference:
[ q_t^{\text{target}} = \bar q_t + \alpha\odot \pi_\theta(\bar q_t,o_t). ]
The target is sent to a low-level PD controller. Initializing the actor’s final layer around zero makes training begin close to the retargeted motion. RL can then adjust finger and wrist targets to produce the forces and timing left outside the kinematic optimizer.
6.2 Reference State Initialization
At reset, REGRIND samples a random phase along the retargeted trajectory and initializes both robot and object near the corresponding reference state. This places exploration around states that already express a plausible manipulation strategy. Episodes terminate when the object keypoint error exceeds 15 cm, since a small residual policy is unlikely to recover after a large departure from the reference.
6.3 Dense Object-Centric Reward
The main tracking error is the mean distance between current and reference object keypoints:
[ \epsilon_{\text{object}} = \frac{1}{N_k} \sum_{i=1}^{N_k} \left| p_{t,i}^o-\bar p_{t,i}^o \right|_2, ]
with shaped reward
[ r_{\text{object}} = \exp!\left(-\epsilon_{\text{object}}/\sigma\right). ]
Object keypoints provide one metric for rigid and articulated objects. The remaining reward terms track object linear and angular velocity and wrist pose, while regularizing action magnitude and action rate. The policy receives no explicit contact schedule or contact reward; the interaction prior is carried by the retargeted reference.
7. Observation and Training Design for Sim-to-Real
REGRIND uses asymmetric actor-critic observations. The actor receives the object pose and articulated joint state, current and previous robot joint positions, the previous action, and a normalized phase variable. The critic additionally observes robot fingertip positions and joint velocities.
The actor deliberately avoids measured velocity. Joint and object velocities are often noisy or delayed on hardware, so excluding them reduces observation mismatch. Object states come from a nine-camera motion-capture system during deployment, which lets the study isolate dynamics transfer from perception error.
Training uses PPO with 4,096 parallel Isaac Sim environments. Simulation runs at 120 Hz, while the policy acts at 30 Hz. Domain randomization covers object center of mass, geometry, mass, hand/table/object friction, PD gains, default joint offsets, observation noise, and 0–2 control steps of latency. A curriculum increases gravity from zero to full gravity and later introduces random pushes.
The gravity curriculum is especially relevant for pickup tasks. Early training can learn the motion structure before full gravitational forces destabilize contact; later stages require the learned residuals to maintain the grasp under realistic loading.
8. Dynamic Augmentation from One Demonstration
A single demonstration covers only one initial object pose. REGRIND perturbs the initial object position by up to (\pm5) cm and yaw by up to (\pm30^\circ). It then applies a time-varying rigid transform to the object pose and hand root pose:
[ p^\star(t)=p_t^{\text{ref}}+w(t)\Delta p, \qquad R^\star(t)=R_z!\left(w(t)\Delta\psi\right)R_t^{\text{ref}}. ]
The weight (w(t)) stays at one before pickup, linearly decays between pickup and tool use, and becomes zero afterward. The augmented reference therefore starts from a perturbed object configuration and smoothly rejoins the original task goal.
The same rigid transform is applied to the hand wrist and fingertip positions; finger joint references stay unchanged. REGRIND generates each sample without rerunning retargeting. Because hand and object are transformed together, their relative spatial and contact relationships remain intact, and the trainer can generate effectively unlimited reference variants online.
9. Experiments: Does Interaction Preservation Matter?
The evaluation covers two tasks and two hands:
- Scissors: pick up the scissors, hold them at the target orientation, and complete an opening and closing sweep of at least (20^\circ) each.
- Screwdriver: keep the screwdriver upright and rotate it by at least one full turn.
- LEAP: 16 finger DoFs, using enlarged 3D-printed tools due to hand size.
- WUJI: 20 finger DoFs, using real tools.
The baselines are SPIDER, DexMachina, and Mink IK followed by the same RL training setup.
Simulation Results
| Method | LEAP-Scissors | LEAP-Screwdriver | WUJI-Scissors | WUJI-Screwdriver |
|---|---|---|---|---|
| REGRIND | 99.8% | 99.7% | 98.7% | 98.8% |
| SPIDER | 0.0% | 0.0% | 0.0% | 0.0% |
| DexMachina | 22.3% | 99.7% | 0.0% | 99.3% |
| Mink IK + RL | 2.0% | 0.0% | 0.0% | 3.1% |
REGRIND also keeps object-keypoint tracking error between 5.3 and 6.5 mm across the four settings. DexMachina succeeds on the screwdriver tasks, where the grasp structure is simpler, but breaks down on scissors. Qualitative retargeting reveals why: IK-based solutions contain severe hand-object penetration, and collision projection can move fingers in a direction that destroys the intended grasp.
The comparison supports a specific conclusion. A reference must carry useful interaction geometry before residual RL can exploit it. Physics plausibility, hand-pose similarity, and collision cleanup can still yield a weak exploration prior.
Real-World Results
| Method | LEAP-Scissors | LEAP-Screwdriver | WUJI-Scissors | WUJI-Screwdriver |
|---|---|---|---|---|
| REGRIND | 9/10 | 10/10 | 0/10 | 9/10 |
| DexMachina | 0/10 | 2/10 | — | 5/10 |
| Mink IK + RL | — | — | — | 0/10 |
DexMachina’s screwdriver policies perform well in simulation yet transfer poorly. The authors hypothesize that weaker interaction regularization leaves more room for policies to exploit simulator artifacts, creating aggressive motion, unstable grasps, table collisions, and overshoot on hardware.
For randomized initial poses, REGRIND obtains 8/10 on LEAP-Scissors, 10/10 on LEAP-Screwdriver, and 9/10 on WUJI-Screwdriver. These results are close to the nominal-start results and validate the simple trajectory-warping augmentation.
10. What the Failed WUJI-Scissors Task Teaches
WUJI-Scissors achieves 98.7% success in simulation and 0/10 in reality. The paper attributes the gap mainly to two factors: the WUJI hand uses non-backdrivable motors, and the real scissors mesh is inaccurate. Both factors directly affect contact-rich behavior. Motor backdrivability changes compliance under impact and sustained force; mesh error shifts when and where contact occurs.
The system-identification procedure also finds a real-robot response delay of roughly 1–2 policy steps, or 30–60 ms. REGRIND models this delay during training. The broader lesson is that the closer a task operates to contact and friction limits, the more exact its simulation needs to be. Interaction-preserving retargeting gives the policy the right strategy, while system identification determines whether that strategy survives hardware contact.
11. Strengths and Limitations
Strengths. REGRIND is unusually clear about the complete chain from human data to real robot behavior. The interaction-mesh objective gives the reference an interpretable role. Residual actions, reference-state initialization, and object-centric rewards all reuse the same trajectory, making the design coherent. The real-robot ablations also expose simulation-only success as insufficient evidence for dexterous manipulation.
Limitations. The current system depends on motion capture for object state during deployment, so perception is outside the evaluation. The retargeter uses manually defined semantic keypoint correspondences and task-dependent object sampling; scaling to arbitrary hands and objects will require automated correspondence and contact-region discovery. The kinematic objective omits forces, friction cones, and compliance. Real-world transfer still requires careful per-platform system identification, and one of four task-hand combinations fails completely on hardware.
The study also uses one demonstrated trajectory per task. Dynamic augmentation broadens the initial pose distribution while staying within the same contact strategy and tool-use mode; recovery behaviors far from the reference remain uncovered.
12. My Takeaway
REGRIND’s strongest insight is that retargeting quality shapes the entire RL problem. A reference trajectory is simultaneously a nominal action, an exploration distribution, and an implicit contact prior. Errors in that trajectory therefore enter the learner three times.
For hand retargeting, the useful target is the local relationship among robot fingers and object surfaces. The interaction mesh and Laplacian objective provide a compact geometric representation of that relationship. Hard feasibility constraints keep it usable as a simulator reset state. Residual RL then concentrates on dynamics and robustness.
If TopoRetarget is read primarily as a retargeting-method paper, REGRIND is best read as a system recipe and an empirical study of what happens after retargeting. It shows that a strong interaction-aware reference can make single-demonstration dexterous RL work, while the final gap to hardware is governed by contact-sensitive system identification.