[Paper Notes] FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
Converting a human manipulation dataset into robot trajectories becomes expensive when every demonstration needs its own optimization run. FlashDexRetarget trains across many reference motions at once, sharing experience while conditioning on object geometry and upcoming motion. Its implementation uses separate left- and right-hand actor–critic pairs, with each pair shared across the reference collection.
On 50 motions with XHand, the paper reports 90% success under SR_SPIDER at 29 GPU-hours, compared with CHORD’s 46% at 2,847 GPU-hours. Under the stricter object-tracking metric SR_MT-Obj, the corresponding results are 86% and 2%. The practical contribution is the throughput of converting a known reference collection into usable trajectories. The experiments do not establish zero-shot retargeting of unseen demonstrations.
Paper and source version
Kyungmin Lee, Sibeen Kim, Dongyoon Hwang, Yoonsang Oh, Donghu Kim, Youngdo Lee, I Made Aswin Nahrendra, Jaegul Choo, and Hojoon Lee, KAIST AI and Holiday Robotics. These notes follow arXiv:2610.01849v2, revised October 4, 2026, especially Sections III–IV and Table I. See the PDF, official project page, and official implementation. The paper is treated here as an arXiv preprint; results below are author-reported.
1. Share optimization across reference motions
Each demonstration contains human hand states and object poses over time. During training, different references are assigned to parallel physics environments. A reference-conditioned controller observes the simulated interaction, tracks the next reference state, and receives an encoding of the next ten reference frames. Successful rollouts are saved as robot data.
Using $i$ for the selected reference and $h$ for the hand, the controller can be summarized as
\[a_t^h\sim\pi_{\theta_h}(\cdot\mid s_{i,t}), \qquad a_t=[a_t^{\mathrm{left}};a_t^{\mathrm{right}}], \qquad h\in\{\mathrm{left},\mathrm{right}\}.\]The parameters $\theta_h$ are reused across all reference motions. Both hands receive the full observation, including the other hand’s state, but each actor emits its own hand’s action and each actor–critic pair learns from its own reward. This preserves information needed for coordination while separating the learning signals for roles such as holding a container and manipulating a tool. “Single policy” in the paper therefore means a shared controller across demonstrations, implemented with two hand-specific agents.
flowchart TD
A["Human hand-object reference collection"] --> B["Assign references to parallel physics environments"]
B --> C["Current state, object geometry, next target, future 10 frames"]
C --> D["Left actor-critic shared across references"]
C --> E["Right actor-critic shared across references"]
D --> F["Joint simulated interaction"]
E --> F
F --> G["Hand-specific tracking rewards and replay"]
G --> D
G --> E
F --> H["Store successful robot trajectories"]
2. Geometry and future motion make the reference identifiable
The observation combines robot and object states with 128 sampled object-surface points, represented at both the current simulated pose and the next reference pose. Geometry features are expressed in the wrist frame. Signed distances from the five fingertips and wrist to the object, together with surface normals, describe the local interaction. Distances are queried from a precomputed $128^3$ SDF grid covering a 60 cm cube, avoiding a mesh-distance calculation at every simulation step.
The future-motion input is
\[z_{i,t+1}^{\mathrm{ref}} =\operatorname{Enc}_{\psi}\!\left(x_{i,t+1:t+K}^{\mathrm{ref}}\right), \qquad K=10,\qquad z^{\mathrm{ref}}\in\mathbb R^{128}.\]These future hand and object states are relative to the wrist frame at time $t$. The temporal encoder trains jointly with the actor and critic, and its latent supplements the explicit next-frame target. A single pose may occur before a lift, a rotation, or a release; the future window helps distinguish those continuations. The ablation removes this encoding while retaining the next reference frame and finds slower accumulation of successful conversions.
3. Two geometric rewards carry the interaction
For each hand, the reward combines object tracking, hand–object interaction, hand tracking, and regularization:
\[r_t^h=w_{\mathrm{obj}}r_t^{\mathrm{obj},h} +w_{\mathrm{int}}r_t^{\mathrm{int},h} +w_{\mathrm{track}}r_t^{\mathrm{track},h} +w_{\mathrm{reg}}r_t^{\mathrm{reg},h}.\]The object term measures how corresponding surface points move under the simulated and reference poses. First, object-frame points are scaled by the maximum radius:
\[\widetilde p_m=\rho\frac{p_m}{\max_n\|p_n\|_2}, \qquad \rho=0.058\ \mathrm m.\]This scales the whole point set while preserving its shape. After transforming the points into the world frame under the two poses, let $\mathcal I_t^{(3)}$ index the three largest pointwise discrepancies. Suppressing the hand index, the reward is
\[e_t^{\mathrm{obj}}=\frac13\sum_{m\in\mathcal I_t^{(3)}} \left\|\widetilde P_{t+1,m}^{\mathrm{cur}} -\widetilde P_{i,t+1,m}^{\mathrm{ref}}\right\|_2, \qquad r_t^{\mathrm{obj}}=\exp\!\left(-\frac{e_t^{\mathrm{obj}}}{0.1\ \mathrm m}\right).\]Translation and rotation now contribute through the same displacement unit. At the chosen radius, a 30° rotation can produce roughly 3 cm of surface-point displacement. Averaging the worst three discrepancies makes large local misalignments matter. The comparison uses corresponding sampled points; it does not perform a new nearest-neighbor point-cloud registration at each step.
The interaction reward addresses errors in reconstructed human contact. A fingertip that should touch an object may appear a small distance away. A binary contact label can miss that interaction, while exact distance matching would preserve the gap. FlashDexRetarget uses the demonstrated distance as a one-sided proximity target:
\[\epsilon_{h,t}^{(f)} =\max\!\left(0,d_{h,t}^{(f)}-d_{h,t}^{\mathrm{ref},(f)}\right), \qquad r_t^{\mathrm{int},h}=\frac15\sum_{f=1}^{5} \exp\!\left(-\frac{\epsilon_{h,t}^{(f)}}{0.01\ \mathrm m}\right).\]Here $d^{(f)}$ is the robot fingertip’s signed SDF distance, while $d^{\mathrm{ref},(f)}$ is the reference fingertip’s unsigned distance to the nearest object-mesh vertex. The wrist contributes to the observation but is excluded from this reward. A robot fingertip receives maximal reward whenever its signed distance is no greater than the demonstrated gap. It can therefore move closer and establish contact.
For example, with a 5 mm reference gap, robot distances of 2 mm and 5 mm both receive a per-fingertip reward of 1; a 15 mm distance receives $e^{-1}$. This is an illustration of Equation (7), not an experimental result. It also exposes the term’s boundary: a negative signed distance satisfies the inequality too. Physical contact handling and the rest of the objective remain necessary; this term alone is not a nonpenetration constraint.
The hand-tracking term rewards wrist translation, wrist rotation, and fingertip alignment, with a behavior-cloning term pulling commanded joints toward kinematic retargets. Regularization penalizes large actions and tracking failures. Timeouts and successful clip completion do not incur the failure penalty.
4. Experience reuse needs enough buffer and critic capacity
FlashDexRetarget adapts the off-policy learner FlashSAC to the broader state distribution created by many references. It enlarges the replay buffer from 10 million to 50 million transitions and widens the critic’s hidden dimension from 256 to 1,024. Sharing training introduces a capacity problem: the replay must retain varied interactions, and the critic must estimate their values well enough to improve the policy.
The XHand ablation uses 300 million environment steps and cumulative SR_MT-Obj conversions:
| Training configuration | Approximate conversion success |
|---|---|
| PPO in the multi-reference setting | Near 0% |
| Default FlashSAC | 34% |
| FlashSAC with 50M replay capacity | 60% |
| Larger replay and wider critic | 86% |
These are the approximate values described in Section IV-C. Equal environment-step budgets do not imply equal GPU-hours. The experiment supports the complete learning configuration for this setting; it does not isolate a universal advantage of one RL algorithm across tasks.
The geometry ablations also change multiple ingredients together. Removing point-cloud matching removes point-cloud observations and replaces the object reward with DexMachina’s formulation. Removing the interaction reward also removes the distance observations. Their degradation therefore measures the contribution of each observation–reward package. It cannot be attributed solely to the reward equation.
5. Read success rates together with their thresholds
The main benchmark contains 25 single-object and 25 two-object motions from HOT3D, TACO, and OakInk2. Inactive clips are excluded. Experiments use Isaac Sim and NVIDIA RTX 3090 GPUs; FlashDexRetarget receives 300 million environment steps for XHand and 400 million for Sharpa Wave Hand.
Table I reports the following results:
| Embodiment | Method | SR_SPIDER | SR_MT-Obj | SR_MT | GPU-hours |
|---|---|---|---|---|---|
| XHand | Do as I Do | 36% | 4% | 0% | 66 |
| XHand | DexMachina | 48% | 32% | 12% | 447 |
| XHand | CHORD | 46% | 2% | 0% | 2,847 |
| XHand | FlashDexRetarget | 90% | 86% | 86% | 29 |
| Sharpa Wave Hand | Do as I Do | 64% | 10% | 10% | 62 |
| Sharpa Wave Hand | CHORD | 50% | 22% | 0% | 3,314 |
| Sharpa Wave Hand | FlashDexRetarget | 86% | 80% | 80% | 44 |
SR_MT-Obj uses object position and rotation thresholds of 3 cm and 30°. SR_MT additionally checks fingertip and hand-joint position errors against 6 cm and 8 cm thresholds. These latter quantities are positional errors, not angular joint limits. SR_SPIDER uses 10 cm and 0.5 rad object thresholds and averages object errors over active hands. That averaging can hide one object’s error when another barely moves, so the authors use SR_MT-Obj for their component studies.
The headline comparison is specific: $2847/29\approx98.2$, with a 44-percentage-point SR_SPIDER gain on XHand. Sharpa’s compute ratio is $3314/44\approx75.3$, and its success gain is 36 points. The baseline CHORD uses its released PPO implementation; DexMachina is modified to initialize from random reference states. Those implementation choices belong with the comparison.
The paper measures cumulative conversion: once a qualifying rollout is saved, that reference counts as converted. Section IV-A defines this stored-rollout accounting using SR_MT-Obj, while Table I separately reports all three success metrics. The 90% SR_SPIDER result should therefore retain its metric label. None of these numbers should be read as the final policy’s repeated-execution reliability. Single-reference compute sums all attempted motions, including unsuccessful runs; the multi-reference method uses a fixed training budget.
Object errors in Table I are also conditional on SR_SPIDER success. For XHand, FlashDexRetarget’s mean position error is 10.95 mm versus CHORD’s 55.70 mm, but mean rotation error is 0.25 rad versus 0.16 rad. Each mean is computed over that method’s successful subset. Higher coverage and lower positional error do not imply superiority on every reported error metric.
6. More converted data, with deployment still to validate
The scaling study trains on 200, 500, and 1,000 references, each with a 600-million-step budget. Figure 6 compares successful conversions during the first 60 GPU-hours. Larger collections yield more successful trajectories within that compute window. This demonstrates improved conversion throughput; the plot does not establish a constant success percentage as dataset size grows or generalization to held-out motions.
Real-world evidence consists of replaying successful trajectories for wiping a board, pouring into a pan, and closing a lid. The paper provides qualitative executions without a trial-count success table. Its conclusion identifies dependence on relatively clean human trajectories, accurate object geometry, and privileged simulator state. Deployment from real observations, transfer to unseen motions or embodiments, and task-aware evaluation remain open.
I would use this method first as a dataset-conversion stage when object meshes and reconstructed trajectories are available. The one-sided distance reward is especially useful when small reconstruction gaps make contact labels unreliable. For noisier video reconstructions or closed-loop real-world deployment, the next decisive test would be repeated task completion under state-estimation errors. The current evidence is strongest for producing reference-conditioned robot data in simulation, with selected trajectories demonstrated on hardware.