[Paper Notes] DexWeave: Learning Dexterous Humanoid Loco-Manipulation from Human Demonstrations
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A human wrist can be retargeted accurately while the robot’s fingers miss the object. DexWeave addresses this coupling twice: it jointly refines arms, wrists, and fingers when constructing robot references, then uses anatomical tokens and directed attention to coordinate their execution. The lower-body support trajectory stays fixed during geometric refinement; during policy learning, the legs receive body context through a pathway without direct object conditioning.
On five loco-manipulation references, DexWeave reports 97.5% episode completion in MuJoCo, compared with 85.0% for an object-conditioned MLP. It trains one policy per reference. Physical G1 demonstrations provide qualitative transfer evidence; the 97.5% figure is a simulation result with a reference-tracking completion criterion.
Paper and source version
DexWeave: Learning Dexterous Humanoid Loco-Manipulation from Human Demonstrations is by Naichuan Sun, Haotian Shen, Yizhang Zhang, Luying Feng, Haoze Wang, Yuanbo Xiangli, Yaochu Jin, and Peidong Liu, from Westlake University and Shanghai Jiao Tong University. Sun and Shen contributed equally.
These notes follow the September 28, 2026 preprint arXiv:2609.34724v1, including its 26-page PDF and appendices. The project website links videos and an official repository. Experimental numbers below are author-reported; I have not reproduced the training runs.
1. Preserve the interaction while changing the embodiment
DexWeave starts with body motion, articulated hand observations where available, and object geometry and pose trajectories. GRAB and HUMOTO supply detailed hand–object interaction; OMOMO supplies body–object motion without observed fingertip targets; LAFAN1 supplies body-only motion. This is a pipeline for structured demonstrations with scene information. Recovering all of that information from arbitrary video remains a separate problem.
The robot is a Unitree G1 with Inspire RH56DFQ hands: 29 body joints and 12 independent hand drivers. Fixed mimic relations expand the hand commands to 24 articulated finger joints. A reference contains robot configuration $q_t^*$ and object pose $X_t^O$ at 50 Hz. Object trajectories stay fixed during retargeting, after common coordinate alignment and any necessary scene scaling.
flowchart TD
A["Human body, hand and object demonstration"] --> B["Contact-aware body retargeting"]
A --> C["Hand codebook retrieval and local fitting"]
B --> D["Coupled arm, wrist and finger refinement"]
C --> D
D --> E["Robot and object reference at 50 Hz"]
E --> F["One PPO policy per reference"]
G["Robot state and previous action"] --> F
H["Measured and reference object poses"] --> F
F --> I["41 joint-position commands and PD control"]
Wrist targets transition from body proportions to scene alignment
A robot-scaled human pose may put the wrist beyond a handle or inside an object. DexWeave blends a morphology-adapted wrist target with the wrist position in the preprocessed scene:
\[\hat p_{t,s}=(1-\alpha_{t,s})p^{\mathrm{morph}}_{t,s} +\alpha_{t,s}p^{\mathrm{scene}}_{t,s},\qquad s\in\{L,R\}.\]The interaction weight $\alpha$ increases with persistent hand–object proximity. The appendix uses a 5 cm proximity threshold and a nominal transition interval of $4/30$ seconds, discretized to seven frames at 50 Hz; GRAB and HUMOTO also use source finger-contact labels. Far from an object, morphology adaptation dominates. Near sustained contact, the wrist follows the scene target. Wrist orientation retains its mapped target.
Body fitting combines landmark tracking, collision handling, temporal regularization, and joint constraints. Source foot motion identifies support intervals, during which horizontal robot toe positions enter the solve as hard equality constraints. This establishes a support trajectory before fine hand alignment begins.
Hand retrieval provides a feasible starting configuration
Each hand has a small geometric codebook: three samples along each of six independent driver ranges produce $3^6=729$ entries. Forward kinematics describes finger directions, bending, reach, and inter-finger distances in wrist-local coordinates. The solver retrieves the 12 nearest entries and blends their driver values using weights proportional to $\exp(-D/0.06)$, where $D$ is the masked descriptor distance.
Continuous fitting then adjusts these drivers under joint bounds and mimic relations, with geometric, posture, and temporal terms. The codebook supplies a warm start and a soft posture anchor. Fingers start neutral when articulated hand observations are absent; OMOMO’s later refinement uses robot-surface contact targets because its source joints are not fingertip measurements.
2. Let the arm and fingers share the correction
The second retargeting stage optimizes 14 arm/wrist joints plus 12 hand drivers. The floating base, legs, and waist retain their initialized trajectories. Restricting the active coordinates keeps hand corrections from disturbing established support.
For articulated-hand inputs, the fingertip objective in Appendix A is
\[\mathcal L_{\mathrm{tip}} =\sum_{s,f}\frac{\omega_f}{\ell_{t,s}^2} \left[ \alpha_{t,s}^2\|p_{t,sf}-y_{t,sf}\|_2^2 +(1-\alpha_{t,s})^2\|u_{t,sf}-v_{t,sf}\|_2^2 \right].\]Here $p,y$ are robot and target fingertip positions in the world frame; $u,v$ are their wrist-local counterparts; $\ell$ normalizes hand scale. The same interaction weight that controls wrist targets shifts finger fitting toward world-frame placement during interaction and toward local hand shape away from it. Thumb and index weights are 2.0 each, versus 0.20 for each remaining finger.
The full objective also tracks palm normal and other hand axes, penalizes insufficient clearance, and regularizes posture and temporal corrections. Wrist and palm position anchors weaken as interaction activates. That freedom allows the arm to move a few centimeters when finger articulation alone cannot repair the grasp geometry. Joint limits, mimic relations, and active self/ground collision boundaries constrain the solve; hand–object clearance and contact recovery enter as soft objectives.
One detail worth retaining in an implementation is the temporal term for the arms: it smooths the change in the correction $\delta_t^U=q_t^U-\bar q_t^U$. The target motion itself can keep moving while the correction remains stable. Hand drivers receive direct temporal smoothing.
The ablation exposes the cost of this freedom. On GRAB, refinement reduces thumb/index error from 36.033 to 4.342 mm, while palm-normal error rises from 0.189° to 3.652°. On HUMOTO, the corresponding errors change from 32.703 to 7.198 mm and 0.159° to 4.439°. The optimization prioritizes fingertip placement and fewer collision violations, accepting some orientation deviation.
3. Encode which body regions can read which information
The actor receives 238 scalar inputs: 211 robot features and 27 object features. Robot inputs include reference joint positions, non-finger velocities, current joint state, torso-anchor information, and the previous applied action. Eight regional encoders create 128-dimensional tokens for the base, left/right legs, waist, left/right arms, and left/right hands:
\[h_i=E_i(x_i)+p_i,\qquad Z=F_{\mathrm{robot}}(H;M_R).\]The learned embedding $p_i$ identifies the body region. Two Transformer blocks use a directed mask $M_R$. The base, legs, waist, and arms communicate with one another but cannot attend to hand tokens. Each hand attends to the waist, both arms, and itself. This supplies the fingers with upstream body context while excluding hand-token features from the body representation.
Object information arrives through a separate token. In the torso frame, let $\phi(T)$ concatenate a 3D position and a 6D rotation representation. The object input is
\[x_o=\left[\phi(T_o);\phi(T_o^{\mathrm{ref}}); \phi(T_o)-\phi(T_o^{\mathrm{ref}})\right]\in\mathbb R^{27}.\]The last term is a coordinate difference between pose representations; it is not an $SE(3)$ relative transform. A one-block fusion Transformer lets each waist, arm, or hand token attend only to itself and the object token. The object token attends only to itself. Base and leg tokens bypass this fusion.
The leg decoders also retain the waist token from before object fusion. Without that bypass, an object-conditioned waist feature would provide an indirect route from the object token to leg actions. Each leg head reads the base, its own leg, and the original waist; upper-body heads use the object-conditioned features. Seven heads jointly output 41 normalized position commands:
\[q_t^{\mathrm{cmd}}=q^{\mathrm{default}}+s\odot a_t.\]Joint-level PD control executes those targets. The structural independence concerns direct object observations at a policy step. Object forces can still change robot state, and the legs can respond to those changes. It does not establish that lower-body behavior is physically independent of manipulation.
4. Single-stage RL still depends on a detailed training recipe
All actor components train jointly with PPO and a privileged 463-input critic. No tracking-policy pretraining, teacher–student distillation, or later residual-policy training is used. Reference construction remains an earlier optimization stage, and each reference gets a separate learned controller.
Training uses 4,096 environments, 24 steps per environment per update, 200 Hz physics, and 50 Hz control. Body-only policies train for 8,000 PPO iterations; manipulation policies train for 12,000. The robot Transformer has two blocks, object fusion has one, and each block uses four attention heads. The comparison MLP has three hidden layers of width 621 and a comparable parameter count.
Rewards supervise body motion, hand drivers, wrist-relative hand poses, wrist–object relations, and object pose. Tracking terms use $r_k=\exp(-E_k/\sigma_k^2)$, supplemented by action smoothness, joint-limit, and undesired-contact penalties. Dynamics randomization covers friction, mass, motor strength, gains, delay, and external pushes. Actor object measurements are sampled at 30 Hz with 20–60 ms latency, ±1 cm position noise, and ±0.035 rad rotation perturbations; the critic receives uncorrupted simulation observations.
Half of default episode starts use the first reference frame, and half use sampled phases. Sampled-phase selection mixes uniform and failure-adaptive sampling. This makes later portions of a motion trainable even when early failures would otherwise prevent reaching them.
5. Read the results with their completion rule
Table 3 reports MuJoCo transfer after IsaacLab training. The manipulation results average equally across five references:
| Actor | Episode success | Body position error | Object position error | Object rotation error |
|---|---|---|---|---|
| InterMimic | 57.74% | 6.51 cm | 3.58 cm | 7.71° |
| Object MLP | 85.00% | 5.69 cm | 3.84 cm | 6.16° |
| DexWeave | 97.50% | 4.93 cm | 3.43 cm | 6.83° |
The closest controlled comparison is Object MLP versus DexWeave on the same G1–Inspire embodiment. InterMimic uses its official SMPL-X human skeleton setup, so its row is a cross-embodiment reference. DexWeave gains 12.5 percentage points over Object MLP, while the MLP retains lower object-orientation error. Figure 5 suggests approximately twice as fast convergence for manipulation; it does not provide a matched wall-clock throughput benchmark.
For each reference, checkpoints from the final 1,000 training iterations are saved every 100 iterations and evaluated with five evaluation seeds each. Metrics are averaged across runs, checkpoints, and then references. These seeds vary evaluation; this protocol does not establish replication across independently trained seeds.
“Success” means reaching the reference end or the applicable time limit without early termination. A sampled-phase episode may therefore complete only an evaluated segment. Manipulation terminates, among other conditions, when object position error exceeds 0.30 m or orientation error exceeds 2.0 rad. 97.5% should be read as completion under this protocol, with the reported tracking errors providing additional information about execution quality. It is not a hardware success rate or evidence of a shared policy generalizing to unseen tasks.
The geometric results support the reference-construction mechanism. Against OmniRetarget + SBR + arm–wrist IK, DexWeave lowers thumb/index error from 14.925 to 4.342 mm on GRAB and from 29.999 to 7.198 mm on HUMOTO. Penetration-duration fractions fall from 0.190 to 0.001 and 0.511 to 0.007. Those fractions count violations above the evaluation thresholds: 1 cm for ground/self penetration and 2 cm for object penetration. A near-zero fraction permits smaller penetrations.
Mask ablations cover only two reference motions. Full directed masking achieves 100% completion, versus 91.30% with dense robot-region attention and 99.71% with bidirectional object fusion. Dense robot attention yields slightly better object pose errors despite lower completion. The evidence favors the masks for completion and body tracking in this setting; it does not show a uniform gain across all metrics.
6. What I would carry into a new system
My strongest takeaway is to couple wrist placement and finger fitting before asking RL to track the result. The retargeting ablation gives that choice a concrete justification. I would preserve the original support trajectory, measure primary-fingertip error and collision frequency separately, and inspect the palm-orientation tradeoff instead of compressing them into one quality score.
The policy mask is a useful starting hypothesis when detailed finger features interfere with body tracking. Its value should be tested again if the task needs anticipatory balance under changing payloads: this architecture keeps direct object information out of the leg pathway, and the paper’s two-motion mask ablation leaves that broader question open.
The physical G1 demonstrations show that some learned skills transfer, but the paper supplies no aggregate hardware trial count or success-rate table. A practical follow-up would measure hardware completion over repeated resets, document the object-pose estimation system, and test a shared policy on held-out references. Those experiments would determine how far the current per-reference skill pipeline can scale.