[Paper Notes] WARP: Whole-Body Retargeting for Learning from Offline Human Demonstrations
Published:
WARP is best understood as an analytic retargeting system that makes offline human motion usable as robot supervision. The retargeting part is not a learned neural model and is not a generic weighted IK optimizer. It is a geometry-first pipeline built around a closed-form Shoulder-Elbow-Wrist solver. Learning enters after retargeting: WARP-generated robot trajectories are used to train a behavior-cloning policy.
My read: the important contribution is the separation of two problems that often get mixed together. First, convert human whole-body motion into precise, consistent, robot-feasible actions. Second, train a policy on those actions. WARP argues that if the first step is noisy, multi-modal, or seed-dependent, then behavior cloning inherits that ambiguity as supervision error.
Paper Info
The paper is “WARP: Whole-Body Retargeting for Learning from Offline Human Demonstrations” by Zhenyang Chen, Chuizheng Kong, Chuye Zhang, Yuanshao Yang, Lawrence Y. Zhu, Shreyas Kousik, and Danfei Xu from Georgia Institute of Technology. It is available as arXiv:2606.29940, with project page warp-retarget.github.io.
WARP stands for Whole-body-Aware Retargeting from human Pose.
The Problem
Human demonstrations are attractive because they can be collected without robot hardware in the loop. A person can move naturally, use their torso, route their elbows around obstacles, shift their body to extend reach, and demonstrate contact-rich whole-body behavior. The hard part is that a human demonstration is not directly a robot demonstration.
Offline retargeting is stricter than online teleoperation. During teleoperation, a human operator can watch the robot and compensate when the wrist drifts, contact fails, or the IK solver chooses an awkward branch. Offline data has no such correction. The retargeted trajectory itself becomes the target action sequence for policy learning.
The paper frames two failure modes:
- Imprecision: a retargeter may trade off end-effector accuracy against torso or elbow similarity, turning geometric error into action-label error.
- Inconsistency: redundant humanoid kinematics can map similar human poses to different robot configurations depending on solver initialization, creating multi-modal labels for behavior cloning.
WARP’s goal is therefore precise and consistent retargeting. The robot should match the task-critical palm pose while preserving the demonstrator’s whole-body structural intent.
Is WARP Optimization-Based Or Learning-Based?
The clean answer is:
WARP retargeting is analytic / geometric. The downstream policy is learning-based.
The retargeter constructs a target robot skeleton and solves closed-form geometric subproblems. It avoids the usual weighted multi-task IK formulation:
[ \dot q^\ast = \arg\min_{\dot q} \sum_i w_i\lVert J_i\dot q-\dot x_i^\ast\rVert^2 +w_{\mathrm{posture}}\lVert \dot q-\dot q_{\mathrm{posture}}\rVert^2. ]
That optimization-based baseline, represented by MINK variants in the paper, makes palm tracking, torso orientation, elbow swivel, and posture regularization compete through task weights. WARP instead makes palm matching a hard geometric constraint and resolves the remaining arm configuration through SEW geometry.
Learning appears later. After retargeting, the generated robot trajectory provides proprioceptive states and action supervision for behavior cloning:
[ q_t^r = \mathrm{Ret}(H(t)). ]
The policy maps observations and recent robot state history to future action chunks. So WARP is not an end-to-end learned retargeting map; it is an analytic data-conversion pipeline paired with a learned controller.
Core Representation: SEW
The core mechanism is the Shoulder-Elbow-Wrist (SEW) representation. For each arm, WARP represents the human skeleton as:
[ H=(s,e,w,\mathbf{H},t), ]
where (s,e,w,t\in\mathbb{R}^3) are shoulder, elbow, wrist, and palm/tool positions, and (\mathbf{H}\in SO(3)) is the palm orientation. These quantities are expressed in an upper-body-centric frame built from the shoulder line and an anchor point.
The representation separates arm shape from embodiment scale. Human limb configuration is captured by scale-invariant directions:
[ u=\mathrm{unit}(e-s), \qquad l=\mathrm{unit}(w-e). ]
For humanoid arms with a spherical shoulder, pin-joint elbow, and spherical wrist, SEW admits a closed-form IK solution with at most one valid joint configuration. This is the key to consistency: similar human poses should map to a single robot action, not a family of seed-dependent IK branches.
Vanilla SEW-Mimic aligns limb directions. WARP changes the target. For offline policy learning, the robot palm must land exactly where the human palm indicates task contact. WARP therefore uses SEW as the deterministic IK backbone while adding constrained skeleton alignment for exact palm matching.
WARP Retargeting Pipeline
WARP solves retargeting in two main stages.
First, adaptive offset accounts for link-length differences. If the robot copies the human arm directions (u^{hm}, l^{hm}), its predicted palm (\hat t^{rb}) may drift from the human palm (t). WARP computes a two-arm centroid offset:
[ p_{\mathrm{offset}} = \frac{1}{2}(t_L+t_R) - \frac{1}{2}(\hat t_L^{rb}+\hat t_R^{rb}). ]
This shifts the robot upper-body origin so the left/right palm centroid aligns with the human palm centroid.
Second, per-arm palm alignment solves each arm independently. The robot hand orientation transfers from the human hand:
[ \mathbf{H}^{rb}=\mathbf{H}^{hm}. ]
The robot wrist is recovered from the desired palm pose and the fixed wrist-to-palm offset (p_{WT}). The hard constraint is:
[ w^{rb}+\mathbf{H}^{rb}p_{WT} = (T^{hm\leftarrow rb})^{-1}t. ]
After shoulder and wrist are fixed, the remaining question is where the robot elbow should lie. WARP transfers the human elbow configuration through the SEW elbow angle (\psi), computes the elbow half-plane normal (\hat n), and uses a geometric subproblem, SP3, to find the robot elbow:
[ e^{rb} = s^{rb} + R(\hat n,\theta_{\mathrm{SEW}}) \ell_{SE}\hat e_{SW}. ]
The corrected robot skeleton ((s^{rb},e^{rb},w^{rb},\mathbf{H}^{rb})) is then passed to the SEW solver to recover joint angles in closed form. The paper is explicit that no iterative solver is invoked in this retargeting path. The torso pose is also solved with a closed-form IK-Geo solver.
Lazy Mobile-Base Control
Whole-body human motion implies a base pose, but directly tracking every small upper-body shift with the mobile base would make the wheels chase jitter and introduce lag. WARP’s design is to let the torso absorb fine adjustments while the base moves for real relocation.
It keeps a filtered base target (q_b=(p_b,\theta_b)) and compares it with the desired base pose (q_d=(p_d,\theta_d)). Small errors inside a deadband are ignored:
[ \tilde e_{xy} = \max(0,\lVert p_d-p_b\rVert-\delta_{xy})\hat e_{xy}, \qquad \tilde e_\theta = \mathrm{sign}(e_\theta)\max(0,\lvert e_\theta\rvert-\delta_\theta). ]
Then a damped second-order filter updates the base:
[ \ddot p_b=\omega_n^2\tilde e_{xy}-2\zeta\omega_n\dot p_b, \qquad \ddot\theta_b=\omega_n^2\tilde e_\theta-2\zeta\omega_n\dot\theta_b. ]
This is a small systems detail with a large effect: smooth base motion makes open-loop replay and downstream policy learning less brittle.
Policy Learning After Retargeting
The policy is trained after WARP converts human motion into robot actions. The authors instantiate the policy with HPT, a transformer-based action-chunk model trained with flow matching. The action chunk is decomposed into base, torso, arm, and hand components:
[ a_{1:H}=[a_b,a_\tau,a_r,a_h]_{1:H}. ]
The model uses a proximal-to-distal hierarchy:
[ b \preceq \tau \preceq r \preceq h. ]
A block-causal attention mask lets proximal body blocks inform distal blocks, while temporal tokens over the action horizon remain jointly visible. The final policy is lightweight, with about 11M trainable parameters. This policy detail matters because WARP’s retargeting is designed to produce unimodal, consistent action labels that a standard behavior-cloning model can learn.
Experiments
The experiments test three claims: WARP improves retargeting quality, WARP-generated data improves downstream policy learning, and whole-body retargeting is necessary for tasks where torso, elbow, or base configuration matters.
For retargeting quality, the authors sample 514 manipulation demonstrations from BONES-SEED-SOMA and compare WARP with SEW-Mimic and MINK variants. MINK-EF tracks end-effectors only. MINK-TE adds soft torso and elbow objectives. WARP achieves very low palm error without joint-limit constraints: 0.0046 mm mean palm position error and 0.046 mm P95 error, compared with 0.701 mm / 1.853 mm for MINK-EF and 18.557 mm / 73.980 mm for MINK-TE. It also has lower joint-limit and self-collision fractions than the MINK variants. The paper reports roughly 30x faster solving than iterative optimization-based retargeters: converting SEED takes about one CPU hour for WARP versus a full day for the baseline.
For simulation policy learning, the authors use DexMimicGen demonstrations on three bimanual tasks: can sort, pouring, and coffee. WARP and MINK have similar replay success, about 80.0% versus 79.5% on average. The policy gap is larger: policies trained on WARP data reach 71% average success, compared with 59% for MINK. On the coffee task, WARP policy success is 34%, while MINK is 8%. This is the paper’s most important empirical point: replay success can hide retargeting defects that later hurt learning.
For real-world evaluation, the robot platform is RB-Y1 with a holonomic base, 6-DoF torso, two 7-DoF arms, and 12-DoF XHands. Human data is collected with a Meta Quest headset. The tasks are rotate box, push cart, pick up laundry, and fridge-door closing with the elbow. Across tasks, WARP outperforms the end-effector-centric MINK baseline. In rotate-box policy rollout, WARP reaches 65% success versus 40% for MINK. In the elbow-mode fridge task, WARP succeeds in 90% replay trials, showing why end-effector-only retargeting is too narrow for whole-body manipulation.
Why The Baselines Fail
The appendix is useful because it explains the failure mechanism in addition to reporting metrics. MINK-style retargeting optimizes in joint space with weighted soft objectives. This creates three issues:
- Seed-dependent null-space choices create inconsistent action labels.
- Increasing posture weights can degrade palm tracking, while decreasing them can create unsafe whole-body configurations.
- Absolute keypoint or skeleton mapping suffers from embodiment mismatch between human and robot link lengths.
WARP removes these tradeoffs structurally. It returns one solution per non-degenerate pose, enforces palm matching as a hard constraint, and compares human and robot arms through scale-aware SEW geometry.
Limitations
The main limitation is that WARP is still primarily kinematic. It generates reference joint trajectories that satisfy geometric constraints, but it does not explicitly model controller error, actuator bandwidth, latency, contact dynamics, or disturbances from object interaction. A trajectory can be geometrically valid and still fail if the robot cannot track it under contact.
The policy and perception setup are also simplified. The real-world experiments use low-dimensional object-state inputs from AprilTag tracking, with Vicon localization for the robot base. This isolates retargeting quality, but it does not solve marker-free visual generalization in cluttered home environments.
Finally, human data quality depends on the capture device. A Quest headset is scalable and accessible, but inside-out tracking can introduce jitter, drift, discontinuities, and inaccurate limb estimates. Since WARP is an offline pipeline, those artifacts can propagate into robot supervision.
Takeaway
WARP’s central message is that retargeting quality is policy-learning quality. If offline human data is converted into imprecise or inconsistent robot actions, a behavior-cloning policy sees ambiguous labels even when replay appears acceptable.
For robot learning, the method is a useful reference point because it refuses to treat whole-body retargeting as just another weighted IK tuning problem. It makes task-critical palm tracking exact, preserves elbow and torso structure through SEW geometry, smooths the base through a lazy controller, and leaves learning to the policy stage after the action labels are made consistent.