[Paper Notes] UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis
Published:
UniCross proposes a compact view of held-object dexterity. Grasping, relocation, in-hand rotation, and in-hand translation all control the relation between a hand and an object while keeping the object secure. Once those relations are expressed in shared hand and root frames, the four skills can use the same observation variables, full-hand action space, network architecture, and reward vocabulary. Skill identity enters through a task mode, a motion axis, and pose targets whose components are marked tracked, fixed, or free.
This common interface supports ten PPO experts—one for grasping, one for relocation, six for signed rotation axes, and two for translation directions—and makes their distillation into one DAgger policy almost lossless. On Allegro Hand, the unified policy reaches 98.7% grasp, 99.0% relocation, 98.8% rotation, and 99.1% translation success in the paper’s general simulation setting. Chaining grasp–relocate–rotate and grasp–relocate–translate yields 87.4% and 96.3% end-to-end success.
Paper Info
“UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis” is by Hui Zhang, Julian Ferchow, Jie Song, and Mirko Meboldt, with affiliations at ETH Zürich, inspire AG, and HKUST (Guangzhou). It is an arXiv preprint, arXiv:2607.28198, submitted in July 2026. The project page contains videos for Allegro, MANO, and Sharpa Wave hands; its code link is still marked “Coming soon” as of this post.
Reframing Four Skills as Relational Motion
Skill-specific dexterous controllers often bake a contact regime into the task. A grasping controller rewards persistent force; an in-hand translation controller may assume an upward-facing palm; a rotation method may fix the wrist or rely on a special hand morphology. Those assumptions create incompatible states at skill boundaries. A grasp that works for lifting can leave the fingers poorly arranged for rotation, and a palm-supported translation state may become invalid when relocation changes wrist orientation.
UniCross defines the task through hand–object relational motion in two coordinate systems. A root frame is fixed to the initial wrist pose for the entire episode. The current hand frame moves with the wrist. Object and hand poses are represented in both frames, letting the same variables describe four behaviors:
- Grasp: the object stays fixed in the root frame while the hand approaches it.
- Relocate: the object stays fixed relative to the hand while the wrist carries it to a root-frame target pose.
- Rotate: the wrist and object position stay fixed while object orientation changes around a hand-frame axis.
- Translate: the wrist and object orientation stay fixed while object position changes along a hand-frame axis.
The complete learning pipeline is therefore a sequence of shared interfaces:
flowchart TD
A["Hand state"] --> D["Shared relational formulation"]
B["Contact and local geometry features"] --> D
C["Task mode, axis, current and target poses"] --> D
D --> E["Ten PPO skill experts"]
E --> F["DAgger: unified-policy rollouts plus expert queries"]
F --> G["One cross-skill policy"]
G --> H["Continuous grasp → relocate → rotate / translate"]
One Observation Space Across Skills
The policy observation is
[ o_t=(s_t^h,s_t^o,g_t). ]
The hand state (s_t^h=[q_t,q^{\text{target}}_{t-1}]) contains current joint positions and the previous joint targets for all finger joints plus six virtual wrist joints. The object representation
[ s_t^o=[c_t,f_t,v_t] ]
contains per-link binary contacts (c_t), contact-force magnitudes (f_t), and vectors (v_t) from every finger link to its nearest object-surface point. These local measurements encode the geometry that currently matters for interaction: a round surface supports smooth rolling contacts, while a prism edge calls for periodic finger reconfiguration.
The 55-dimensional objective vector is
[ g_t=[I_t,d_t^h,g_t^{\text{current}},g_t^{\text{target}}]. ]
Here (I_t\in\mathbb{R}^{10}) is a one-hot task mode, (d_t^h\in\mathbb{R}^{3}) is the desired motion axis in the hand frame, and the current/target terms contain object and hand poses in both hand and root frames. The direction points from object to hand for grasping, from current to target object position for relocation, along one of (x^\pm,y^\pm,z^\pm) for rotation, and along (z^\pm) for translation.
Tracked, Fixed, and Free Targets
The central unification mechanism assigns a role to each target pose. Tracked variables move toward the skill objective. Fixed variables retain their episode-initial values. Free variables are reset to their current values, removing their constraint.
| Skill | Tracked | Fixed | Free |
|---|---|---|---|
| Grasp | object position in hand frame | object position and orientation in root frame | object orientation in hand frame; wrist pose in root frame |
| Relocate | object and wrist poses in root frame | object pose in hand frame | none |
| Rotate | object orientation in hand and root frames | wrist pose; object position in hand and root frames | none |
| Translate | object position in hand and root frames | wrist pose; object orientation in hand and root frames | none |
For relocation, root-frame object targets interpolate toward a sampled final 6D pose, and wrist targets follow while preserving the current hand-frame object pose. Rotation updates the target orientation by a small angle around (d_t^h) at each step. Translation advances the hand-frame object target by (v_{\max}\Delta t\,d_t^h). The variables and update logic stay consistent even though each skill activates a different slice of the relation.
Full-Hand Actions and a Common Reward Vocabulary
The action controls every finger joint and the six virtual wrist joints. A normalized incremental command becomes a joint-position target:
[ q_t^{\text{act}} = \operatorname{clamp} \left(q_t^{\text{ref}}+\alpha\odot a_t,,q_{\min},,q_{\max}\right). ]
Finger references use the previous action target, producing smooth incremental motion. Wrist references use the current target wrist pose, which makes large relocation motions easier to explore. A PD controller converts these targets to torques. Simulation runs at 120 Hz and the policy controls at 20 Hz.
Every skill uses the reward decomposition
[ r_t=r_t^{\text{goal}}+r_t^{\text{track}}+r_t^{\text{reg}}, \qquad r_t^{\text{goal}}=r_t^{\text{contact}}+r_t^{\text{motion}}. ]
The contact term combines link-to-surface distance, contact indicators, and force magnitude:
[ r_t^{\text{contact}} = \frac{1}{N}\sum_{i=1}^{N} \left(-w_{\text{dis}}d_i+w_{\text{con}}c_i+w_f f_i\right), ]
while (r_t^{\text{motion}}=w_p\min(v_o^h\cdot d^h,v_{\max})) rewards object velocity along the requested axis. The tracking term measures hand/object pose errors in both frames. Regularization penalizes excessive finger deviation, wrist velocity, torque, and dropping. The reward structure is shared; the appendix assigns different contact and motion weights to grasping and the other skills.
From Ten PPO Experts to One DAgger Policy
The authors first train the ten task-direction experts independently with PPO in Isaac Gym. All experts use the same architecture: a two-layer ([128,64]) encoder for objective features and a three-layer ([512,256,128]) MLP for actions. The single cross-skill policy has the same architecture.
Vanilla DAgger rolls out the unified policy, queries the corresponding expert at every visited state, and minimizes MSE on the aggregated state–action buffer. Environments are distributed uniformly across valid object–skill pairs. This detail matters: supervised learning on expert trajectories alone would miss the off-expert states induced by the student’s own errors.
The distilled policy nearly matches its teachers. For grasp, relocation, rotation, and translation, the ten experts reach 99.0%, 99.1%, 99.1%, and 99.3% success; the unified policy reaches 98.7%, 99.0%, 98.8%, and 99.1%. A same-capacity network absorbs all ten modes with only tenths-of-a-point losses, providing the paper’s strongest evidence that the skill solutions are compatible.
Experimental Setup and Main Results
Training uses boxes and cylinders from two size regimes: compact wrappable objects and elongated objects. The split contains 400 training and 400 test geometries. Each object is evaluated from 25 initial conditions; rotation covers six directions and translation covers two. Success requires secure holding plus the skill criterion: lift for grasping, 3 cm / 0.15 rad target tolerance for relocation, more than (\pi/2) rotation, or more than 5 cm translation.
In the general Allegro setting, UniCross obtains 98.7% grasp success; 99.0% relocation success with 0.54 cm / 0.0134 rad error; 98.8% rotation success with 13.6 rad average accumulated rotation; and 99.1% translation success with 20.3 cm average displacement. The adapted skill-specific baselines reach 97.1%, 92.8%, 62.7%, and 70.6% success. Rotation and translation baselines deteriorate when wrist orientation is no longer fixed upward, exposing their reliance on palm support.
For geometry generalization, a separately sampled set of spheres, hexagonal prisms, and elongated octagonal prisms is unseen during training. Success remains 95.8%, 98.6%, 98.3%, and 96.0% across the four skills. Persistent random disturbances reach up to (10m_{\text{obj}}g); success still stays between 97.6% and 99.0%. The local distance/contact representation also produces qualitative scale- and shape-adaptive contact patterns.
The formulation transfers to three morphologies, with one separately trained unified policy per hand. Allegro reaches 98.7–99.1% across skills; the 26-DoF MANO hand reaches 94.5–96.8%; and the 28-DoF Sharpa Wave reaches 98.0–99.1%. This demonstrates morphology portability of the formulation and training recipe. It does not demonstrate a single checkpoint operating across all hands.
Long-Horizon Composition
The long-horizon test switches the task mode within one uninterrupted rollout. The policy grasps an object from a table, relocates it to a randomized 6D pose, then rotates or translates it in hand. A sequence counts only when all three phases succeed.
For grasp–relocate–rotate, phase success is 99.7%, 92.2%, and 95.1%, producing 87.4% overall. For grasp–relocate–translate, the phases reach 99.9%, 98.3%, and 98.0%, producing 96.3% overall. Rotation is harder after relocation because the wrist often ends downward or diagonally downward, where the object slips more easily. This is exactly the state-compatibility pressure that isolated palm-up controllers avoid.
What the Ablations Reveal
The ablations show that unification comes from coordinated conditioning across observation, action, and reward.
- Removing target observations causes modest degradation because the task axis still communicates much of the objective.
- Removing target-conditioned wrist actions collapses relocation success from 99.0% to 37.4%, showing that direct target references drive large wrist exploration.
- Removing pose-tracking reward lowers relocation to 62.9% and raises position error to 5.47 cm.
- Removing contact indicators hurts rotation most, reducing success from 98.8% to 82.7% and accumulated rotation from 13.6 to 8.78 rad.
- Removing nearest-surface distance vectors reduces all finger-intensive skills, confirming that local geometry is essential when contacts must be reconfigured.
Strengths and Boundaries
UniCross is conceptually clean. It unifies the task interface first, then uses standard PPO and DAgger. The near-lossless distillation, unseen-shape tests, persistent-force tests, three morphologies, and uninterrupted skill chains evaluate complementary consequences of that interface. The paper also makes an important distinction between a general framework and hand-specific weights: morphology transfer requires retraining, while the formulation stays unchanged.
The current evidence is entirely simulation-based. Observations include privileged contact states, force magnitudes, nearest surface vectors, and exact relative poses; deploying the policy on hardware would require tactile sensing plus reliable object geometry and pose estimation. Training objects are synthetic primitives, and the separate unseen set remains geometric. The policy receives an externally chosen one-hot skill mode, motion axis, and target pose, so task planning, transition timing, and goal inference are outside the learned controller. Initial finger poses are also designed for the two object regimes. Finally, the held-object formulation excludes release/regrasp, non-prehensile interaction, bimanual coordination, tool–environment contact, and recovery after a drop.
Takeaways
The most reusable idea is relational task factorization. Multi-skill learning becomes easier when each skill activates tracked, fixed, and free components of the same state description. This turns skill composition into a change of objective conditioning while preserving the controller’s input/output contract and reachable state distribution.
UniCross also shows that policy distillation works best after expert interfaces are aligned. DAgger handles student-induced states, but the low distillation loss comes from a deeper property: every expert speaks the same observation and action language and produces states that remain meaningful to the others.
The next research step is a hardware-compatible version of this interface. Replacing privileged contacts and geometry with tactile/visual estimates, learning mode transitions from task intent, and extending the relation set to release, environmental contact, and bimanual manipulation would test whether the shared solution space survives real-world uncertainty.