[Paper Notes] Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
UHAS addresses a structural obstacle in cross-embodiment dexterous learning: joint-space actions from one hand have no direct meaning on another hand with different joints, finger counts, dimensions, or ranges of motion. The paper replaces joint actions with deformations of a normalized canonical sphere. A policy predicts how that sphere should deform, and a hand-specific Cascade Inverse Kinematics (CIK) controller converts the deformation into executable joint targets.
The representation is compact and geometric. Each finger owns one driving plane that controls lateral angular motion (\Delta\theta), plus two driving vectors that control radial deformation (\Delta r). With five planes, the policy produces a 15-dimensional continuous action shared by Allegro, LEAP, Shadow, and MANO hands. Hand scale and kinematic details enter only through sphere construction, surface correspondence, and CIK.
In simulation, a single multi-hand policy matches hand-specific policies: success rates remain between 98.7% and 99.5% across four hands. Leaving the target hand out during training still yields 85.7–98.1% zero-shot success. Transfer across four- and five-finger morphology is weaker, and real-world zero-shot policies average fewer than one consecutive cube reorientation. UHAS therefore succeeds as an action interface, while the experiments also show that a shared coordinate system cannot erase differences in reachable workspace, dynamics, or hardware reliability.
Paper Info
The paper is “Cross-Embodiment Robot Manipulation via a Unified Hand Action Space” by Luis Felipe Casas, Robert Teal, Keval Shah, Abhijit Tadepalli, Wanxin Jin, and Yu Xiang, from the University of Texas at Dallas and Arizona State University. It was presented at the 4th Workshop on Dexterous Manipulation at Robotics: Science and Systems (RSS), 2026.
- Paper: arXiv:2607.03570
- Project page, code, data, and videos: irvlutd.github.io/UHAS
1. The Cross-Embodiment Action Problem
Consider two dexterous hands that can both rotate a cube. Their successful behaviors may share the same semantics—spread two fingers, close the thumb, roll the cube toward the palm—while their joint commands are incompatible. An Allegro joint vector cannot be sent to a LEAP hand, and matching vector dimensions does not solve the problem because joint axes and ranges still mean different things.
This creates two barriers to generalist dexterous policies:
- The action space is embodiment-specific: each hand exposes a different joint vector.
- The proprioceptive observation is embodiment-specific: raw joint positions and velocities have different dimensions and semantics.
UHAS moves both sides of the policy interface into a canonical geometric frame. The action becomes a deformation of a sphere in the hand’s grasping workspace. Proprioception becomes the normalized positions and velocities of corresponding points along the fingers. A single network can then receive and produce tensors with consistent meaning across embodiments.
The complete control chain is:
[ \text{homogeneous state} \xrightarrow{\pi_\theta} (\Delta\theta,\Delta r) \rightarrow \text{deformed canonical sphere} \xrightarrow{\mathrm{CIK}_e} q_e^{\text{target}}, ]
where (e) identifies the target hand. The policy is shared; sphere construction and (\mathrm{CIK}_e) contain the embodiment-specific geometry.
2. Automatically Constructing a Sphere for Each Hand
UHAS begins with a robot hand URDF and an open-hand configuration. It identifies palm and fingertip frames, computes the palm center from the average finger-root position, and measures the average distance (l) from the palm center to the fingertips. The physical sphere radius is set to
[ r_h = \frac{2l}{\pi}. ]
This radius makes the fingertip span correspond approximately to a (90^\circ) arc. The sphere center is placed one radius above the palm center along the outward palm normal, locating it inside the hand’s natural grasping workspace.
The hand-specific sphere frame is oriented consistently:
- (+z) follows the outward palm normal.
- (+x) points toward the middle finger.
- (+y) follows from the right-hand rule.
All distances in this frame are divided by (r_h), turning every hand-specific sphere into a unit sphere. A large hand and a small hand therefore share the same normalized coordinates. The original center, orientation, and radius remain available to CIK when it reconstructs physical joint configurations.
This normalization removes hand scale from the learned interface. It does not assume that every hand has the same workspace; CIK still has to realize each target under the actual hand kinematics.
3. Binding the Hand Surface to the Canonical Sphere
The sphere becomes useful only after establishing correspondence with the hand. UHAS uniformly samples points on the sphere and assigns each point spherical coordinates
[ (\theta,\phi,r), ]
where (\theta) is azimuth, (\phi) is polar angle, and (r=1) on the undeformed unit sphere. These points are projected onto nearby locations on the interior surface of the palm and fingers.
Each projected hand-surface point retains the spherical coordinates of its source point. Its 3D location changes as the hand moves, while its identity in the canonical domain stays fixed. The result is a dense, configuration-invariant correspondence:
[ \text{canonical sphere coordinate} \longleftrightarrow \text{semantic hand-surface location}. ]
Different hands can now refer to comparable regions through the same spherical domain. A deformation near one finger’s driving plane has a consistent geometric interpretation even though the joint chain that produces it differs across hands.
This is the conceptual center of UHAS. The sphere serves as a control canvas, and the surface correspondence tells each embodiment how to interpret that canvas.
4. A Compact Sphere-Deformation Action
Predicting a displacement for every surface point would produce an unwieldy action vector. UHAS instead uses sparse control primitives and reconstructs a continuous deformation through interpolation.
Driving Planes: Lateral Motion
A driving plane passes through the sphere center at a fixed azimuth (\theta_{\text{plane}}). One plane is aligned with each fingertip. Rotating a plane produces an azimuthal displacement (\Delta\theta), corresponding to lateral finger motion such as abduction and adduction.
Driving Vectors: Closing and Opening
Each plane contains control points at selected polar angles. Their radial displacements (\Delta r) contract or expand the sphere locally, representing how fingers close around or move away from the object-centered workspace. The final radial field is interpolated over ((\theta,\phi)).
The paper uses five planes and two driving vectors per plane. The action dimension is therefore
[ 5\;\Delta\theta + 5\times2\;\Delta r =15. ]
The two radial vectors sit at (\phi=60^\circ) and (120^\circ), with actions in ([-2,2]). Negative radius is allowed during deformation because it lets the controller command rapid finger closure during aggressive cube rotations. Sphere points with negative final radius are removed before CIK; when an encompassing joint has no reachable target points left, it is commanded fully closed.
The design strikes a useful balance. One radial vector per finger lacks control flexibility. Three vectors achieve slightly higher final performance, while two reach strong performance with the shortest training time. Four vectors enlarge the action space without a consistent benefit.
5. Unifying Four- and Five-Finger Hands
The shared policy uses five driving planes. Shadow and MANO naturally associate one plane with each of their five fingers. Allegro and LEAP have four fingers, so UHAS inserts an extra plane at the ring-finger azimuth. The duplicated ring-finger plane and its radial actions are averaged before sphere interpolation.
The observation side uses the same padding strategy: ring-finger observations are duplicated for four-finger embodiments so that all hands expose an equal-sized tensor.
This makes the interface dimensionally compatible while preserving five independently controlled channels for five-finger hands. It is a practical convention, and it also exposes a limitation: a missing finger cannot be recovered through padding. Cross-morphology policies still face different reachable contact patterns and coordination options.
6. Cascade Inverse Kinematics
Once the policy has deformed the sphere, the controller must find executable joint targets. Generic numerical IK over all hand surface points would be too expensive for high-rate control. CIK uses the geometry of UHAS to decompose the problem.
6.1 Automatic Joint Classification
For each hand, every joint is swept across its range in an open-hand configuration. Forward kinematics records how the fingertip changes in spherical coordinates. Each joint is assigned to one of two classes:
- Lateral joints mainly change fingertip azimuth (\theta), producing side-to-side motion.
- Encompassing joints mainly change radial distance (r) and polar angle (\phi), making the finger wrap around the sphere.
If a joint axis points toward the fingertip and its effect is ambiguous in the open pose, the remaining joints are partially flexed and the sweep is repeated. Classification happens once from the URDF.
6.2 Lateral Lookup
CIK precomputes a lookup table for every lateral joint. It samples the joint over its full range, resolves the encompassing joints on the undeformed sphere, and records the resulting fingertip azimuth:
[ q_{\text{lateral}} \longmapsto \theta_{\text{fingertip}}. ]
At runtime, the policy supplies (\Delta\theta), giving
[ \theta_{\text{target}} = \theta_{\text{initial}}+\Delta\theta. ]
The table returns the lateral joint target in constant time, bounded by the hand’s attainable range.
6.3 Proximal-to-Distal Encompassing Cascade
After setting the lateral joints, CIK visits encompassing joints from the finger root toward the fingertip. For each joint, it transforms the associated surface targets and all descendant targets into the joint’s local frame, then directly computes the angle that places them on the deformed sphere.
Each joint is solved once in a single forward pass. Fingers are kinematically independent in this formulation, so their cascades run independently. The final output is an embodiment-specific joint target (q) for the low-level controller.
CIK reaches approximately 150 Hz on the real setup. The paper reports that serial communication and AprilTag pose estimation, not CIK, dominate system latency.
7. A Homogeneous Observation Space
A unified action alone is insufficient: the policy also needs comparable proprioception. UHAS samples seven candidate points from each finger root to fingertip and obtains their positions through forward kinematics. Velocities are calculated from the corresponding Jacobians and joint velocities.
The final policy keeps only two points per finger—the midpoint and fingertip. Their positions and velocities are represented in the canonical sphere frame and divided by the hand-specific radius. This produces scale-normalized geometric proprioception with consistent semantics across hands.
Object state, goal orientation, and the remaining task variables come from the cube-reorientation environment. Raw robot joint values are excluded from the shared policy observation because their dimensions, limits, and meanings are embodiment-specific.
An ablation shows that one to four observation points per finger all reach about 98.8–99.1% success. Additional points add computation with marginal gain, supporting the two-point choice.
8. Policy Learning
The authors train PPO policies in NVIDIA Isaac Lab on an in-hand cube reorientation task. Each simulated episode evaluates ten sequential target orientations. A target must be reached within 30 seconds; the environment resets after a cube drop and continues evaluating the remaining targets.
The reward follows Isaac Lab’s Reposing Cube task, with two extra joint regularizers:
[ R = w_d d +w_r r_{\text{rot}} +w_{\text{lat}}p_{\text{lat}} +w_{\text{rad}}p_{\text{rad}} +b_{\text{success}} +p_{\text{fall}}. ]
Here, (d) measures object-to-goal distance, (r_{\text{rot}}) rewards orientation alignment, and (p_{\text{lat}},p_{\text{rad}}) regularize lateral and encompassing joints around a reference configuration. The regularizers discourage an embodiment-specific shortcut in which the policy overuses one joint class.
Domain randomization covers object scale, mass, friction, robot mass and friction, joint armature, effort limits, stiffness, damping, hand-base inclination, and driving-vector angle. These variations prevent the policy from depending on a single hand’s cube-palm dynamics or exact action geometry.
The main evaluation metrics are individual reorientation success rate and average consecutive reorientations before the first drop, capped at ten in simulation.
9. Main Simulation Results
| Test hand | Single-hand UHAS | Joint control | Multi-hand UHAS | Zero-shot UHAS |
|---|---|---|---|---|
| Allegro | 99.1 / 9.6 | 98.5 / 9.2 | 99.2 / 9.5 | 95.3 / 7.7 |
| LEAP | 99.7 / 9.8 | 98.6 / 9.3 | 99.1 / 9.5 | 95.5 / 7.7 |
| Shadow | 99.3 / 9.6 | 98.0 / 9.1 | 98.7 / 9.2 | 85.7 / 4.4 |
| MANO | 99.8 / 9.9 | 99.6 / 9.8 | 99.5 / 9.8 | 98.1 / 8.9 |
Each entry is success rate (%) / average consecutive reorientations. The multi-hand model is one policy trained on all four hands. Its performance is close to separately trained policies and generally stronger than direct joint control. This is the clearest evidence that the sphere representation retains enough control authority for dexterous reorientation.
For zero-shot evaluation, the test hand is held out and the policy is trained on the other three. Transfer remains high on Allegro, LEAP, and MANO. Shadow is harder at 85.7% success and 4.4 consecutive reorientations, indicating that shared semantics still leave an embodiment gap.
10. Morphology Transfer and Fast Adaptation
Training on the two five-finger hands and testing on four-finger hands yields 66.2% success on Allegro and 80.8% on LEAP. Training on the two four-finger hands transfers at 83.2% to Shadow and 95.0% to MANO. These figures are meaningful, yet clearly below the near-perfect in-distribution results.
The direction of transfer is asymmetric. In the one-source-hand experiments, a LEAP-trained policy transfers far better than a MANO-trained policy. The paper connects this to LEAP’s large range of motion: training on a flexible hand exposes the policy to more transferable behaviors, while a constrained hand encourages solutions tied to its own workspace. Similar finger count alone does not guarantee transfer; MANO-to-Shadow performs poorly even though both have five fingers.
Finetuning is much easier than training from scratch. Starting from a MANO policy and adapting for 500 iterations raises success to 96.3% on Allegro, 96.2% on LEAP, and 95.8% on Shadow. Full training takes about 4,500 iterations. UHAS therefore provides a useful initialization even when zero-shot behavior is imperfect.
11. Real-World Results
The real system uses AprilTags on all cube faces, one or two Intel RealSense cameras, and a pose-fusion pipeline that rejects inconsistent tag estimates. Policies run at 20 Hz because LEAP serial communication is the main bandwidth constraint.
| Physical hand | Zero-shot | Multi-hand | Target-hand trained | Joint baseline |
|---|---|---|---|---|
| LEAP | 0.9 | 1.1 | 2.0 | 0.6 |
| Allegro | 0.8 | 2.1 | 2.1 | — |
Values are mean consecutive reorientations over ten trials. UHAS variants outperform the available joint-control baseline on LEAP, and the Allegro multi-hand policy matches the hand-specific model. Still, the gap from simulation is large. Zero-shot transfer averages below one reorientation on both physical hands, and trial-to-trial variance is high.
The hardware appendix explains why sim-to-real is difficult. Effective LEAP PD gains differ from manufacturer specifications, damping is almost negligible, motor communication drops reads and writes, joints overshoot, structural parts deform under load, and repeated use loosens the fingers. The authors add velocity limits, broaden randomization, restrict joint ranges, and reinforce the hand mechanically.
These observations put the results in perspective: UHAS unifies the command semantics, while contact dynamics and hardware quality remain embodiment-specific.
12. What the Ablations Say
The ablations reveal three practical design choices:
- Two driving vectors per plane give the best training-efficiency trade-off: 98.7% success in 4.5 hours. Three vectors reach 99.5% but require 6.5 hours.
- Two observation points per finger are sufficient. Denser finger sampling produces only small gains.
- Four versus five driving planes makes little difference on cube reorientation because the pinky contributes little to this task. Three planes fail to learn reliable reorientation, showing that the action space still needs a minimum level of expressiveness.
Domain randomization particularly helps morphologically different targets: Shadow zero-shot success rises from 80.1% to 85.7%, and MANO rises from 97.1% to 98.1%. Allegro changes from 96.7% to 95.3%, so the benefit is not uniform across every hand.
13. Strengths and Limitations
Strengths. UHAS is interpretable, compact, and executable. The canonical sphere cleanly separates shared action semantics from hand-specific kinematics. Automatic sphere construction reduces per-hand engineering, and CIK is fast enough for real-time control. The evaluation covers four different simulated hands, held-out embodiments, finger-count transfer, short-budget finetuning, and two physical hands.
Limitations. The task is limited to in-hand cube reorientation with a fixed palm-up setup. The sphere naturally fits enclosing interactions around a compact object; its value for tool use, precision pinch, articulated objects, bimanual tasks, and arm-hand coordination remains open. Surface correspondence and CIK preserve geometric intent but do not explicitly model contact force, tactile feedback, friction state, or dynamic feasibility.
Cross-morphology transfer loses substantial performance, and the four-finger padding rule provides dimensional consistency without semantically replacing absent fingers. The policy also depends strongly on reward design, PD parameters, and extensive domain randomization. Real-world zero-shot performance remains modest.
Finally, UHAS creates an embodiment-agnostic hand interface, not a complete embodiment-agnostic manipulation system. Perception, object representation, palm motion, low-level actuation, system identification, and hardware maintenance still require task- and platform-specific work.
14. My Takeaway
UHAS offers a useful way to think about universal robot actions: search for a geometric intermediate object on which different embodiments can express comparable intent. Here, the intermediate object is a deformable sphere. The policy learns sphere motion; CIK compiles it into the joints of a particular hand.
The strongest result is the multi-hand policy. Near-single-hand performance across four kinematically different hands shows that joint vectors are not necessary for high-quality dexterous control when a sufficiently expressive geometric interface is available. The zero-shot and real-world numbers supply the necessary qualification: common coordinates improve transfer, while the feasible behavior distribution still depends on morphology and dynamics.
For future dexterous foundation models, UHAS is best viewed as a candidate action tokenizer before tokenization—a continuous, physically interpretable canonical layer that could let heterogeneous hand datasets supervise one policy. Extending it beyond spherical enclosure tasks will determine how universal that layer really is.