[Paper Notes] Assembling Two Parts in One Hand
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
Capping a marker with the hand already holding it requires several fingers to support the body while others align and insert the cap. Assembling Two Parts in One Hand learns this coordination with simulation-based reinforcement learning. A relative-pose reward specifies the assembly goal; a multiplicative auxiliary reward encourages useful finger contacts and limits departure from a single human reference pose. An LSTM combines estimated object states with joint-angle history to control the hand under noisy vision.
On a 22-DoF Sharpa Wave Hand, the resulting policies complete Bottle, Syringe, and Marker assembly in 15/20, 17/20, and 16/20 real trials. Each trial starts with manually positioned objects, sometimes supported until the policy engages. The contribution is a working method for the assembly segment and a useful probe of finger coordination; autonomous acquisition of both parts remains outside the demonstrated workflow.
Paper info
Liuao Pei, Tianyue Wu, Hui Zhang, Ping Luo, and Jie Song authored the paper, with affiliations at the University of Hong Kong, HKUST (Guangzhou), HKUST, and ETH Zurich. Pei and Wu contributed equally. These notes cover the 19-page arXiv:2609.10137v1, submitted September 9, 2026; the arXiv record states that it will appear at CoRL 2026.
The project page hosts the paper PDF and videos, including the full 20-trial sequences. As of September 11, 2026, the code repository is a placeholder, with training and deployment code planned for release before November 2026.
1. Put the assembly goal between the objects
The formulation names the two parts $O_h$, the held object, and $O_f$, the fixed object. Both are movable and supported by the same hand. For a bottle, the thumb and index finger manipulate the cap while the middle, ring, and little fingers support and adjust the bottle. Those supporting fingers also resist the reaction forces generated during insertion.
This makes the target a relative configuration of two objects. In $O_f$’s coordinate frame, the method measures the lateral distance $e_{xy}$ and axial distance $e_z$ between a task-defined reference point and goal point. Axis misalignment is
\[e_\theta=1-\hat z_h^\top\hat z_f.\]The goal reward in Eq. (3) is
\[R_t^{\mathrm{goal}} =5\left[ e^{-60e_{xy}-60e_z-5e_\theta} +e^{-200e_{xy}-120e_z-30e_\theta} \right].\]Distances are in meters. The two exponentials reward progress at different spatial scales: the broader term supplies guidance away from the target, while the narrower term favors precise final alignment. Bottle, Syringe, and Marker share this recipe through their geometric reference points. All three are plug-in assembly tasks; the bottle experiment does not demonstrate screwing a threaded cap.
2. Shape finger roles without prescribing a trajectory
A goal reward alone leaves many possible ways to move the parts, including awkward contacts that work only from a narrow range of initial states. The method adds
\[R_t^{\mathrm{aux}} =\rho_t^{\mathrm{pinch}} \rho_t^{\mathrm{grasp}} \rho_t^{\mathrm{pose}}, \qquad \rho_t^{(\cdot)}\in[0,1].\]The pinch factor is a normalized contact-count signal for the thumb and index finger touching $O_h$. The grasp factor encourages the remaining fingers to cage $O_f$:
\[\rho_t^{\mathrm{grasp}} =\exp\!\left(-20\max_{i\in F_f}d_i\right), \qquad F_f=\{\text{middle, ring, little}\},\]where $d_i$ measures finger–object separation. Using the maximum distance makes the least engaged support finger matter. A masked joint-space penalty keeps the hand near a nominal configuration:
\[\rho_t^{\mathrm{pose}} =\exp\!\left(-\|q_t-q_0\|_M^2/12\right).\]Multiplication means that one poor factor suppresses the auxiliary reward even when the other two are high. In practical terms, a good pinch should coexist with support and a workable hand posture. This is a soft preference: the policy can leave the nominal contacts when the assembly objective justifies it. In a bottle disturbance demonstration, the index finger corrects the cap without maintaining a pinch.
The human input is one snapshot per task. Geometric retargeting produces a robot hand pose, with the index-finger joint angles reduced by 30% to widen the thumb–index opening. Object poses are estimated separately. Combining these estimates can cause interpenetration, so a short simulation phase holds the hand fixed while small random forces and torques settle the objects into nearby configurations. Gravity is temporarily removed from the objects during the initial warmup, giving the policy time to establish contacts.
The snapshot therefore supplies both an initial arrangement and a pose prior. Fine finger motion emerges through RL. I find this division of supervision useful: a human pose conveys how to organize the hand, while the simulator determines which motions remain feasible for the robot’s joints and contacts.
3. A small observation space, with memory
The default policy receives 34 numbers:
\[o_t=[p_t^h,\hat z_t^h,p_t^f,\hat z_t^f,q_t] \in\mathbb R^{3+3+3+3+22}.\]Each object contributes its centroid and symmetry-axis direction; the hand contributes measured joint angles. Rotation about the object’s own symmetry axis is omitted. This representation suits the nearly axisymmetric parts in these experiments, but extending it to keyed or threaded parts would require checking which orientation information the task needs. Policies trained across hand tilts additionally receive wrist roll and pitch.
There is no tactile or force input in this actor observation. A recurrent Gaussian policy uses a two-layer LayerNorm-LSTM with 1,024 hidden units, followed by a 512–128–64 ELU MLP. The history lets the controller integrate object observations with how the fingers have moved. The authors propose this as an explanation for tolerance to estimation errors; they do not supervise or directly validate an explicit internal pose-correction module.
Actions are 22 normalized joint-target increments. To distinguish the command from the measured joint angle, write the previous target as $\bar q_t$:
\[\bar q_{t+1} =\operatorname{clip}(\bar q_t+0.1a_t,q_{\min},q_{\max}), \qquad a_t\in[-1,1]^{22}.\]The policy updates at 15 Hz. Hardware deployment further limits changes to 0.12 rad per step, interpolates commands to 60 Hz, and relaxes targets when persistent tracking error indicates a stalled joint. These details matter when two objects jam against each other and prolonged position commands can overheat motors.
flowchart TD
A["One human snapshot per task"] --> B["Retargeted hand pose and object initialization"]
B --> C["Simulation PPO training"]
D["Relative assembly goal and finger-role rewards"] --> C
E["Randomized dynamics, observations and disturbances"] --> C
C --> F["Trained LSTM policy"]
G["RealSense RGB-D and object meshes"] --> H["FoundationPose and tracking filters"]
H --> I["Object centroids and symmetry axes"]
I --> F
J["Measured joint angles"] --> F
F --> K["Joint target increments at 15 Hz"]
K --> L["Hand and object motion"]
L --> G
L --> J
The simulation recipe
Appendix A specifies Isaac Sim with SDF mesh collisions, a 120 Hz physics timestep, and PPO through Isaac Lab and rl_games. Fixed-tilt training uses 256 parallel environments and 128-step rollouts; tilt-randomized training uses 1,024 environments. PPO uses four optimization epochs per rollout, a learning rate of $10^{-4}$, discount $\gamma=0.995$, and GAE $\lambda=0.95$. The reported ablation checkpoints are evaluated at training epoch 600 across three independent runs.
Randomization includes object-position observation noise with 1 cm per-axis standard deviation, object mass from 0.05 to 0.15 kg, friction and joint stiffness/damping scaled by 0.5–2.0, action delays of 0–3 policy steps, and intermittent external forces. These perturbations expose the policy to imperfect sensing and contact dynamics before transfer. A shared training formulation across tasks should not be read as evidence for one policy that assembles arbitrary objects.
4. Single-camera deployment includes a tracking system
A RealSense D435 supplies 640×480 RGB-D at 30 Hz. FoundationPose registers known object meshes using color-segmented initialization masks and then tracks their poses. The actor consumes the reduced geometric state, so this is a pose-based controller with a visual frontend.
Appendix B describes several protections against tracking the occluding hand surface: reject an object-center depth discrepancy above 8 cm, reject a frame-to-frame translation jump above 6 cm, roll back invalid tracker updates, and smooth accepted poses. Persistent low confidence triggers re-registration after 15 frames. The frontend can temporarily retain the last accepted pose for up to 120 frames. These thresholds are tracking heuristics, not the assembly tolerance.
The coordinate transform also depends on camera calibration and the CAD geometry of the hand bracket. “Single camera” describes the sensor count; it still requires object meshes, initialization masks, calibration, and pose filtering.
5. What the experiments establish
Closed-loop feedback changes the outcome
Table 1 reports 20 real trials per task and method. Alignment means insertion deeper than 1 cm; assembly means final depth within 1 cm of the target.
| Task | Closed-loop alignment | Closed-loop assembly | Open-loop alignment | Open-loop assembly |
|---|---|---|---|---|
| Bottle | 18/20 | 15/20 (75%) | 2/20 | 2/20 |
| Syringe | 18/20 | 17/20 (85%) | 1/20 | 0/20 |
| Marker | 16/20 | 16/20 (80%) | 0/20 | 0/20 |
The baseline replays successful simulation trajectories. Its failures show how quickly small execution errors accumulate without feedback. The counts are convincing evidence for closed-loop control in this setup, though the comparison does not establish superiority over other learned assembly controllers. The success thresholds also do not certify submillimeter accuracy, seating force, or a functional seal.
Removing vision or contact guidance hurts
Appendix C provides assembly success rates for Marker over 500 simulated episodes per variant under domain randomization:
| Variant | Cumulative goal reward | Assembly success |
|---|---|---|
| Full method | 1280.1 | 65% |
| Proprioception only | 338.0 | 11% |
| Eight-frame history MLP | 995.3 | 49% |
| Without finger-function reward | 629.4 | 27% |
| Without reference-pose reward | 981.8 | 40% |
The gap between 65% and 27% supports using explicit finger-role guidance during learning. Removing the pose reward also hurts, but it retains the snapshot-based initialization; this ablation does not remove all human-derived information. The history-MLP comparison changes both temporal processing and architecture, so its result supports the chosen recurrent controller within the tested training budget.
Simulation and hardware use different initial-state distributions and sample sizes. The 65% simulated Marker result and 80% real result should not be interpreted as evidence that transfer improves the policy.
Pose noise and physical occlusion are different tests
Figures 6 and 12 mainly plot cumulative goal-reaching reward. For Marker, 3 cm Gaussian observation noise gives a reward of 963.7, while a 3 cm constant offset added to the baseline Gaussian noise gives 670.9; the proprioception-only reference is 338.0. These numbers describe retained task progress, not success percentages under that noise. Large systematic offsets can be especially harmful: Syringe drops below its proprioception-only baseline.
Appendix D separately holds each object stationary and varies its visible fraction. Most valid estimates stay within 1.5 cm of the unoccluded reference, while errors above 2 cm and tracking loss count as failures and are reported separately. Severe occlusion can have a high failure rate despite small errors among the remaining valid estimates. This is a characterization of the tracker under controlled occlusion; it does not establish closed-loop assembly success at every visibility level.
6. Hand morphology and the next useful test
On simulated Syringe assembly, Sharpa and Wuji attain smaller residual errors than Allegro and XHand. Allegro can align the parts laterally but struggles to achieve insertion depth; the authors attribute this to its missing fifth finger. XHand also struggles with lateral alignment, which they connect to its more restricted finger joints. The plot uses 128 error samples from trials that did not terminate early. It suggests that finger reach, joint range, and contact placement matter for assembly, but it is not a controlled experiment isolating finger count or a general ranking of robotic hands.
Several failures expose the method’s boundary. An initially tilted cap can prevent a stable pinch, fingers can block the insertion path, and the policy may never learn to clear that obstruction. The authors also identify a mismatch between rigid-body contact approximations and soft fingertip contact patches, especially when a long object starts close to horizontal. Those orientations remain difficult even with tilt randomization.
My strongest takeaway is the division of work among fingers as a learning prior. It gives a compact way to guide exploration when full motion demonstrations transfer poorly. I would first test that prior with varied initial contact arrangements and explicit recovery from blocked insertion paths. Success after autonomous acquisition of both parts would be a stronger next result than another carefully initialized plug-in task.