[Paper Notes] Mana: Dexterous Manipulation of Articulated Tools
Published:
Mana studies a hard corner of dexterous manipulation: small articulated tools such as tongs, pliers, clothespins, and syringes. These objects require the hand to grasp thin moving parts, stabilize the tool body, and apply enough force to actuate an internal joint.
My read: the main contribution is a data-generation view of tool use. Mana treats manipulation like animation. A few human affordance clicks define functional regions; procedural keyframes sketch the important poses; motion planning fills geometric reaching; short-horizon RL fills contact-rich inbetweening; a point-cloud diffusion policy then learns from the generated simulation trajectories and transfers zero-shot to the real robot.
Paper Info
The paper is “Mana: Dexterous Manipulation of Articulated Tools” by Zhao-Heng Yin, Guanya Shi, Pieter Abbeel, and C. Karen Liu from UC Berkeley, CMU, Stanford University, and Amazon FAR. It is available as arXiv:2606.13677, with project page zhaohengyin.github.io/mana.
Why Articulated Tools Are Hard
Rigid object manipulation already has difficult contact dynamics, but articulated tools add another layer: the robot must control the tool’s internal degree of freedom while keeping the whole tool stable. A tong must open and close; a plier must squeeze; a syringe requires plunger motion; a clothespin needs forceful actuation around a spring-loaded joint.
The paper emphasizes that successful force directions are often misaligned with stable surface normals. Pressing along the normal may reduce slip, yet it can rotate or eject the tool. Pressing in the desired actuation direction may generate the correct moment, yet it may also leave the friction cone. Small contact-location errors change moment arms and can destabilize the entire coupled hand-tool system.
This also explains why common data sources are weak here. Position-based teleoperation retargets fingertip poses, so it cannot reliably command the precise contact forces needed for centimeter-scale handles. End-to-end RL from scratch faces a sparse exploration problem: it must discover functional contacts, maintain them as the tool moves, and generate large task-specific forces in a high-dimensional hand action space.
Mana as Manipulation Animation
Mana reframes articulated tool use as a coarse-to-fine motion synthesis problem. The input is a tool mesh, an articulated joint model, and a small set of human-provided functional affordance annotations. The user marks regions such as handles, arms, plungers, or barrels in a 3D interface; the paper reports that this takes less than one minute per tool instance.
The output is a dataset of successful simulated manipulation trajectories. Each trajectory contains wrist motion, hand joint commands, object state, and semantic phase labels such as grasping, opening, and closing. These trajectories become training data for the real-world visuomotor policy.
The pipeline has three main layers.
Grasp generator. Mana builds on Lightning Grasp and adds a collision-aware IK procedure, called Lightning Grasp+, for thin tabletop objects. It uses annotated functional regions to construct contact domains, optimizes fingertip contact locations and finger joints, filters grasps in IsaacLab, and samples dense keyframes across relevant tool configurations.
Trajectory generator. Mana decomposes the episode into pre-grasping, grasping, and in-hand actuation. Pre-grasping uses GPU-accelerated RRT-Connect because it is mostly geometric. Grasping can be procedural when contact normals and squeezing motions are reliable, and it can use RL for thin or unstable cases. In-hand actuation uses RL because it needs coupled position-force control.
Visuomotor policy. After simulation data is generated, Mana trains a point-cloud-conditioned diffusion policy. The policy observes the segmented tool point cloud in the wrist frame plus proprioception, then outputs delta 6D wrist poses and delta hand joint position targets.
The RL Inbetweening Objective
The RL part is used for the phases where geometry alone is insufficient. Each episode starts from a generated stable grasp or pre-grasp keyframe and targets a meaningful tool configuration, such as moving from open to closed.
The dense reward has the form:
[ r = r_{\mathrm{tool}} + w_1 r_{\mathrm{hand}} + w_2 r_{\mathrm{contact}}. ]
The tool term tracks the target tool joint, position, and orientation. The hand term regularizes the hand gesture, and in grasping also tracks wrist pose. The contact term counts active contacts between finger links and tool links:
[ r_{\mathrm{contact}}=\sum_{i\in\mathrm{Finger},j\in\mathrm{Tool}} \mathbf{1}[f_{ij}>\epsilon]. ]
The appendix makes the physical story clearer. The RL teachers are trained with strong force-related randomization: action noise, robot PD gain randomization, tool PD gain randomization, tool mass randomization, friction randomization, and object force perturbations. The goal is to prevent policies from depending on a single brittle force balance.
This matters because tool use is force sensitive. A policy that looks good under one simulated stiffness, friction, or mass can fail immediately when the real tool pushes back differently.
Real Robot Policy and Hardware
The real platform is a 7-DoF xArm7 with a 16-DoF Allegro hand. Mana also uses custom flattened compliant fingertips because standard hemispherical rigid fingertips create unstable point contacts on thin handles. The soft silicone layer enlarges the contact patch and helps tolerate small pose errors during high-force actuation.
For perception, the system uses one RealSense D435 RGB-D camera. During deployment, SAM 3 segments the image, Fast Foundation Stereo produces an object point cloud, and the point cloud is represented in the wrist frame. The system runs at about 10 Hz on a workstation with two RTX 4090 GPUs.
The student policy uses 512 point-cloud points. A Perceiver-style transformer compresses the point cloud into four 128-dimensional tokens. Proprioception, including hand joint positions and targets from the past two frames, is encoded into another token. A lightweight transformer diffusion head then predicts the action. The policy is trained with a standard denoising objective:
[ L=\mathbb{E}_{(o_i,a_i)\sim D,\;t\sim U[0,T]} \left[\lVert a_i-\pi(\tilde{a}_i,o_i,t)\rVert^2\right]. ]
Point-cloud randomization, including noise and random part masking, is used to bridge imperfect segmentation and depth noise in real deployment.
Results
Mana evaluates four articulated object categories: tongs, pliers, clothespins, and syringes, with two instances per category. The tools are thin, about 0.8-1.5 cm thick, and require around 3-7 N of actuation force.
The main table reports roughly 70% success for both grasping and in-hand manipulation across categories, with each cell evaluated over 10 trials per instance. Teleoperation performs poorly, often near zero on clothespins and syringes, and only around 30% on some tong phases. Open-loop execution of Mana-generated trajectories is better than teleoperation but still sensitive to mesh and pose errors. The closed-loop learned policy is consistently strongest.
The ablations support the data-generation thesis. Real-world success improves with more generated trajectories, more grasp keyframes, and stronger force randomization. That is an important result: for articulated tool use, robustness scales with dense state coverage around functional contacts and with physical diversity in simulation.
The paper also composes learned skills into functional tasks, using manual wrist teleoperation only for fine alignment to task sites. It reports 7/10 success for tong picking, 5/10 for plier cutting, 6/10 for clothespin use, and 5/10 for syringe injection. These numbers are lower than phase-level results because tool-object interaction introduces unseen perturbations and additional slip modes.
Strengths and Limitations
The strength of Mana is its decomposition. It avoids asking one method to solve everything. Human clicks provide functional intent. Procedural grasp generation gives the system contact candidates that RL would struggle to discover. Motion planning handles collision-free approach. RL focuses on short contact-rich segments. Diffusion policy learning turns the generated trajectories into a perception-conditioned controller.
The limitations are also concrete. The Allegro hand cannot handle stiff tools requiring more than about 10 N because of motor torque limits. The work focuses on precision grasps and does not explore human-like power grasps, partly because the Allegro hand is around twice the size of a human hand. Perception under occlusion and slip detection remain difficult, especially for very small tools. Finally, the demonstrated tool-use workflows still use wrist teleoperation for fine alignment, so full autonomous skill chaining needs another policy layer.
Takeaway
Mana is a useful reminder that dexterous manipulation data does not need to come only from human demonstrations or blind RL exploration. For articulated tools, the most effective supervision may be structured: sparse affordance annotation, dense procedural keyframes, simulation-based inbetweening, and learned closed-loop deployment.
The larger lesson is that contact-rich tool use benefits from separating function, geometry, force, and perception. Mana gives each part a suitable mechanism, then composes them into a sim-to-real pipeline that can grasp and actuate thin articulated tools with meaningful real-world success.