[Paper Notes] DITTO-X: Forward and Reverse Teleoperation for Dexterous Manipulation and Human Intervention
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A robot is holding a pair of tongs when its policy needs help. If the operator takes over with a different hand pose, the first human command can loosen the grasp and drop the tool. DITTO-X addresses this transfer: an actuated hand exoskeleton moves the operator’s fingers along with the policy before intervention, then lets the operator assume control from an already aligned configuration.
The same hardware supplies joint-level force feedback and, when fingertip tactile sensing is available, a brief vibration at contact onset. It supports Sharpa Wave, Wuji 2, and Inspire RH56DFX through hand-specific mappings. The strongest learning evidence comes from Sharpa: after two intervention rounds, task success reaches 86.7% for tong use, 93.3% for raspberry placement, and 90.0% for battery insertion, with 30 evaluation rollouts per task and condition.
Paper and source version
DITTO-X: Forward and Reverse Teleoperation for Dexterous Manipulation and Human Intervention is by Zhanpeng He, Joaquin Palacios, Zhangyu Wang, Chenhao Li, Katelyn Lee, Matei Ciocarlie, C. Karen Liu, and Jiajun Wu, from Stanford University and Columbia University. These notes follow arXiv:2610.00781v2, revised October 5, 2026, and its nine-page PDF. The first submission was September 30. The project website includes demonstrations. Results below are author-reported; the source is an arXiv preprint.
1. Preparing the human hand for takeover
During autonomous execution, the robot may already be squeezing a fragile object or applying force through a tool. Camera views can hide the contact, and a passive tracking glove leaves the operator to reconstruct the robot’s grasp. DITTO-X uses its actuators to make this configuration available through the operator’s own fingers.
The two operating modes exchange who supplies the motion reference:
| Mode | Motion reference | Exoskeleton’s role |
|---|---|---|
| Forward teleoperation | Operator’s measured hand configuration | Measure motion and render contact feedback while the robot follows |
| Reverse teleoperation | Policy’s commanded robot-hand configuration | Drive the operator’s fingers toward its projection onto the exoskeleton |
Section III-B specifically projects the commanded configuration. Contact and tracking error can separate that command from the robot’s measured state; the paper does not establish exact physical state equality under every contact condition.
Let $g_h$ denote the forward position mapping for hand $h$, and $P_h$ its reverse projection. The following is a compact restatement of the mechanism:
\[q_f^{\mathrm{cmd}}=g_h(q_l),\qquad q_l^{\mathrm{proj}}=P_h(q_f^{\mathrm{cmd}}).\]Here $l$ denotes the leader exoskeleton and $f$ the follower robot hand. For joints with direct correspondence, the reverse mapping is exact. The thumb needs geometric retargeting: compute the robot fingertip pose with forward kinematics, express it in the exoskeleton palm frame, undo the hand-specific length scale $s_f$, and solve exoskeleton inverse kinematics. Under different kinematics or actuation constraints, $P_h$ should be understood as a projection into the available motion space.
A keypress requests control; movement completes the transfer
After a keyboard takeover request, policy control continues until at least one measured exoskeleton joint differs from the projection by more than $\delta_{\mathrm{release}}$. In compact notation, the gate is
\[\max_j\left|q_{l,j}^{\mathrm{meas}}-q_{l,j}^{\mathrm{proj}}\right| >\delta_{\mathrm{release}}.\]An accidental keypress alone therefore does not transfer authority. The operator must actively depart from the pose that the exoskeleton is following. Control can later return to the policy through the keyboard trigger, allowing several interventions within one episode.
The inverse-kinematics initialization matters here. Each solve starts from the previous projection; only the first solve starts from the measured human pose. If a redundant IK solver were initialized from the currently displaced hand at every step, it could follow that displacement along equivalent solutions. The reference would move toward the operator, reduce the measured error, and hide the action meant to open the takeover gate. Keeping the projection independent preserves a usable signal of human intent.
This is the implementation detail I would check first in a reproduction. A geometrically valid inverse mapping can still break the authority-transfer logic if its initialization absorbs the operator’s motion.
2. How one exoskeleton controls three hands
DITTO-X extends the earlier DITTO interface with a motorized middle-finger module and a simplified thumb. The robot’s ring and little fingers follow the middle finger, so the interface independently controls three digits while retaining power grasps involving all five.
For Sharpa and Wuji, the index and middle fingers admit joint correspondence for MCP abduction/adduction, MCP flexion, and PIP flexion. Unmatched joints are held at neutral angles. Thumb geometry varies more, so thumb control uses fingertip retargeting in palm frames anchored at the index MCP joint. Inspire’s underactuated fingers receive a single flexion command derived from total exoskeleton finger flexion.
| Robot hand | Actuated DoF | Feedback used by DITTO-X |
|---|---|---|
| Sharpa Wave | 22 | Current-based torque estimates, fingertip tactile forces, and contact vibration |
| Wuji 2 | 20 | Current-based torque estimates; no fingertip contact vibration |
| Inspire RH56DFX | 6 | One force magnitude per finger, mapped into exoskeleton feedback; no fingertip contact vibration |
Table I groups Inspire’s sensing under a scalar per-finger entry; Section III-A2 describes the exposed quantity as a force magnitude. It is not a full vector of independently sensed joint torques. More generally, support across hands still requires position mappings, scale factors, and sensing-specific feedback choices. The shared component is the exoskeleton hardware.
3. Sustained force and contact onset use different feedback paths
Mapping robot contact into exoskeleton torques
Where corresponding joints exist, current-derived robot torque estimates can be assigned directly to the matching exoskeleton joints. For the thumb, the method passes through fingertip force. Following Eq. (1),
\[\hat F_f=(J_f^\top)^\dagger\hat\tau_f, \qquad \hat\tau_l^{\mathrm{des}} =J_l^\top R_lR_f^\top\hat F_f.\]$J_f$ and $J_l$ are fingertip position Jacobians, $\dagger$ denotes the pseudoinverse, and the rotation term changes frames using the paper’s convention. Sharpa’s tactile force vector can enter this mapping directly, bypassing the current-to-force estimate. Inspire’s scalar force measurement supplies a more restricted input.
Each desired torque is low-pass filtered and converted to motor current. Eq. (2) can be written equivalently as
\[i_{l,j}^{\mathrm{cmd}} =-k_j\operatorname{sign}(\tilde\tau_{l,j}^{\mathrm{des}}) \max\!\left(\left|\tilde\tau_{l,j}^{\mathrm{des}}\right|-\tau_{\mathrm{th},j},0\right).\]The gain sets the rendered strength. For current-derived estimates, the deadband rejects bias during free motion; for tactile-derived torques, the threshold is zero. The device also uses software joint limits, conservative joint-dependent current limits below 200 mA, reported torque limits of at most 0.08 N·m, and an emergency stop.
A short vibration marks a new contact
Filtering helps stabilize force feedback but attenuates rapid changes. DITTO-X adds linear resonant actuators beneath the operator’s fingertips to signal contact onset. This path is implemented only for Sharpa, which supplies fingertip tactile measurements.
The contact detector selects different force thresholds during loading and unloading to account for sensor hysteresis. On a transition into contact, force magnitude sets the initial vibration intensity, which decays to zero over about 0.5 seconds. Vibration remains off until contact clears and a new onset occurs. Sustained grasp force continues through the motor-torque path. This avoids using continuous vibration to represent a continuously held object.
In the six-person study, with vision removed, both feedback modes together yield 86.1% size identification and 91.7% compliance identification, against a three-choice chance level of 33.3%. Removing vibration gives 61.1% and 80.6%; removing force feedback gives 61.1% and 47.2%. Each participant performs six trials per condition in each identification task. These ablations support complementary information from the two channels within this setup.
4. What improves in data collection and intervention?
The user study compares DITTO-X with Manus Pro, a passive hand-tracking glove. Six subjects perform ten trials per interface for full tool-use collection (T3) and ten for intervention (T4). In T4, both interfaces take over the same type of policy trained on DITTO-X demonstrations after it has picked up the tongs. Dropping the tool counts as task failure.
| User-study condition | DITTO-X success | Manus success | Time per success, DITTO-X / Manus |
|---|---|---|---|
| T3: full tool-use collection | 70.0% | 46.7% | 56.1 / 85.5 s |
| T4: takeover after tool acquisition | 78.3% | 27.7% | 35.7 / 97.4 s |
These are the values reported in Section V-A; the paired success comparisons have $p=0.005$ for T3 and $p=0.007$ for T4. The paper prints 27.7% for Manus T4, which these notes retain. T4 asks the human to complete a shorter portion of the task than T3, so their absolute rates do not measure identical difficulty.
For policy learning, a proficient operator collects equal numbers of demonstrations with each interface: 80 for tong use and 60 each for raspberry placement and battery insertion. The arm is a Franka Panda. All policies use the UNet variant of Diffusion Policy, trained for 50 epochs with the same architecture and hyperparameters. Observations include wrist and side images, end-effector pose, hand joint angles, and tactile signals when available.
Each task and interface then receives two DAgger rounds of 20 intervention trajectories, with ten fine-tuning epochs per round on accumulated data. The operator chooses when to intervene; successful interventions are retained. Every DITTO-X intervention uses reverse teleoperation. Each interface corrects its own learned policy, so the two branches encounter different failure distributions.
5. Corrective data changes which states the policy learns from
Figures 7–8 and Table II report the following final-task success rates. Every policy condition is evaluated over 30 rollouts, using matched starting configurations; at least ten of those configurations per task fall outside the pretraining object-placement distribution.
| Task | Manus initial | DITTO-X initial | DITTO-X DAgger r1 | DITTO-X DAgger r2 | DITTO-X equal-count, non-iterative data |
|---|---|---|---|---|---|
| Tong use | 13.3% | 63.3% | 73.3% | 86.7% | 70.0% |
| Raspberry placement | 43.3% | 63.3% | 73.3% | 93.3% | 76.7% |
| Battery insertion | 36.7% | 40.0% | 56.7% | 90.0% | 36.7% |
The last column uses the same demonstration counts as round two: 120 for tong use and 100 for each other task, collected non-iteratively from the original task distribution. Its gap to DAgger supports the value of collecting corrections at states the current policy actually visits and struggles with. The control matches trajectory counts, not necessarily operator time or total training frames.
The stage breakdown is more informative than the overall tong score alone. Initial Manus and DITTO-X policies acquire the tongs at similar rates, 73.3% and 76.7%. Their cumulative success at picking up the toy then separates to 23.3% and 66.7%. The useful difference appears when the grasp must transmit force through the tool. For battery insertion, DITTO-X’s second correction round increases final insertion success from 56.7% to 90.0%, after grasping and alignment have already become more reliable.
Manus DAgger ends at 36.7%, 43.3%, and 46.7% on these tasks; tong and raspberry decline from round one. The authors attribute this to corrections that first disturb an existing grasp during takeover. That is a plausible explanation supported by qualitative examples, but the comparison changes the complete interface. It does not isolate reverse teleoperation from force feedback with a DITTO-X intervention ablation.
6. Scope and the next experiment
Wuji policies reach 83.3% on toy pick-and-place and 66.7% on drawer manipulation; Inspire reaches 93.3% on bunny lifting. Reverse teleoperation is implemented for both hands, while DAgger experiments on them remain future work. These results demonstrate reusable collection hardware across several embodiments; they do not establish a single policy that transfers between hands.
The evaluation has six user-study participants, one proficient collector for the policy-data comparisons, and 30 rollouts per policy condition. It leaves uncertainty about variation across operators, independently trained seeds, and a broader range of contact tasks. Coupling the ring and little fingers also limits independent digit control. Current-based force estimates and retargeting remain dependent on the target hand’s sensing and geometry.
My takeaway is to treat takeover quality as part of the data-collection method. A human correction can begin by breaking a good grasp, and the resulting recovery then enters the training set. For a system that often requests help while holding tools, I would test pose alignment before expanding demonstration volume. The decisive next experiment would keep the DITTO-X hardware, force feedback, policy, and starting states fixed, toggle reverse projection, and measure grasp loss immediately after takeover alongside the quality of the resulting correction data. That would establish how much of the learning benefit comes specifically from preparing the operator’s hand.