[Paper Notes] DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A bimanual reference can ask one hand to support an object while the other releases and repositions. Learning that motion requires useful feedback for each hand, including when its partner fails. DONIC combines object tracking with contact rewards assigned to each hand’s reference role, then trains a shared policy using PPO and verified successful demonstrations. Clip-Centered Advantages (CCA) remove a reference-specific mean from the policy’s learning signal.
The public-data model reaches 86.79% success on unseen trajectories and 57.32% on unseen objects. These numbers measure reference tracking in simulation. The interesting research question is how much of the improvement comes from the reward, the executable demonstration library, and the way heterogeneous trajectories share policy updates.
Paper and source version
DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations, by Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao, Fangzhou Hong, Zhaoxi Chen, and Ziwei Liu, all affiliated with Ropedia. These notes follow arXiv:2610.29997v1, submitted October 21, 2026, and its 26-page PDF. The authors provide an official project page. All numerical results below are author-reported; the final research suggestions are my interpretation.
Figure 2 from Sun et al., DONIC, arXiv:2610.29997v1, p. 6, cropped from the PDF; CC BY 4.0. The demonstration library receives verified executions from specialists and the shared policy.
1. The policy corrects a supplied motion reference
DONIC receives trajectories for both robot hands and the object. Each simulated Wuji Hand V1 has six wrist coordinates and twenty finger joints, giving a 52-dimensional action for the pair. The 3,016-dimensional observation includes proprioception, object pose and velocity, upcoming reference targets, tracking errors, fingertip-to-object vectors, and an object descriptor with 128 basis-point-set features. Forty contact-force channels are zeroed; simulated forces are used in the reward.
At each step, the policy outputs bounded corrections around the next reference configuration:
\[\mathbf q_t^{\mathrm{pd}} =\operatorname{clamp}\left( \hat{\mathbf q}_{t+1}+\mathbf s\odot\mathbf a_t, \mathbf q_{\min},\mathbf q_{\max}\right), \qquad \mathbf a_t\in[-1,1]^{52}.\]The PD controller executes this target. Corrections are re-anchored to the reference every step and do not accumulate. Their bounds are 0.1 m for wrist translation, 1.0 rad for wrist rotation, and 0.15 rad for finger joints (Appendix C.1).
This defines the scope of the generalist: one checkpoint tracks new supplied references across objects and motions without further training. The interface still requires reference construction, object geometry, and state estimates. It also models wrist motion through six virtual coordinates, so deploying on a complete arm-hand system introduces additional kinematic and control constraints.
2. Give object motion and each hand a usable reward
The object reward multiplies agreement in position, orientation, and linear velocity:
\[r_t^{\mathrm{obj}} =e^{-\kappa_p\|\mathbf p_t^o-\hat{\mathbf p}_t^o\|_2} e^{-\kappa_R d_{SO(3)}(R_t^o,\hat R_t^o)} e^{-\kappa_v\|\mathbf v_t^o-\hat{\mathbf v}_t^o\|_2}.\]This term remains active independently of the contact gate. Velocity agreement penalizes holding the object stationary while the reference moves. The coefficients are $(\kappa_p,\kappa_R,\kappa_v)=(9,2,14)$, with distances in metres and velocities in metres per second.
Contact supervision starts from a reference-defined set $\mathcal C_t^h$: the fingertips of hand $h$ that should contact the object at time $t$. A simulated fingertip counts as contacting when its net contact force exceeds 0.5 N and its distance to the object surface is below 3 cm:
\[\alpha_{i,t}^h =\mathbf 1[f_{i,t}^h>0.5\,\mathrm N]\, \mathbf 1[\delta_{i,t}^h<0.03\,\mathrm m].\]For each required fingertip, the contact reward mixes a smooth proximity term with this binary indicator, using equal weights. A separate hold reward measures the fraction of required fingertips currently in contact. It permits sliding and rolling: instantaneous contact coverage does not impose a no-slip constraint. Net force also cannot identify which body caused a collision, so the distance check improves contact attribution while remaining a proxy.
The aggregation across hands is especially useful. Define $\mathcal E_t=\lbrace h:\mathcal C_t^h\neq\varnothing\rbrace$. For either contact-related reward $u$,
\[r_t^u= \frac{\sum_{h\in\mathcal E_t}w_h r_t^{u,h}} {\sum_{h\in\mathcal E_t}w_h}, \quad \mathcal E_t\neq\varnothing,\]and set it to zero when no hand is required to contact. The implementation fixes $w_L=0.5$ and $w_R=1$. A hand scheduled to release contributes no contact reward; if both hands should engage and one fails, the other still supplies feedback. The full reward adds hand tracking, object tracking, contact acquisition, and contact coverage with coefficients $(0.25,0.45,0.15,0.15)$, then multiplies the sum by $1+w_L$ (Section 4.3 and Appendix C.2).
3. CCA changes the comparison within each reference
PPO batches mix trajectories with different contact patterns and difficulty. The authors argue that unequal critic accuracy can leave reference-dependent offsets in GAE advantages. A relatively useful transition in a difficult clip may then receive a negative advantage along with the rest of that clip.
CCA groups on-policy transitions by reference clip identity. Multiple parallel environments executing the same reference contribute to the same group $\mathcal G_m$. After ordinary GAE, it computes
\[\bar A_m=\frac{1}{|\mathcal G_m|} \sum_{j\in\mathcal G_m}A_j^{\mathrm{GAE}}, \qquad \widetilde A_j=A_j^{\mathrm{GAE}}-\bar A_{m(j)}.\]As an illustrative calculation, advantages $\lbrack -8,-6,-4\rbrack$ become $\lbrack -2,0,2\rbrack$. The best transition in that group now has a positive learning signal, while every pairwise difference is preserved. This example explains the mean subtraction; it is not an experimental result.
CCA subtracts only the group mean. There is no division by a per-clip standard deviation, and the critic’s regression targets remain unchanged. The centered values enter the usual clipped PPO actor loss. Appendix C.3 explicitly cautions that these algebraic properties do not establish an unbiased policy-gradient estimator. I read CCA as a practical adjustment for joint training whose value should be tested against critic quality, group size, and sampling balance.
4. Successful robot executions supply the BC targets
The human recordings first become geometric references. BiHOI contributes 1,575 sequences, 229 objects, and 14.77 hours, captured with 32 RGB cameras. Its 65.91% simultaneous-contact rate uses frames with at least one hand in contact as the denominator. Multi-view reconstruction estimates hand motion, object geometry, and object pose; kinematic fitting maps the hands to the robot. Appendix B.2 states that joint reconstruction refinement has no explicit contact or collision loss.
Open-loop replay with zero policy correction screens references before training. Immediate object loss and ejection are excluded; contact without lifting and partially executed references remain eligible because feedback could recover them. Trajectory-specific policies then seek executable versions of these references. Successful rollouts must pass replay verification before entering the demonstration library, and verified successes of the shared policy are added later. Verification checks object displacement and lifting when required; the benchmark uses a separate tracking-fidelity criterion.
The actor objective is
\[\mathcal L_{\mathrm{actor}}(\theta) =\mathcal L_{\mathrm{PPO}}(\theta;\widetilde A) +\lambda_{\mathrm{BC}} \mathbb E_{(\mathbf o^D,\mathbf a^D)\sim\mathcal D} \left\|\mu_\theta(\mathbf o^D)-\mathbf a^D\right\|_2^2.\]BC regresses the policy mean toward verified robot actions paired with their execution states. Demonstration samples use a separate budget and do not enter PPO’s importance ratio or GAE. This distinction explains why directly cloning retargeted human poses would omit part of DONIC’s training signal.
The reported recipe uses Isaac Gym with 8,192 parallel environments, separate actor and critic MLPs of widths $\lbrack 1024,1024,1024,512\rbrack$, and Adam at $2\times10^{-5}$. Each rollout spans 64 steps; PPO uses minibatches of 16,384 and six passes. BC uses 4,096 pairs per minibatch with $\lambda_{\mathrm{BC}}=50$. Adaptive recovery sampling and object-position perturbation curricula are disabled (Appendices C.4–C.5).
5. Read the success rate together with its thresholds
The public-data training pool contains 1,845 clips from GRAB, GigaHands, OakInk2, TACO, and H2O. Adding BiHOI raises it to 3,420. Evaluation reserves 53 unseen trajectories of training objects and 164 clips involving held-out objects; these references are excluded from policy training, specialist training, and demonstration collection.
| Method | Unseen trajectory SR | Unseen object SR | Unseen object Comp |
|---|---|---|---|
| DexTrack-bi | 0.00% | 0.00% | 0.88% |
| Dexplore-bi | 9.43% | 11.59% | 26.37% |
| DONIC without CCA | 81.13% | 48.17% | 43.89% |
| DONIC without contact gate | 81.13% | 48.78% | 49.52% |
| DONIC | 86.79% | 57.32% | 54.69% |
| DONIC + BiHOI | 88.68% | 59.76% | 55.35% |
Selected columns from Table 2. Each method uses one checkpoint across both splits, without fine-tuning. The two baselines are adaptations of released single-hand tracking systems to the bimanual interface.
For each trajectory, SR requires both hands to pass the following tests on trajectory-averaged errors:
\[\frac{T_o+T_h}{2}\leq10\,\mathrm{cm}, \qquad \frac{R_o+R_h}{2}\leq60^\circ, \qquad E_{\mathrm{finger}}\leq40^\circ.\]Comp applies the same tests frame by frame and averages the valid-frame fraction across trajectories. A valid frame after an earlier violation still counts. Consequently, SR can be substantially higher than Comp: DONIC’s unseen-trajectory values are 86.79% and 67.62%. These metrics assess tracking fidelity under the specified tolerances; task completion such as successfully fastening or assembling an object would need its own test.
Object error also gives context. DONIC’s unseen-object translation error is 15.74 cm, falling to 13.96 cm with BiHOI. The success test averages object and wrist errors, so a successful trajectory need not keep object error alone below 10 cm.
The ablations isolate narrower changes than their labels might suggest. Removing CCA costs 9.15 percentage points of unseen-object SR. Removing the contact gate costs 8.54 points, but Appendix D.4 defines this as deleting the 3 cm proximity condition from the contact proxy while retaining the force test, object reward, and reward weights. It does not ablate the entire object-centered per-hand reward. Adding BiHOI improves unseen-object SR by 2.44 points; the experiment adds data volume and diversity together.
6. What I would reuse and what I would test next
I would first reuse the separation between object-motion feedback and reference-dependent hand contact roles. It offers a direct way to train asymmetric coordination without suppressing one hand’s progress when its partner fails. CCA is a small additional experiment once the observation-action interface and verified demonstration pipeline work.
The large baseline gap does not by itself attribute the gain to either component. A reproduction should hold the demonstration library and interaction budget fixed, then compare the full per-hand reward against alternatives and CCA against batch-wide centering. That would help distinguish improved exploration, better imitation targets, and a better policy update.
The manuscript’s quantitative scope is reference-conditioned tracking of a single rigid object in simulation. Figure 1 does show real-world evaluation examples; the reported held-out success rates remain simulation results, and the text does not provide a matching quantitative hardware benchmark. Held-out operation-category generalization, unconstrained goal-conditioned manipulation, and broader multi-object interaction remain outside the reported quantitative evaluation (Appendix D.6). For hardware use, the next evidence I would want is tracking and task success under measured state-estimation error, physical arm constraints, and repeated contact disturbances.