[Paper Notes] NestDex: Nested Policy Learning with Copilot Assisted Teleoperation for Dexterous Manipulation
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
NestDex changes the role of learned hand policies in dexterous robot learning. Small proprioceptive policies first learn reusable hand skills such as grasping, pinching, or button pressing. During data collection, these inner policies act as a copilot: the human moves the robot arm and controls skill progress with a reversible one-degree-of-freedom clutch, while the selected policy generates coordinated finger motion from recent joint positions and efforts. A vision-language selector chooses the appropriate skill at task boundaries.
The resulting complete-task demonstrations train a separate outer visuomotor policy that predicts both arm and hand actions. The inner policies and vision-language selector disappear at deployment. A hand variational autoencoder, or H-VAE, compresses each 20-dimensional hand command into a 10-dimensional latent action so the outer policy can learn coordinated finger motion from only 20 successful trajectories per task.
Across six real-world tasks, copilot-assisted collection succeeds in 100% of 20 attempts per task; the AnyTeleop baseline ranges from 0% to 75%. Autonomous outer policies with H-VAE reach 100%, 75%, 90%, and 100% success on four single-arm tasks. In a bottle-grasp study, fixed-command replay succeeds in 3/10 trials, closed-loop prediction in 7/10, and closed-loop prediction with temporal ensembling in 9/10. The paper’s useful lesson is that learned skills can be temporary infrastructure for producing better training data, while the final policy remains end-to-end and independent.
Paper Info
“NestDex: Nested Policy Learning with Copilot Assisted Teleoperation for Dexterous Manipulation” is by James Zhao, Jinhe Tang, Mingyuan Ba, and Weiming Zhi, with affiliations at the University of Sydney, the Australian Centre for Robotics, and Vanderbilt University. This note covers arXiv:2608.13362v1, posted on August 13, 2026. The project page provides system diagrams and real-robot videos.
1. The Demonstration Bottleneck
Dexterous imitation learning has a data problem before policy training begins. A parallel-jaw gripper exposes roughly one hand coordinate, while a five-finger hand demands continuous coordination across many joints and changing contact states. A teleoperator must simultaneously decide where the arm should move and how every finger should interact with the object. Errors in either stream can invalidate a complete long-horizon demonstration.
NestDex divides the work into two nested layers:
- The inner layer contains reusable, contact-aware hand policies. It assists a human during demonstration collection.
- The outer layer is a visuomotor policy trained on those complete demonstrations. It controls the full arm–hand system autonomously.
This separation distinguishes NestDex from a conventional hierarchical controller. The skill library and selector support data generation; they are absent from the final autonomous controller. The outer policy learns the complete visual-to-action mapping from the demonstrations that the nested system made easier to collect.
2. Learning the Inner Hand Skills
Multi-View Retargeting
Each inner skill begins with natural human-hand demonstrations. Synchronized keypoints from several calibrated cameras are triangulated into a 3D hand pose, reducing the depth ambiguity and occlusion of a single view. The robot configuration is obtained with a robust version of AnyTeleop’s vector retargeting objective:
[ q_t^=\arg\min_{q_{\min}\le q\le q_{\max}} \sum_{i=1}^{M} \rho_\delta!\left(\left|\alpha v_{i,t}^{h}-v_i^{r}(q)\right|_2\right) +\beta\left|q-q_{t-1}^\right|_2^2. ]
Human and robot hand vectors are matched after a scale correction (\alpha). The Huber penalty (\rho_\delta) limits the influence of large tracking errors, and the final term encourages temporal smoothness. The recorded skill trajectory contains measured robot-hand joint positions (q_t) and efforts (e_t).
Proprioceptive Action-Chunk Policies
One Transformer policy is trained for each hand skill. Its state is
[ x_t=[q_t,e_t], ]
and each training example maps a history of (h) states to a chunk of (H_{\mathrm{in}}) future joint-position commands:
[ o_t=[x_{t-h+1},\ldots,x_t], \qquad A_t=[q_{t+1},\ldots,q_{t+H_{\mathrm{in}}}]. ]
The experiments use a four-encoder-layer, one-decoder-layer Transformer, a 30-step observation history, a 30-step action chunk, and 10 trajectories per skill. Training runs for 20,000 steps. Inner-policy control runs at 100 Hz.
The policy sees no camera image or object identity. Joint positions reveal where motion has reached; joint efforts reflect evolving contact. A grasp policy trained on four objects consequently produces different hand configurations as different physical constraints appear.
3. The Copilot Interface
The operator directly controls a 7-DoF follower arm through a matching leader arm. A one-DoF clutch controls normalized hand-skill progress:
[ p_t=\operatorname{clip}!\left( \frac{c_t-c_{\mathrm{start}}}{c_{\mathrm{end}}-c_{\mathrm{start}}},0,1 \right). ]
The progress value selects a target index along the learned skill:
[ s_t^{\mathrm{in}}=\left\lfloor p_t(T_{\max}-1)\right\rfloor. ]
The current execution index moves toward that target by at most one step per control cycle:
[ r_{t+1}^{\mathrm{in}}=r_t^{\mathrm{in}}+ \operatorname{clip}(s_t^{\mathrm{in}}-r_t^{\mathrm{in}},-1,1). ]
Forward motion queries the inner policy using the latest proprioceptive history, so new commands incorporate the contact state created by earlier commands. Reversing the clutch walks backward through buffered commands. The operator can reopen a grasp, reposition the arm, and resume forward prediction from the updated physical state.
A pretrained vision-language agent receives the wrist image and a numbered list of skill descriptions. It chooses a skill at startup and whenever the active skill has been fully reversed to index zero. Selection is locked during skill execution, which avoids mid-motion switching. Toast Preparation, for example, uses Tongs Grasp, Button Press, Plate Grasp, and Tongs Grasp again; the same inner skill can reappear at multiple task stages.
4. From Assisted Collection to Autonomous Control
Each complete demonstration records wrist images, arm and hand state, and arm and hand commands. The hand command has 20 dimensions, with strong correlations among finger joints. NestDex trains a task-specific H-VAE to encode it as
[ z_k=\mu_\phi(a_k^{\mathrm{hand}})\in\mathbb R^{10}. ]
The posterior mean removes sampling noise from the behavior-cloning target. Arm commands remain in their original 7-dimensional joint space, so one outer-policy label is
[ a_k=[a_k^{\mathrm{arm}},z_k]. ]
The decoder reconstructs the complete hand command during deployment. The H-VAE uses hidden layers of sizes 128 and 64 and trains for 100 epochs with reconstruction and KL losses.
The outer policy is a visuomotor Transformer with four encoder layers, one decoder layer, and DINOv3 visual features. It receives a (256\times256) wrist image plus arm–hand joint positions and efforts, then predicts a 100-step action chunk. Training uses behavior cloning for 50,000 steps. At inference, overlapping chunks are combined through temporal ensembling, and the H-VAE decoder converts predicted hand latents back to 20 joint commands.
5. Tasks and Evaluation Protocol
The follower platform combines a 7-DoF Piper Nero arm, a 20-DoF five-finger WujiHand I, and a wrist camera. The six tasks span tool use, simultaneous multi-object grasping, sequential object transfer, and bimanual long-horizon manipulation:
| Task | Mode | Goal |
|---|---|---|
| Tongs Transfer | Single arm | Extract tongs, grasp a carrot, and move it to a pan |
| Bottle Disposal | Single arm | Open a bin, grasp a bottle, and place it inside |
| Dual-Object Transfer | Single arm | Grasp two blocks with separate finger groups and transfer both |
| Ingredient and Pot Transfer | Single arm | Move an ingredient into a pot, then relocate the pot |
| Toast Preparation | Dual arm | Use tongs, operate a toaster, position a plate, and return the toast |
| Binder Filing | Dual arm | Retrieve paper, punch it, insert it into a binder, and close the rings |
For demonstration collection, one trained operator performs 20 attempts per task with NestDex and AnyTeleop after familiarization. A trial succeeds only if every task stage is completed. Time per successful demonstration includes failed attempts and amortizes the one-time collection of ten inner-skill trajectories. The paper does not include policy-training compute in this collection-time metric.
For autonomous learning, the authors obtain 20 successful complete-task demonstrations whenever the collection method can produce them. Each outer policy is then evaluated for 20 rollouts.
6. Demonstration Collection Results
| Method | Tongs | Bottle | Dual-object | Ingredient + pot | Toast | Binder |
|---|---|---|---|---|---|---|
| NestDex success | 100% | 100% | 100% | 100% | 100% | 100% |
| NestDex time/success (s) | 44.33 | 41.37 | 36.19 | 43.26 | 327.46 | 221.80 |
| AnyTeleop success | 0% | 50% | 30% | 75% | 0% | 0% |
| AnyTeleop time/success (s) | N/A | 88.88 | 121.63 | 55.29 | N/A | N/A |
NestDex produces a successful demonstration in every attempt across all six tasks. On the three tasks where AnyTeleop obtains at least one success, the copilot also reduces time per success. The gap is largest when the task requires continuous regrasping, tool operation, or coordination across two arms and many fingers.
The result supports the proposed division of labor: direct arm teleoperation preserves human task judgment, while the inner policy removes high-dimensional finger coordination from the operator’s moment-to-moment workload.
7. Autonomous Policy and H-VAE Results
| Training data and action target | Tongs | Bottle | Dual-object | Ingredient + pot |
|---|---|---|---|---|
| Copilot demonstrations, direct hand actions | 65% | 60% | 80% | 85% |
| Copilot demonstrations, H-VAE | 100% | 75% | 90% | 100% |
| AnyTeleop demonstrations, direct hand actions | N/A | 40% | 20% | 75% |
Copilot data improves the direct-action policy wherever both sources are available. Tongs Transfer exposes the harder failure mode: AnyTeleop produces no successful demonstration, so an autonomous policy cannot be trained from that source.
H-VAE improves every copilot-trained task by 10 to 35 percentage points. The largest gain occurs on Tongs Transfer, from 65% to 100%. Compressing coordinated hand actions provides a useful inductive bias for a small dataset, while retaining arm commands directly avoids forcing two mechanically different action groups through one latent bottleneck.
8. Why Closed-Loop Execution Matters
The bottle-grasp experiment isolates three execution modes for the same inner policy:
| Execution mode | Success |
|---|---|
| Fixed successful command replay | 3/10 |
| Closed loop, no temporal ensemble | 7/10 |
| Closed loop + temporal ensemble | 9/10 |
Replaying one previously successful trajectory is brittle under small contact changes. Querying the policy from the latest joint-position and effort history raises success from 3/10 to 7/10. Adding temporal ensembling reaches 9/10; the difference between fixed replay and the full method is statistically significant at (p=0.0198).
Temporal ensembling mainly improves smoothness. Closed-loop execution without ensembling has 2.30× higher executed-command P95 jerk than the ensembled version, with (p=1.8\times10^{-4}), while closing duration does not increase. The two mechanisms therefore serve distinct functions: feedback adapts motion to contact, and ensembling suppresses discontinuities between overlapping action chunks.
9. Strengths and Limitations
NestDex presents a coherent path from low-cost hand-skill demonstrations to complete-task data and autonomous deployment. Its strongest design decision is the temporary use of hierarchy: skill structure improves the data-generation process without constraining the deployed policy to a fixed skill graph. The reversible clutch is also more expressive than a binary trigger because it gives the operator direct control over progress and recovery.
The evaluation uses one robot platform and one trained operator, with nominal object resets and 20 attempts or rollouts per condition. The comparison isolates the interface on identical hardware, though broader operator studies are needed to measure learning effort and inter-user variation. Collection time amortizes the ten inner demonstrations but excludes policy-training compute.
The autonomous outer-policy evaluation covers the four single-arm tasks. Toast Preparation and Binder Filing demonstrate successful dual-arm collection and qualitative skill switching; autonomous execution is not reported for them. The skill selector is also evaluated qualitatively, without a selection accuracy or failure analysis.
H-VAE is trained separately for each task, leaving cross-task latent-action reuse open. The four-object grasp study demonstrates adaptation among objects seen during inner-policy training and does not establish generalization to unseen geometry. Finally, all inner skills require predefined demonstrations and textual descriptions; scaling the library raises questions about skill discovery, selector ambiguity, and maintenance.
Takeaway
NestDex reframes shared autonomy as a data-production tool. A human contributes long-horizon task judgment and arm motion; small proprioceptive policies supply contact-rich finger coordination; a clutch exposes reversible control over skill progress; and a vision-language model routes stages to reusable skills. The collected trajectories then supervise a compact, independent outer policy.
The broader lesson is valuable for dexterous learning systems: intermediate autonomy can make demonstrations more reliable even when the final objective is a unified end-to-end controller. NestDex’s 100% collection rate across six tasks, the consistent H-VAE gains, and the 3/10-to-9/10 closed-loop grasp improvement show how interface design, action representation, and online feedback reinforce one another.