[Paper Notes] Pre-training Visual Dexterity in Simulation
Published:
Simulation Pre-training for Dexterity (SPD) asks whether simulation can become a scalable data source for pre-training real-world dexterous policies. Five operators use VR to teleoperate a simulated bimanual robot, producing 75 hours of action-labeled demonstrations across six scenes in one week. A 222M-parameter diffusion transformer learns from this dataset, then adapts to each physical task with only 1–2 hours of real demonstrations.
The strongest result comes from combining pre-training with the right temporal design. A policy using a 32-step attention window and short 8-step action chunks reaches 76.7% average task progress across five real-world tasks, compared with 58.9% for the identical architecture trained from scratch. History supplies temporal coherence; short chunks preserve reactivity. This pairing gains about 17.8 percentage points from simulation pre-training, while the other context/chunk variants gain at most about three points.
Paper Info
“Pre-training Visual Dexterity in Simulation” is by Sarthak Kamat, Adam Rashid, Satvik Sharma, Aseem Doriwala, Chelsea Finn, Phillip Isola, and C. Karen Liu, with affiliations at Stanford University, MIT, and Scale AI. The 2026 paper, project page, and PDF introduce the framework together with plans to release the spd-75h dataset, spd-vr collection software, and six simulation scenes.
Why Simulation Data?
Large robot-policy datasets have mostly grown around parallel-jaw grippers. Multi-fingered hands present a harsher collection problem: the hardware is expensive and fragile, resets are slow, and teleoperation must control many coupled degrees of freedom. Human video offers scale, yet contact occlusion makes hand-pose recovery noisy, and the recovered motion still needs to cross a substantial embodiment gap before it becomes a robot action.
SPD collects supervision directly on the target robot embodiment inside MuJoCo. The operator sees a virtual scene in a Meta Quest 3 headset; tracked wrists and fingertips drive the simulated arms and hands through inverse kinematics. Contacts remain physically simulated, every action is labeled in robot coordinates, and collection can run without a physical robot. Virtual resets, parallel operators, and decentralized collection turn simulation into a human-demonstration engine instead of using it solely as an RL environment.
The simulation and physical systems are deliberately aligned. Both use the same pair of 22-DoF Sharpa Wave hands, 6-DoF arms, top and wrist camera viewpoints, related objects, and similar retargeting. This design keeps the action interface and embodiment stable while fine-tuning absorbs the remaining appearance and dynamics gap.
The SPD Training Pipeline
flowchart TD
H["Five VR operators"] --> S["spd-vr in MuJoCo<br/>on-embodiment teleoperation"]
S --> D["spd-75h<br/>1,930 episodes · 75 hours · 6 scenes"]
D --> A["Rendering and augmentation<br/>textures · colors · left/right symmetry"]
A --> P["222M causal diffusion transformer<br/>flow-matching pre-training"]
R["44–121 min of real demonstrations<br/>per downstream task"] --> F["Full-policy fine-tuning"]
P --> F
F --> T["Five real bimanual dexterous tasks"]
1. Collect long-horizon behavior in VR
The dataset covers Jenga bricks, spelling blocks, mugs, dishes, cups, and bottles. Reset functions randomize assets, object poses, and physical properties. Prompts specify outcomes such as building a tower, spelling a word, hanging mugs, racking dishes, stacking cups, or tossing bottles, while leaving strategy and subtask order open. This allows different operators—and even the same operator—to generate multiple valid solutions.
The complete dataset contains 1,930 episodes and approximately 75 hours. The appendix’s task-level table covers 1,916 episodes and 4,516 minutes because tasks with fewer than ten episodes are omitted. Extended spans with more than ten seconds of no hand–object contact are removed. Trajectories are then rendered at (224\times168) with instance masks, which support random object colors, table and background textures, and a left–right symmetry augmentation that swaps arms while reflecting images, proprioception, and actions.
This data retains a useful form of diversity: it contains different objects, initial states, strategies, contacts, and long-horizon transitions. Its scope is still concentrated in six curated scenes, a limitation that matters when interpreting generalization.
2. Pre-train a history-conditioned diffusion policy
The policy receives multi-view images, 56-D proprioception, previous 56-D actions, and noised future action chunks. It does not use language because the dataset has no dense language annotation. Each training sequence spans 256 timesteps at 30 Hz, about 8.5 seconds.
Images from three cameras pass through a frozen DINOv3 ViT-B/16. Four learned queries per camera pool patch features into compact visual tokens. Camera-specific cross-attention revisits the original patch bank every two transformer blocks, keeping the pooling dependent on the current sensorimotor context. Images are sampled every eight timesteps to reduce redundancy.
The shared trunk has eight transformer blocks, hidden size 768, and 12 attention heads. Each layer uses causal sliding-window attention over 32 timesteps. The action-denoising expert contributes its own 58M parameters, following the expert separation used in π₀-style flow policies.
3. Learn actions with flow matching
For expert action chunk (x_1) and Gaussian noise (x_0\sim\mathcal{N}(0,I)), training samples (t\sim\mathcal{U}[0,1]) and constructs
[ x_t=(1-t)x_0+t x_1. ]
The network predicts the constant transport velocity
[ v=x_1-x_0. ]
Every noised action token also receives embeddings for flow time and its position inside the action chunk. Training denoises all chunks in a 256-step sequence simultaneously under a causal mask. This prefix-parallel formulation shares the cost of processing history across 32 supervised chunk predictions. During deployment, a rolling KV cache retains the 32-step context; ten Euler steps integrate the flow ODE and produce the next eight actions.
This architecture connects two timescales. Long sensorimotor context helps disambiguate contact and occlusion, while an 8-step chunk at 30 Hz lets the robot revise its plan about four times per second.
4. Fine-tune on the physical robot
The physical platform has two upgraded YAM Pro arms, two 22-DoF Sharpa Wave hands, and three RealSense D405 cameras. Real teleoperation uses Quest controllers for wrists and calibrated Manus gloves for fingertips. The full pre-trained policy is fine-tuned separately for each task.
The real datasets remain small: 44–121 minutes and 161–270 episodes per task. They teach the policy the true visual appearance, contacts, actuator behavior, and task-specific details after simulation has supplied broader manipulation experience.
Experiments: What Actually Improves?
The evaluation covers five bimanual tasks: plate racking, mug hanging after a handover, removing and restacking a Jenga block, unstacking cups into a pyramid, and tossing four bottles into a bin. Each checkpoint receives 20 physical trials per task from randomized object placements.
The reported metric is normalized task progress, computed from a task-specific stage rubric. For example, Jenga awards separate points for pushing a block out, pulling it free without collapse, and placing it on top. The metric reveals partial completion and should not be read as binary end-to-end success.
| Training | Plates | Mugs | Jenga | Cups | Bottles | Mean |
|---|---|---|---|---|---|---|
| SPD pre-trained, (w=32,c=8) | 80.6 | 93.3 | 85.0 | 55.6 | 68.8 | 76.7 |
| BC from scratch, (w=32,c=8) | 66.9 | 80.0 | 65.0 | 35.0 | 47.5 | 58.9 |
| Gain | +13.7 | +13.3 | +20.0 | +20.6 | +21.3 | +17.8 |
SPD improves progress on every task for the selected architecture. Its training loss also begins lower and converges lower than the scratch policy; the authors use this as supporting evidence because prior behavior-cloning work found training loss more predictive of robot performance than validation loss.
The Most Important Ablation: Context × Chunk Length
The authors sweep history window (w\in{1,32}) and action chunk length (c\in{8,32}), training every configuration both from SPD and from scratch.
- With one observation frame, reducing the chunk from 32 steps to 8 makes the policy shaky and collapses performance. A long open-loop chunk supplies temporal smoothness when history is absent.
- With a 32-step history window, the 8-step chunk becomes the best configuration in both training regimes. Context supplies coherence, so the controller can re-plan frequently.
- The (w=32,c=8) configuration captures nearly all of the pre-training benefit. Its average progress rises by about 18 points; the other three configurations gain roughly three points or less.
This interaction is more informative than a generic conclusion that more context helps. Pre-training has stored reusable sensorimotor patterns, but the downstream controller needs enough history to recognize those patterns and a short enough horizon to correct contact-rich actions before errors compound.
Strengths
SPD turns simulation into an intermediate data regime between real-robot teleoperation and human video. It preserves robot action labels and embodiment alignment, removes robot wear and reset time from pre-training, and keeps human strategic diversity. The one-week collection of 75 hours by five operators gives concrete evidence for collection throughput.
The paper also controls its core comparison well: the SPD and scratch policies share architecture, task data, training time, and evaluation protocol. The context/chunk sweep exposes an architectural condition under which the pre-training signal becomes useful. Finally, the paper provides enough system detail to make the collection stack reproducible, including simulation frequency, rendering, retargeting, augmentation, tokenization, flow objective, optimizer, and real-robot control rates.
Limitations and Open Questions
The simulated scenes require tuned masses, friction, and contact responses. Operators may learn simulator-specific strategies when those properties diverge from reality. Pre-training covers six scenes, and downstream objects are similar to their simulated counterparts, so the experiments establish transfer within a related task family more clearly than broad out-of-distribution generalization.
The real evaluation uses 20 trials per checkpoint and a stage-progress metric. This is appropriate for long tasks, though it leaves complete-task reliability and confidence intervals for individual stages less explicit. The paper compares pre-training against per-task behavior cloning from scratch; it does not yet isolate how performance scales with simulation hours, scene count, operator count, or the amount of real fine-tuning data.
SPD also keeps the embodiment fixed across simulation and reality. That design cleanly tests sim-to-real pre-training, while cross-robot transfer remains open. Future mixtures could combine simulated on-embodiment actions, real teleoperation, egocentric human video, and RL-generated simulation experience, using each source for the kind of diversity it provides best.
Takeaways
SPD’s main contribution is a practical answer to the dexterous data bottleneck: collect human intent and contact-rich behavior in simulation, retain actions in the robot’s native embodiment, and use a small real dataset to ground the learned prior in physical reality.
The architecture result is equally valuable. Reactive dexterity benefits from short action chunks, but short chunks become unstable without temporal context. A causal history-conditioned policy resolves this tension and unlocks the benefit of pre-training. For future dexterous foundation models, the data source and the temporal interface must be designed together.