[Paper Notes] GALATEA: Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers
Published:
TL;DR
A generated video can show a hand lifting a jar by its neck, yet leave the robot without the contacts and motor commands needed to reproduce it. GALATEA reconstructs the hand and object together, then uses simulation to learn a controller that follows their motion through physical interaction. The video specifies both the grasp style and the intended object motion; reinforcement learning supplies the executable actions.
The authors obtain about 2,000 usable references from 2,500 generated clips and ground more than 1,500 trajectories in simulation. Category experts reach 78.6% mean success across 42 training objects. After distillation, one controller reaches 66.6% on unseen trajectories and 54.2% on novel objects in simulation. Real-world evaluation on 40 unseen plans yields 27/40 successes (67.5%). These are separate evaluation settings, with trajectory tracking as the success criterion.
My main takeaway is that reference quality and the reward design have to be solved together. Keeping fingers close to a reconstructed pose is useful only while it helps produce the intended interaction.
Paper information
Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers, by Tianyue Wu, Boyuan An, Shuqi Zhao, Heyu Guo, Wanli Xing, Yi Ma, Kaifeng Zhang, Ruihai Wu, and Masayoshi Tomizuka. Affiliations: UC Berkeley, Sharpa Robotics, and the University of Hong Kong. These notes use arXiv:2609.10050v1, September 9, 2026, a preprint.
Sources: paper · arXiv record · project page · code repository. As checked on September 11, the project page labels code availability as “by Nov.”; the repository link alone should not be read as a completed implementation release.
From a video plan to a feedback controller
GALATEA uses Seedance 2.0 to generate human manipulation videos conditioned on a real first-frame image and a language instruction. The authors found that conditioning on simulation renders often produced physically inconsistent interactions. A stereo depth capture of the real first frame supplies metric scale for reconstruction.
The tasks cover grasp-and-move, push-and-pull, and pose-adjust. Tracking the hand as well as the object preserves choices such as grasping a jar around its neck to leave the opening accessible. An object trajectory alone does not specify that choice.
flowchart TD
A["Real first-frame image + instruction"] --> B["Seedance 2.0 video"]
B --> C["Metric hand-object reconstruction"]
D["First-frame stereo depth + known object mesh"] --> C
C --> E["Contact-preserving reference augmentation"]
E --> F["Category experts: tracking RL in simulation"]
F --> G["Behavior cloning + DAgger"]
G --> H["Unified arm-hand controller"]
I["New video plan, reconstructed as an HOI reference"] --> H
J["Measured robot state + online object pose"] --> H
H --> K["30 Hz position control"]
At deployment, the low-level policy follows a reconstructed reference using robot state and online object-pose feedback. The demonstrated closed loop is at this tracking layer; the paper does not establish continuous video regeneration during execution. The system uses a Franka Research 3 arm and a 22-DoF Sharpa Wave Hand, with PhysX in Isaac Gym for training. Method, §III
Recovering contact geometry from generated frames
MoGe-2 predicts depth, SAM2 identifies the object and moving foreground, FoundationPose estimates object poses from a known metric mesh, and WiLoR estimates MANO hand geometry. The estimates initially come from separate models, so their relative placement can be inconsistent.
First, depth is aligned frame by frame. With a fixed camera and static background, pixels outside the moving foreground in both the first and current frames should observe the same scene. Trimmed least squares fits a scale and offset against the first-frame stereo depth:
\[D_t(p)=\left[s_t\widetilde D_t(p)+b_t\right]_+.\]This corrects metric scale and depth offset before object tracking. RANSAC on the initial depth also estimates the support plane.
Joint optimization then applies rigid corrections to the object pose and the hand mesh while retaining the estimated finger articulation. Its objective combines projection consistency, observed silhouettes, detected contacts, and temporal smoothness:
\[\mathcal L= \lambda_{FP}\mathcal L_{FP} +\lambda_{kp}\mathcal L_{kp} +\lambda^O_{sil}\mathcal L^O_{sil} +\lambda^H_{sil}\mathcal L^H_{sil} +\lambda_{con}\mathcal L_{con} +\lambda_{temp}\mathcal L_{temp}.\]The FoundationPose and keypoint terms preserve the initial object projection and WiLoR hand-keypoint reprojection. Silhouette matching constrains position, orientation, and depth while excluding occluded pixels. HOI-DETR identifies contact frames, where a point-to-mesh loss draws contact-labeled hand vertices toward the object surface. The temporal term penalizes first and second differences of object translation and hand centroid. These are geometric constraints; physical execution is learned in the next stage.
The ablation makes the value of joint refinement concrete. On 40 HO-Cap clips, removing it increases wrist-to-object relative-position error from 54.87 to 102.08 mm and contact deviation from 41.36 to 112.43 mm. Object ADD-S changes much less, from 3.88 to 4.02 cm. Accurate object placement alone can therefore coexist with hand placement that is unsuitable for tracking. The full method does not win every metric: the depth-alignment ablation has slightly lower contact deviation, and DO AS I DO has lower wrist-to-object relative-position error. Equations 1–2 and Table I
Augmentation that keeps the grasp intact
Each source trajectory receives five sampled variations in approach and post-contact motion. Before first contact, only the hand is perturbed, with the perturbation decaying to zero at contact. After contact, a gradually increasing transform is shared by hand and object. A trajectory-level yaw transform adds another source of variation.
Writing the common post-contact transform as $A_t$, the preserved relative pose follows directly:
\[\widetilde T_t^H=A_tT_t^H,\qquad \widetilde T_t^O=A_tT_t^O,\] \[(\widetilde T_t^O)^{-1}\widetilde T_t^H =(T_t^O)^{-1}T_t^H.\]This algebra explains the augmentation’s purpose: vary where the interaction goes while keeping the grasp geometry consistent. The paper scales perturbations to 30% of source clearance and motion range, screens candidates for execution, solves arm inverse kinematics, and slows trajectories to respect joint-velocity limits. Reference preprocessing, §III-B.1
Reward the interaction, including a failed lift
The actor receives joint positions, the previous action, wrist and fingertip state, a noisy object pose, hand-object tracking errors, five reference fingertip-to-surface distances, and a BPS object-shape encoding. It has no joint-velocity, force, mass, or center-of-mass input. The asymmetric critic additionally receives privileged simulator state.
The MLP outputs 29 action dimensions: seven arm joint deltas and 22 absolute hand-joint targets. Arm deltas are scaled by 0.03 rad; arm and hand targets are smoothed with EMA coefficients 0.20 and 0.10 before 30 Hz position PD control.
The reward contains hand and object tracking, fingertip proximity, multi-finger contact, lifting, low mechanical power, and regularization:
\[r_t=r_t^H+r_t^O+r_t^{near}+r_t^{multi} +r_t^{lift}+r_t^{eff}-r_t^{reg}.\]Two gates determine when tracking earns reward. Let $c_t^{ref}$ and $c_t$ count reference and simulated fingertip contacts. Object tracking is activated when either has contact:
\[g_t^C=\mathbf 1[c_t^{ref}>0\;\lor\;c_t>0].\]Hand tracking is multiplied by a lift gate. With $\widehat h_t$, $h_t$, and $h_0$ denoting reference, actual, and initially placed object heights,
\[g_t^L=1-\mathbf 1[ \widehat h_t-h_0>0.05\;\land\;h_t-h_0\leq0.05].\]If the reference has lifted the object by more than 5 cm but the rollout has not, dense hand tracking is suppressed. A separate lift reward pays when both exceed that height. This addresses a concrete failure: a hand may follow the demonstrated upward motion while leaving the object on the table. Finger-keypoint tracking weight also drops from 2 in free space to 0.75 during contact, allowing more adjustment during interaction.
SAPG partitions parallel rollouts among PPO agents with different exploration settings and aggregates their experience into a shared update. The recipe also randomizes PD gains, object mass, friction, actuation latency, and actor object-pose observations. At initialization, the arm and object are aligned to a sampled reference frame, but all hand joints start open. Directly initializing retargeted fingers can embed them in the object or table and cause large depenetration impulses.
The alternative of first learning hand imitation and then a residual policy sometimes learns faster early on, yet lowers final success on all ten benchmark objects. The authors hypothesize that noisy reconstructed hand trajectories constrain later exploration. That result is relevant when choosing a training recipe for video-derived references; it does not invalidate staged imitation with cleaner motion-capture data. Reward and training recipe, §III-B.2; ablations, §IV-B
What the success rates measure
Training one policy directly on the full reference set remains difficult under the available compute. GALATEA trains at most two experts per object category, each covering about 40 source trajectories, or roughly 200 after augmentation. Behavior cloning merges their behavior into one controller; DAgger then reduces distribution shift.
| Evaluation setting | Mean success | Median success |
|---|---|---|
| Experts, 42 training objects | 78.6% | 80.5% |
| Experts, ten benchmark objects | 77.4% | 81.0% |
| Distilled policy, training objects | 74.4%* | 74.5%* |
| Distilled policy, 300 unseen trajectories | 66.6% | 68.8% |
| Distilled policy, five novel objects | 54.2% | 52.2% |
The distilled training-object values are calculated from the reported drops of 4.2 and 6.0 percentage points. Simulation success uses 20,000 evaluation rollouts per method–object experiment, beginning at the first reference frame. A rollout must finish the reference without exceeding 4 cm object-position error, 30° object-rotation error, or the specified hand-keypoint thresholds, which range from 6 to 12 cm by group. The means are macro averages across objects. Figure 5 and §IV-B
The reported 10.2 mm position, 17.4° rotation, and 36.2 mm hand-keypoint errors are averaged over segments that execute the intended interaction, with short segments discarded. They are not whole-rollout error guarantees. Selection matters: DO AS I DO’s 7.1° rotation error excludes many unsuccessful pose-adjust plans and uses a floating wrist, limiting direct comparison.
For real-world evaluation, a D455 supplies online object poses through FoundationPose. A separate D435 captures the video-conditioning image and is removed before execution. The authors also calibrate a shared translation bias in object-pose estimates using a rigidly grasped object and known end-effector pose.
| Unseen real-world plan | Success |
|---|---|
| Jar-neck pose adjustment | 6/10 |
| Jar top-down grasp-and-move | 6/10 |
| Mug-rim pushing | 7/10 |
| Mug-handle pulling | 8/10 |
| Overall | 27/40 |
Real-world success follows the same tracking thresholds, using estimated object poses and hand keypoints from measured joints. These four tasks demonstrate transfer to unseen plans with a small trial count. They provide limited evidence about broader manipulation coverage. Table III and §IV-C
Where I would use this approach
I would consider GALATEA when the desired grasp style matters, the object mesh is available, and a calibrated camera can track object pose. Its explicit hand-object reference provides a useful interface between a video planner and an embodiment-specific controller. For fine in-hand manipulation, I would want additional contact-rich data: the authors report that generated references and reconstruction do not reliably capture the subtle finger-object motion involved.
Several costs remain upstream of control. The generator is proprietary; reconstruction assumes a controlled view and static background; 83% of generated clips pass reconstruction quality checks, so filtering remains necessary. The paper’s comparison with the much lower yield of an in-the-wild video audit uses different criteria and cannot establish a controlled data-efficiency ratio. Reconstruction also takes 12.33 minutes per H2O clip and 6.73 minutes per HO-Cap clip on an RTX 5880 in Table I, so the current pipeline does not demonstrate immediate response to a new video request.
Flat objects near the table and human grasps that depend heavily on palm friction remain difficult. During deployment, contact transitions can push the robot away from its reference, and recovery often fails once that deviation grows. My next test would deliberately introduce a small slip or object displacement after contact and measure recovery. The paper’s trajectory-completion results leave that capability unresolved. Discussion, §V