[Paper Notes] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
Published:
Do as I Do is a complete video-to-robot-data pipeline for dexterous manipulation. It starts from ordinary monocular RGB videos of people manipulating objects, reconstructs 4D hand-object interaction, and retargets the recovered motion into executable trajectories for multi-fingered robot hands.
My read: the paper is strongest as a systems bridge between two worlds that usually stay separate. Vision foundation models recover a usable hand-object reference from messy human video. A dynamics-aware optimizer then turns that noisy reference into robot-complete manipulation data. The important shift is from treating human video as a vague prior to treating it as a source of physically replayable trajectories.
Paper Info
The paper is “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos” by Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad “Mahi” Shafiullah, and Jitendra Malik from UC Berkeley. It is available as arXiv:2606.19333, with project page do-as-i-do.com and code at malik-group/do-as-i-do.
The Problem
Dexterous robot learning needs data, but the most scalable data is observational. Humans leave behind enormous amounts of manipulation video; robots need action-labeled, embodiment-specific trajectories. Teleoperation is slow and hardware-bound. Simulation can scale rollouts, but open-ended dexterous tasks are hard to specify with rewards and environments.
The paper frames this as the old “Do as I do” problem under modern constraints: reconstruct what the human did, then retarget it to a different body. The hard setting is everyday monocular RGB video. There is no depth, no motion capture, no known object mesh, no clean hand-object contact signal, and no robot action.
Method Overview
The pipeline has two stages.
First, reconstruction estimates the human hand, object shape, object pose, and camera-aligned hand-object trajectory from RGB video. The hand side uses HaWoR. The object side uses SAM 3D for single-frame mesh generation and a guided-diffusion tracker that keeps object shape fixed while updating pose over time.
Second, retargeting takes the reconstructed hand-object trajectory and solves for a robot trajectory in simulation. The target embodiment in the paper is a 22-DoF Sharpa Wave hand, deployed with UR3e arms for real-world bimanual rollouts. The retargeting step is dynamics-aware: it optimizes in physics simulation, so the result must track the reference while respecting contact, gravity, object motion, and hand-object interaction.
This two-stage split matters because the reconstruction is inevitably noisy. The retargeter is designed around that reality and handles references far messier than clean MoCap.
Object Tracking via Guided Diffusion
The most interesting reconstruction component is the SAM 3D-based object tracker. Applying SAM 3D independently to every frame gives inconsistent meshes and poses. Do as I Do instead anchors the object shape at one frame and uses flow-matching inference to sample pose updates conditioned on the current image and previous pose.
The paper writes the guided update as:
[ x^s_t=(1-\alpha_s)(x^s_{t-\Delta}+\Delta v^s_\theta)+\alpha_s z^s_{\mathrm{ref}}(t), \qquad x^p_t=(1-\alpha_p)(x^p_{t-\Delta}+\Delta v^p_\theta)+\alpha_p z^p_{\mathrm{ref}}(t). ]
Here (x^s) is the shape block, (x^p) is the pose block, and the reference interpolants pull the diffusion sample toward the fixed canonical shape and previous-frame pose. Shape guidance can stay high because objects are rigid. Pose guidance is adaptive: the method estimates object rotational velocity from 2D point tracks, then lowers or raises pose guidance depending on how much motion the object appears to have.
Per frame, the tracker samples multiple candidate poses and clusters them under a weighted SE(3) distance. Candidate selection by clustering performs about as well as log-likelihood scoring while avoiding the expensive trace computation needed for exact flow-model likelihood.
After tracking, the system aligns hand and object scales. HaWoR hand reconstruction and SAM 3D object reconstruction live in different scale conventions, so the method uses MoGe pointmaps and hand/object centroids to slide the object along its camera ray until the visible object position is consistent with the hand.
Dynamics-Aware Retargeting
The retargeting stage begins with a kinematic hand retargeting reference, then runs MPPI-style sampling-based optimization in MuJoCo Warp. It plans every 0.5 seconds over a 3-second horizon, evaluates 1024 samples per planning step, and runs 32 optimization iterations. Rewards track object position/orientation, hand position/orientation, and finger joints, with penalties for excessive penetration.
The paper adds three pieces that make the optimizer robust to reconstructed references.
Warmup steps. A noisy first frame can put the robot hand and object in an unrecoverable state. The method prepends a warmup horizon where the object is temporarily held in place while the robot hand moves into a better configuration. Then the weld is released and normal simulation begins.
Random force perturbation. Some rollouts look good briefly but are dynamically fragile, such as balancing an object on fingertips. Random forces during sampled rollouts push the optimizer toward grasps that survive small disturbances.
Transition reward. Rest-to-in-hand and in-hand-to-rest transitions are discrete interaction events. Tracking loss alone is too soft when references are noisy, so the method adds a penalty when the object should be resting but lacks floor contact, or should be in-hand but lacks hand-object contact.
These additions are practical. They do not require hand-written grasp samplers or task-specific object heuristics; they use the same simulation optimizer and make it less brittle.
Results
On reconstruction, Do as I Do improves over existing hand-object reconstruction and object-tracking baselines. On DexYCB it reports 0.71 F-5, 0.93 F-10, and 0.66 Chamfer distance. On HOI4D it reports 0.72 F-5, 0.91 F-10, and 0.49 Chamfer distance. On a 150-video in-the-wild benchmark, human raters prefer its object tracking over FoundationPose 67% of the time, with 79% win rate among non-tie judgments.
On retargeting, the gap is large. On reconstructed in-the-wild references, the annealed-sampling baseline succeeds 25% of the time. Adding warmup raises success to 66%; adding perturbation reaches 67%; adding transition reward reaches 71%. On clean OakInk2 MoCap references, the same sequence improves from 72% to 81%. This is a good sign: the new components help noisy internet-video references, yet they also improve cleaner human-object trajectories.
For real-world deployment, the pipeline produces 500 high-quality, human-verified dexterous manipulation trajectories from internet videos, egocentric videos, and generated videos. The paper demonstrates real robot rollouts on 10 tasks including whisking, pouring, dusting, squeezing, tamping, erasing, stirring, hammering, spreading, and picking.
Data Filtering Playbook
One especially useful section is the data-quality analysis. From 2,000 ten-second 100DOH clips, only 187 contain meaningful hand-object interaction. After removing boundary failures, shot-boundary issues, camera-motion failures, SAM 3D failures, and other problems, only 83 clips remain suitable for reconstruction, about 4% of the sampled data.
The practical message is blunt: internet video is huge, but usable robot-learning video is sparse after quality filtering. Scaling from human video needs serious preprocessing, not just a bigger crawler. The paper estimates roughly a 20x penalty if one skips careful filtering.
Strengths and Limitations
The strength of Do as I Do is end-to-end completeness. It goes from an everyday RGB video to robot rollouts with dexterous hands, and the key modules are chosen for open-world coverage: SAM 3D for objects, HaWoR for hands, MoGe for metric geometry, and sampling-based simulation optimization for retargeting.
The limitation is also clear. The method assumes rigid objects and reasonably accurate monocular depth. Monocular video has contact ambiguity: visual occlusion can look like physical touch. The reconstruction covers the hand and one object, so it misses scene constraints such as tables, obstacles, articulated supports, and container geometry. The final rollouts also inherit simulator mismatch, which bounds real-world reliability.
Takeaway
Do as I Do is best read as a recipe for making human videos operational. The paper does not claim that human video directly gives robot actions. It builds a layered compiler: RGB video to 4D hand-object reference, reference to physically plausible robot trajectory, trajectory to real-world rollout.
For dexterous manipulation research, the key lesson is that video-scale data becomes useful only after representation and retargeting are designed together. Better object tracking alone is insufficient; better simulation optimization alone is also insufficient. The value comes from making the reconstruction noisy in the right format for a robust dynamics-aware retargeter.