[Paper Notes] LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition
Published:
LUCID is a clean split between two scalable sources of supervision. Internet-scale human video gives task intent: what should change in the scene. Massively parallel simulation gives robot control: how a specific embodiment should realize that intent. The phrase that prompted this note, “Next, control: a generalist sensorimotor policy,” refers to the second half of the system, but the full paper is about the interface between the two halves.
My read: LUCID is valuable because it avoids treating human video as hidden robot action. Human video is used to predict short-horizon intent, represented as object flow plus a rough palm pose. A separate embodiment-specific sensorimotor policy, trained in simulation, converts that intent into low-level robot actions. This division lets the same intent model drive a dexterous LEAP hand and a parallel-jaw gripper, while the controllers remain specialized to their robot bodies.
Paper Info
The paper is “LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition” by Harsh Gupta, Guanya Shi, and Wenzhen Yuan from the University of Illinois Urbana-Champaign and Carnegie Mellon University. It is available as arXiv:2606.11628, with project page lucid-robot.github.io.
The Problem
Most robot learning pipelines still need robot demonstrations, teleoperation, or structured human data. These sources are expensive and tied to a particular robot, camera setup, hand, or interface. Unstructured human videos are much easier to scale, yet they lack robot actions. Simulation can generate action-labeled robot data at scale, while task-level rewards are hard to write for open-ended dexterous manipulation.
LUCID pairs these two imperfect sources. Human video supervises an intent model (f_\theta). Simulation trains an embodiment-specific sensorimotor policy (\pi). The shared interface is a short-horizon reference (R), made of object flow and palm pose:
[ f_\theta\left(I_t,{x^{trk}{n,0}}{n=1}^{N},(p^{palm}0,R^{palm}_0)\right) = R = \left{{x^{trk}{n,\tau}}{n=1,\tau=1}^{N,T},{(p^{palm}\tau,R^{palm}\tau)}{\tau=1}^{T}\right}. ]
In words: from recent RGB-D observations, current object query points, and current palm pose, the intent model predicts where object points should move and where the palm should roughly go over the next short horizon.
Intent Model
The intent model adapts CoTracker3 into a forward predictor. Standard trackers estimate where points went in already observed frames. LUCID changes that direction: it predicts future object point motion from the current observation.
The model input is:
- a stack of recent RGB-D frames (I_t);
- (N=16) current query points on the target object;
- the current palm pose.
The output is:
- (T=8) future steps over a 1-second horizon;
- 3D object flow for the query points;
- a palm-pose trajectory.
The architecture uses frozen DINOv3 ViT-B/16 patch tokens for RGB, a residual depth adapter, and a CoTracker-style EfficientUpdateFormer over object tokens plus a palm token. The loss is a sum of mean-squared errors over object flow, palm position, and palm rotation:
[ L(\theta)=L_{trk}+L_{palm,p}+L_{palm,r}. ]
The supervision pipeline is heavily engineered. For each task, the authors mine 20k clips from datasets such as Panda-70M, Action100M, Something-Something-V2, EPIC-Kitchens, and LVP metadata, mostly in-the-wild YouTube footage. For underrepresented tasks, they record about 1 hour of smartphone demonstrations. Each clip is processed with ViPE for camera and depth, SAM 3.1 for object and human masks, DenseTrack3Dv2 for 3D object tracks, and WiLoR plus a rigid fit for MANO palm pose. Human-region augmentation is strong, so the model learns object motion without depending on human hand appearance.
Generalist Sensorimotor Policy
The sensorimotor policy is the “control” part. It is trained in Isaac Lab with goal-conditioned RL to realize any reference (R) in the same format as the intent model output. The policy is embodiment-specific, so the LEAP hand and the parallel-jaw gripper get separate controllers, but both consume the same intent interface.
Training references are procedural, not generated by the intent model. Each simulated episode samples a procedurally generated object from a pool of blob-like, tool-like, and plate-like shapes, then samples a reference trajectory with four segments:
- approach;
- in-hand motion;
- goal;
- disengage.
This is why the policy is generalist. It sees many object shapes, many object-flow trajectories, and many palm-pose references. At deployment, when the learned intent model emits a new (R), the policy treats it like another goal to track.
The action space is structured:
[ a_t = [a^{arm}_t; a^{eig}_t; a^{hnd}_t]. ]
Here (a^{arm}_t\in\mathbb{R}^6) controls arm joint-position deltas, (a^{eig}_t\in\mathbb{R}^5) controls an eigen-grasp basis, and (a^{hnd}_t\in\mathbb{R}^{16}) adds per-joint hand residuals. The eigen-grasp basis is the top-5 PCA basis from retargeted human grasps. It biases exploration toward coordinated grasp modes, while residual hand joints keep fine control available.
The policy is trained in two stages. A privileged teacher (\pi^T) learns with PPO using full simulator state, full-surface object query points, object pose, contact information, and proprioception. A student (\pi^S) is distilled from the teacher with a hybrid PPO plus behavior-cloning loss:
[ L_{student}=L_{PPO}(\pi^S)+\lambda_D \kappa \mathbb{E}\left[|\mu_S(o^S)-\mu_T(o^T)|_2^2\right]. ]
The student sees deployable observations: camera-visible object query points, palm-pose reference, wrist-mounted depth, and proprioception. Domain randomization covers object scale, mass, friction, robot actuator parameters, reset jitter, observation noise, gravity curriculum, wrench perturbations, and success tolerances. The authors also do real-to-sim actuator calibration with CMA-ES against joint step responses.
Closed-Loop Deployment
Deployment runs a slow intent cycle around a fast control loop. The sensorimotor policy runs at 50 Hz. DenseTrack3Dv2 updates sliding-window point tracking at 30 Hz. SAM 3.1 refreshes masks at 1 Hz, and the intent model also runs at 1 Hz. The full stack runs on a single RTX 5090 GPU.
At each intent cycle, the system refreshes the object mask, samples and back-projects query points, estimates current palm pose from forward kinematics, and asks (f_\theta) for a fresh short-horizon reference (R). Between intent cycles, the point tracker keeps current points aligned with the live scene, and the student policy consumes the advancing lookahead window from (R).
This closed-loop property is the real difference from video-plan methods. If a grasp misses or the object shifts, LUCID can re-query the live scene and produce a new reference. The control policy then follows the updated reference with the same sensorimotor machinery.
Experiments
The paper evaluates five real-world tasks:
- Stirring: pick up a spoon and complete three stirring circles.
- Wiping: pick up a cloth and clear marks or substances from a surface.
- Binning: pick up each object and place it into a target container.
- Push-T: push a T-shaped block to a target pose.
- Cable routing: thread a cable through fixtures.
The first three tasks are supervised only by web-scraped video, with 20k clips per task. Push-T and cable routing use 1 hour of self-collected smartphone video each.
On the web-supervised tasks, closed-loop LUCID reaches 73% average success, compared with 28% for an open-loop video-generation baseline that uses Veo 3.1 to produce a single video plan from the initial scene. The difference is mostly about recovery. Open-loop plans become stale when execution diverges. LUCID can keep asking the intent model what should happen next.
The per-task breakdown is also useful:
- Stirring: LUCID succeeds 19/30, open-loop succeeds 7/30.
- Wiping: LUCID succeeds 26/30, open-loop succeeds 13/30.
- Binning: LUCID succeeds 21/30, open-loop succeeds 5/30.
For embodiment transfer, the same intent model is used with two controllers: the LEAP hand policy and a parallel-jaw gripper policy. On Push-T and cable routing, both embodiments reach 19/30 aggregate success. Cable routing favors the gripper because its two opposing jaws are well-suited to the cable; the hand has a harder time with precise cable grasping. The key result is that the intent model transfers while the controller changes.
Scaling and Ablations
The binning data-scaling sweep uses 1k, 2k, 5k, 10k, and 20k clips. Held-out intent loss decreases steadily, and real-world binning success rises with more clips. At 1k-2k clips, the model often reaches the object but fails to localize the bin. Around 5k-10k clips, container localization appears, but release alignment remains weak. This gives a practical signal: internet video quantity helps, but task-level precision still needs enough coverage.
The sensorimotor ablations explain why the control recipe matters:
- replacing the point-token policy encoder with an MLP loses per-point detail needed for stable grasping and in-hand manipulation;
- removing the eigen-grasp basis expands the exploration space and slows learning;
- DAgger-BC distillation is competitive but cannot fully exploit the student’s own sensing modalities;
- removing the wrist camera leaves close-range object geometry unresolved.
The query-point ablation shows diminishing returns after more visible object points. The paper uses (N=16) because it captures most of the reward gain while keeping intent-model training efficient.
Strengths and Limitations
LUCID’s strongest idea is the explicit intent-control boundary. Human video is used where it is strongest: revealing what humans are trying to make happen to objects. Simulation is used where it is strongest: producing robot actions, contacts, recovery behavior, and embodiment-specific motor control at scale. The 3D object-flow plus palm-pose interface is simple enough to extract from video and concrete enough for RL to track.
The limitations are also clear. The pipeline has many modules: SAM 3.1, DenseTrack3Dv2, ViPE, WiLoR, the intent model, the point tracker, stereo depth, and the policy. Each can fail, especially under occlusion, textureless objects, or bad hand-object segmentation. Tasks without a clear ending condition can loop, as in stirring unless an external stop is imposed. The explicit interface is lossy: object flow and palm pose omit finger configuration, contact timing, force, and fine manipulation cues. A future latent interface jointly optimized across intent and control could preserve more of the signal.
Takeaway
LUCID is best read as a data-scaling recipe for dexterous manipulation without robot demonstrations. It says: use unstructured human video to learn embodiment-agnostic intent; use simulation to learn embodiment-specific control; keep the interface short-horizon and closed-loop.
For my taxonomy, I would label it Human Video Intent Learning / Object Flow + Palm Pose / Generalist Sensorimotor Policy / Sim-to-Real Dexterous Control. The most reusable idea is the separation itself. A robot does not need human video to provide robot actions. It needs human video to say what should change next, and it needs a strong controller to make that change real on its own body.