[Paper Notes] GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
GRAIL is NVIDIA’s fully digital data-generation pipeline for humanoid loco-manipulation. It starts from 3D assets and simulator-ready scenes, uses video foundation models as interaction priors, reconstructs metric 4D human-object interaction trajectories, retargets them to a Unitree G1, and trains task-general policies for object pick-up, whole-body manipulation, sitting, and terrain traversal. The main claim is practical: humanoid robots can get useful loco-manipulation supervision before any physical robot teleoperation or physical scene rebuild.
My read: the paper is about moving the data bottleneck upstream. Instead of collecting robot demonstrations for every object and terrain, GRAIL builds known 3D configurations first, asks a video model to propose plausible human-object interactions inside that configuration, then recovers and retargets those interactions into robot-compatible references. The important distinction is that GRAIL does not try to reconstruct arbitrary internet videos. It gives the reconstruction system known geometry, metric scale, camera parameters, object assets, environment depth, and a character already proportioned for the target humanoid.
Paper and Resources
The paper is “GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors” by Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, and Ye Yuan from NVIDIA and UCLA. It is available as arXiv:2606.05160, with project materials at research.nvidia.com/labs/dair/grail, code at NVlabs/GRAIL, and a released Hugging Face dataset at nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL.
The generated data covers more than 20,000 sequences across object pick-up, whole-body manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, the authors train egocentric visual policies and deploy them on a real Unitree G1, reporting 84% real-world pick-up success and 90% stair-climbing success.
The Problem: Humanoid Data Does Not Scale Cleanly
Humanoid loco-manipulation couples locomotion, balance, object contact, hand actions, and scene geometry. A pick-up trajectory is more than an arm reaching motion: the robot must approach, place feet, bend or squat, establish contact, grasp, lift, and recover balance. Terrain traversal has a similar whole-body structure, where stairs, slopes, curbs, and chairs impose geometry-specific constraints.
Teleoperation and motion capture provide high-quality demonstrations, but their scaling properties are poor. Each new object, scene, or terrain layout may need physical setup, human operation, instrumentation, and robot runtime. In-the-wild videos are abundant, but converting them into robot-ready trajectories is underconstrained: camera, metric scale, object geometry, human morphology, contact, and world-space motion all need to be inferred after the fact.
GRAIL changes the order of operations. It first specifies the 3D world, then asks a video model for behavior inside that world. This gives the system privileged information during reconstruction and turns several hard inference problems into known inputs.
Pipeline Overview
Given a 3D object asset, GRAIL produces three forms of supervision:
- humanoid kinematic motion;
- object kinematic motion;
- robot actions generated by task-general tracking policies.
The pipeline has four stages:
- Robot-centric human video generation: build a known 3D scene, render the first frame, and use a video foundation model to synthesize a human-object interaction video.
- Interaction-aware 4D HOI reconstruction: estimate human motion and object pose, then jointly optimize them using keypoints, projection, depth, contact, and temporal regularization.
- Task-general loco-manipulation tracking: retarget the reconstructed motion to Unitree G1 and train policies that can track whole families of generated references.
- Sim-to-real visual policy training: distill the tracking policies into egocentric RGB policies and deploy on real hardware.
The architecture is built around a specific tradeoff: use generative video models for behavioral diversity, but keep metric geometry and robot compatibility anchored in simulation assets.
Stage 1: Asset-Conditioned Video Generation
GRAIL does not directly generate robot videos. The authors argue that current video foundation models have stronger priors over human motion and manipulation than over robot bodies, and the human motion reconstruction stack is more mature. The pipeline therefore uses a human character asset prefitted to the Unitree G1 morphology, making later retargeting less brittle.
The system constructs candidate scene configurations with Infinigen: an indoor floor-only environment and a furnished room with a table. A VLM decides whether an object should be placed on the floor or table according to affordance. The object is settled into a stable initial configuration, and Blender renders the initial frame with known camera intrinsics and extrinsics. A VLM writes the interaction prompt, and a video foundation model such as Kling generates a static-camera interaction video.
This “known before generation” setup matters. The camera parameters, metric scale, object geometry, object texture, environment depth, and character morphology are all available when reconstruction starts. Compared with unconstrained video mining, GRAIL avoids estimating the entire world from ambiguous pixels.
Stage 2: Interaction-Aware 4D HOI Reconstruction
The reconstruction stack first estimates human and object motion independently. GENMO provides per-frame SMPL-X body pose parameters from the generated video, with body shape fixed to the prefitted character. WiLoR refines hand poses, filling missing detections with interpolation and smoothing. FoundationPose tracks the 6-DoF object pose from the known first-frame object pose and known asset geometry; the paper fine-tunes it for RGB-only tracking by zeroing depth channels.
Independent estimates are not enough. They can produce floating contacts, penetrations, and depth-scale drift. GRAIL therefore performs joint optimization over residual human and object trajectory updates:
\[ L = \lambda_{\mathrm{kp}}L_{\mathrm{kp}}
- \lambda_{\mathrm{proj}}L_{\mathrm{proj}}
- \lambda_{\mathrm{depth}}L_{\mathrm{depth}}
- \lambda_{\mathrm{cont}}L_{\mathrm{cont}}
- \lambda_{\mathrm{reg}}L_{\mathrm{reg}}. \]
Each term has a specific role. Keypoint loss keeps projected human motion aligned with detected 2D body and hand keypoints. Object projection loss preserves image-space alignment with the tracked object pose. Depth loss uses MoGe-2 and SAM2 to generate metric human/object point clouds and align visible mesh vertices through Chamfer distance. Contact loss uses VLM-predicted contact labels to pull relevant hand/body regions and object regions together in depth. Regularization suppresses foot skating, velocity drift, and temporal jitter.
This objective is the mechanism that makes the generated video usable as robot data. The video model provides a plausible interaction prior; optimization turns it into a metric, temporally coherent, contact-aware 4D trajectory.
The paper also filters failures. Generated videos can have texture inconsistency, blurry motion, or geometry mismatch. GRAIL compares SAM2 object masks against rendered silhouettes from FoundationPose predictions and discards sequences whose mask tracking error exceeds a threshold. This is an important engineering detail: the pipeline is scalable because it can generate and filter, not because every VFM sample is reliable.
Stage 3: Retargeting and Task-General Tracking
The optimized SMPL-X motion is retargeted to Unitree G1 with GMR, while the object trajectory provides a reference object pose. GRAIL then trains tracking policies on top of SONIC, a pretrained whole-body controller. The point is to convert retargeted references into robot-action data without fitting a new controller per sequence.
The paper uses two complementary trackers:
| Tracker | Used for | Adaptation |
|---|---|---|
| Object-aware adaptor | pick-up and whole-body manipulation | freezes SONIC and learns a latent residual plus hand open/close primitives |
| Scene-aware tracker | stairs, curbs, slopes, sitting | fine-tunes SONIC with a height-map encoder for terrain-conditioned control |
The object-aware adaptor observes proprioception and object references, including object pose in the robot body frame, hand-to-object transforms, finger contact forces, a BPS shape encoding, and future reference deltas. It outputs a latent residual \(\Delta z_t\) and left/right hand primitives:
\[ (\Delta z_t, a_t^{\mathrm{hand}}) = \pi_\phi(s_t, o_t), \qquad a_t^{\mathrm{body}} = \mathcal{G}(z_t + \lambda \Delta z_t). \]
This design is narrow and effective. The pretrained locomotion prior stays intact, while manipulation-specific adaptation enters through latent residuals and simple hand primitives. For terrain and sitting, hand-object interaction is not the main issue; the scene-aware tracker uses an 11-by-11 local height map around the robot, encoded by a CNN, to condition whole-body control on scene geometry.
Training is large-scale. The appendix reports PPO training in Isaac Lab on 64 NVIDIA L40 GPUs, with 1,024 environments per GPU and 30,000 iterations for each tracker. That is a reminder that GRAIL reduces physical data cost, not overall compute cost.
Results
For 4D HOI generation, GRAIL is compared with CHOIS, HOIDiff, and DAViD on 20 everyday objects. It obtains the lowest contact distance, lowest penetration ratio, highest VLM interaction score, smoothest object trajectories, and a much higher physics-based tracking success rate: 88.9%, compared with 24.0% for DAViD, 15.8% for HOIDiff, and 10.5% for CHOIS.
For task-general loco-manipulation tracking, GRAIL is compared with HDMI and ResMimic on 124 motions across 43 objects. It reports 81.4% success rate, with lower object position error than the baselines. The ablations are informative: removing SONIC hurts body tracking, removing the object-aware adaptor gives the lowest manipulation success despite good body imitation, and replacing relative object observations with absolute ones reduces success. The takeaway is that whole-body imitation alone does not solve interaction; the policy needs object-aware adaptation.
For sim-to-real, the paper trains egocentric visual policies from the generated data and deploys them on a Unitree G1. The setup uses a Luxonis OAK-D W camera and streams inference through a desktop with an NVIDIA RTX 5090. Stair-climbing reaches 90% real-world success. Pick-up is trained on 200 approach-and-pick-up sequences per seen object and achieves 84% success on seen objects and 80% on unseen objects.
What Is Technically Important
The main technical idea is not “use video generation for robotics” in a loose sense. The useful part is asset-conditioned generation plus privileged reconstruction. Because GRAIL owns the 3D setup before video generation, it can use generated video as a motion prior while retaining metric anchors for reconstruction. This is a stronger formulation than taking arbitrary videos and hoping reconstruction can infer everything.
The second important idea is amortization. GRAIL trains task-general trackers over pools of related references. The goal is not to produce one perfect sequence, then replay it. The goal is to generate many related trajectories, retarget them, and train policies whose competence spans a task family.
The third point is the clear split between digital scalability and physical validation. GRAIL’s dataset is generated without teleoperating the robot, but the paper still validates through real Unitree G1 deployment. That matters because humanoid loco-manipulation papers can look strong in simulation while hiding retargeting and sim-to-real failure modes.
Limitations
GRAIL assumes usable 3D object assets, simulator-ready scenes, and a video model that follows the requested interaction. If the VFM creates severe occlusion, fast motion, appearance inconsistency, or geometry mismatch, reconstruction degrades and filtering discards the sample. This makes the pipeline scalable, but not automatic in the sense of accepting arbitrary generated videos.
The method also depends on expensive downstream training. The physical data burden is reduced, but the tracking policies still use large-scale Isaac Lab PPO training. For labs without substantial GPU infrastructure, this is a practical constraint.
Finally, GRAIL’s retargeting route benefits from human-like morphology. The character is prefitted to Unitree G1, and the controller stack is built around SONIC and Unitree-specific deployment. Extending the same data to substantially different humanoids or dexterous hands would require careful retargeting and controller adaptation.
Takeaways
GRAIL is best read as a data-engineering paper for humanoid robotics. Its contribution is a controlled way to turn 3D assets and video priors into robot-compatible 4D references, then use those references to train task-general policies.
For future humanoid systems, the pattern is likely to be important: specify the world first, use generative models for behavioral variation, reconstruct with privileged geometry, filter aggressively, and validate through sim-to-real. Real robot data remains necessary, while a large part of exploration and coverage can move into a digital pipeline.