[Paper Notes] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
Published:
TL;DR
HumanEgo learns deployable robot manipulation policies from roughly 30 minutes of human egocentric demonstrations per task, without task-specific robot data or large-scale pretraining. Its central idea is to represent manipulation through explicit hand–object interaction geometry. Each hand and object becomes a 29D Interaction-Centric Token (ICT) containing entity type, pose, relations to both hands, and grasp state. A conditional flow-matching policy maps these observations to bimanual action chunks, while three auxiliary objectives forecast object motion, 2D traces, and future interaction state.
The experiments make a sharp representational point. Closing the visual appearance gap alone reaches at most 32.5% success on the representation ablation, whereas adding ICT to raw human RGB raises success from 7.5% to 85%. With the full pipeline, HumanEgo reports 92.5% average success across four real-world tasks using 30 minutes of human data per task, and 75% with 15 minutes. The approach is compelling because it identifies a compact transferable state; its main dependency is the perception stack that must recover stable, metric 3D hand and object poses.
Paper Information
- Title: HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
- Authors: Zhi (Leo) Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, Yiannis Aloimonos
- Affiliation: University of Maryland
- Status: arXiv preprint, arXiv:2605.24934v2, May 2026
- Links: Project page · Paper · Code
The Problem: Transfer the Interaction, Not the Body
Human egocentric video is easy to collect, but a robot cannot directly imitate its pixels or human joint trajectories. The visual gap includes different arms, grippers, cameras, and backgrounds. The kinematic gap includes different morphologies, workspaces, grasp mechanisms, and executable motion constraints. Existing methods often encode hand trajectories, object trajectories, sparse points, or visually retargeted robot images. Each captures part of a skill while leaving the relation between the manipulator and the object implicit.
HumanEgo defines the transferable unit as interaction geometry: how the hands approach, grasp, transport, coordinate around, and release task entities. The complete pipeline is:
Aria egocentric demonstrations
-> hand/object tracking and motion optimization
-> arm inpainting + virtual-gripper/keypoint rendering
-> Interaction-Centric Tokens
-> flow-matching bimanual action policy
-> zero-shot robot deployment
This decomposition assigns separate mechanisms to separate gaps. Visual preprocessing reduces appearance mismatch. ICT exposes the spatial state shared by human and robot. Flow matching models multiple valid action trajectories. Auxiliary forecasting extracts more learning signal from each short demonstration set.
From Human Hands to Executable Gripper Actions
A demonstrator wears Aria Gen1 glasses, which provide synchronized RGB, calibrated SLAM, and stereo-based 3D hand keypoints through Meta’s Machine Perception Services. Demonstrations are recorded at 30 Hz. The system uses five stable keypoints per hand: wrist, thumb MCP, thumb tip, index MCP, and index tip.
The virtual parallel-jaw gripper position is the midpoint between thumb and index fingertips:
\[\mathbf{p}_{ee}=\frac{1}{2}\left(\mathbf{p}_{thumb\ tip}+\mathbf{p}_{index\ tip}\right).\]For orientation, HumanEgo builds a Gram–Schmidt frame from the wrist and the thumb/index MCP joints. MCP joints remain separated during a pinch, avoiding the orientation degeneracy that occurs when fingertip-based axes collapse at contact. The gripper aperture is a normalized thumb–index distance:
\[g=\operatorname{clip}\left( \frac{\lVert\mathbf{p}_{thumb\ tip}-\mathbf{p}_{index\ tip}\rVert-d_{min}} {d_{max}-d_{min}},0,1\right),\]followed by filtering and binary open/close control at deployment. Confidence filtering, gap interpolation, Savitzky–Golay position smoothing, and EMA rotation smoothing convert noisy hand estimates into coherent action labels.
Objects are detected with Grounding DINO, segmented with SAM2, tracked in 2D with CoTracker3, and triangulated into 3D using camera intrinsics and Aria SLAM. Orient-Anything V2 estimates object orientation. When a grasp causes occlusion, kinematic latching rigidly attaches the estimated object pose to the hand until release.
Interaction-Centric Tokens
For each entity $k$—a left hand, right hand, or task object—HumanEgo constructs a 29D token:
\[\mathrm{ICT}_k= [\tau\;\Vert\;{}^{REF}T_E\;\Vert\;{}^ET_{LH}\;\Vert\;{}^ET_{RH}\;\Vert\;g].\]Here, $\tau$ is the entity type; ${}^{REF}T_E$ is the entity pose in a shared reference frame; ${}^{E}T_{LH}$ and ${}^{E}T_{RH}$ express both hand poses in that entity’s local frame; and $g$ is the grasp state for a hand or a sentinel value for an object. Every $SE(3)$ transform is flattened to 9 dimensions using normalized translation and a continuous 6D rotation representation.
The token has three useful properties. First, hand poses expressed in an object’s frame directly encode approach, contact, transport, and release. Second, relative transforms reduce sensitivity to embodiment and viewpoint. Third, the entity list is variable-length, so the same policy interface can represent tasks with different numbers of objects.
HumanEgo still supplies RGB observations. It removes the human arm with SAM2 and LaMa, then renders a virtual gripper and tracked object keypoints. This channel retains visual context, while ICT carries the explicit 3D interaction state that is difficult to infer reliably from monocular pixels.
Flow Matching and Dense Auxiliary Supervision
Given RGB and ICT state $s_t$, the policy generates a $K$-step bimanual action chunk containing both end-effectors’ positions, 6D rotations, and binary grasps. Conditional flow matching transports Gaussian noise $x_0$ to a ground-truth action chunk $x_1$ along
\[x_t=(1-t)x_0+t x_1, \qquad t\sim\mathcal{U}(0,1).\]The velocity model learns the target displacement $x_1-x_0$ with separate weights for position, rotation, and grasp:
\[\mathcal{L}_{FM}=\mathbb{E}\left[ w_p\lVert\Delta p\rVert^2+ w_r\lVert\Delta r\rVert^2+ w_g\lVert\Delta g\rVert^2 \right].\]The implementation uses a 6-layer, 8-head transformer decoder with embedding dimension 384. At inference it integrates the learned ODE with 20 fixed Euler steps and produces a 50-step action chunk.
Three heads share the context encoder and forecast future scene evolution:
- Object motion: predicts the manipulated object’s future 6-DoF trajectory, supervising physical 3D dynamics.
- 2D trace: predicts future image-plane trajectories of entity anchor points, tying state features to visual motion.
- Latent consistency: predicts future ICT hand states, encouraging a temporally predictive interaction representation.
The full objective is
\[\mathcal{L}=\mathcal{L}_{FM} +\lambda_{OM}\mathcal{L}_{OM} +\lambda_{2D}\mathcal{L}_{2D} +\lambda_{LC}\mathcal{L}_{LC}.\]All targets come from the same perception pipeline, so the auxiliary tasks require no extra manual labels. Together they act as a lightweight interaction world model and as multi-task regularization, with the strongest value in the low-data regime.
Real-World Results
The main evaluation uses dual Trossen WidowX arms and a top-mounted RealSense D405. Each result is measured over 40 trials with randomized object positions. The four tasks cover distinct control demands: Serve Bread is pick-and-place; Downstack Cups is long-horizon and multi-step; Water Flowers requires contact-rich bimanual ordering; Adjust Table requires sustained rotational control.
| Method / data budget | Average | Serve Bread | Downstack Cups | Water Flowers | Adjust Table |
|---|---|---|---|---|---|
| HumanEgo, 30 min human | 92.5 | 95.0 | 87.5 | 95.0 | 92.5 |
| HumanEgo, 15 min human | 75.0 | 82.5 | 67.5 | 75.0 | 75.0 |
| ACT, 30 min robot teleop | 51.2 | 52.5 | 45.0 | 45.0 | 62.5 |
| Best human-video baseline per task | — | 62.5 | 45.0 | 45.0 | 47.5 |
HumanEgo improves over matched-time ACT by about 41.3 percentage points in average success. On Serve Bread, eight minutes of human data reaches 57.5%, already exceeding ACT trained from 30 minutes of robot teleoperation at 52.5%. The paper attributes this efficiency to smoother motion, less idle time, higher signal-to-noise ratio, and broader spatial and trajectory coverage in the human demonstrations.
The same policies are evaluated without retraining under new robot embodiments, cameras, viewpoints, heights, lighting, backgrounds, objects, and distractors. Reported success remains between 85% and 91.25% across these conditions. Transfer to Franka and UR10 arms is especially relevant because the training demonstrations contain no robot hardware.
What the Ablations Show
The representation study gives the paper’s clearest result:
| Input representation | Success on Water Flowers |
|---|---|
| Raw human RGB | 7.5% |
| Inpainted RGB + keypoints | 20.0% |
| Robot RGB | 32.5% |
| Raw human RGB + ICT | 85.0% |
| Full HumanEgo | 95.0% |
Even robot RGB, which removes the visual embodiment mismatch, reaches only 32.5%. ICT adds the explicit spatial relation that pixels fail to recover from a small dataset. The 7.5% to 85% jump from adding ICT to raw human RGB supports the paper’s core claim more directly than the aggregate benchmark.
At the 15-minute budget, the auxiliary-objective study starts from a 50% baseline. Object motion reaches 67.5%, latent consistency 62.5%, and 2D trace 55%. Combining all three reaches 75%, a cumulative gain of 25 percentage points. On Serve Bread, the largest low-data gap appears at eight minutes: 57.5% with auxiliary losses versus 37.5% without them. Once the dataset reaches 30 minutes, both variants converge near 95%, showing that dense supervision mainly changes sample efficiency.
Strengths and Limitations
The strongest contribution is the alignment between the claimed problem and the chosen representation. ICT makes the hand–object relation explicit, preserves variable numbers of entities, and gives human and robot observations a common interface. The auxiliary heads reuse signals already available in each trajectory, which is a practical way to regularize a policy trained from minutes of data. The paper also evaluates tasks beyond simple pick-and-place and reports ablations that separate visual preprocessing, spatial representation, and learning objectives.
The main limitations are equally concrete:
- Perception quality is a hard dependency. Replacing Aria’s stereo hand tracking with monocular WiLoR, HaMeR, or MediaPipe drops Serve Bread success from 95% to 45%, 32.5%, and 0%. Metric depth, temporal smoothness, and tracking persistence directly affect the action labels and ICTs.
- The preprocessing chain is long. Grounding DINO, SAM2, CoTracker3, triangulation, orientation estimation, inpainting, and hand tracking create multiple opportunities for cascading errors.
- Object tracking is offline and occlusion-sensitive. Per-frame detection and kinematic latching fit the evaluated tasks; in-hand manipulation, fast motion, and complex occlusion need stronger online tracking.
- Precision plateaus near one centimeter. Sub-centimeter contact-rich behavior may require downstream reinforcement learning or simulation refinement.
- The evidence remains task-scale. Four curated real-world tasks demonstrate strong transfer, while open-world object diversity, unseen task composition, failure recovery, and long-duration deployment remain open.
Takeaways
HumanEgo suggests a useful recipe for learning robot actions from small, high-quality egocentric datasets. Capture metric hand motion with a reliable wearable system; retarget the hand into an executable end-effector representation; encode hands and objects through relative $SE(3)$ relations; train a multi-modal action generator; and predict future dynamics in several complementary spaces.
The deeper lesson is representational. Embodiment transfer becomes more manageable once the state describes what entities are doing relative to one another. Visual editing can reduce appearance mismatch, but explicit interaction geometry supplies the transferable control state. HumanEgo’s reported results show how far this idea can go with minutes of demonstrations, while its hand-tracking ablation shows exactly where the current pipeline remains fragile.