[Paper Notes] Data Pyramid for Embodied Manipulation
Published:
Data Pyramid for Embodied Manipulation gives the fragmented embodied-data landscape a useful coordinate system. It organizes five sources—real-robot, UMI-style, egocentric/exocentric, simulation, and general multimodal data—along a central tension: data becomes easier to scale as it moves toward the base, while direct alignment with robot execution generally becomes stronger toward the apex.
My main takeaway is that the pyramid should guide data composition, not rank datasets with a single score. Each layer supplies a different missing capability. General data provides semantics and reasoning; human video provides interaction structure; simulation provides controllable action-consequence pairs; UMI preserves portable end-effector supervision; real-robot trajectories ground the final policy in hardware, sensing, and contact. The unresolved research problem is how much of each source to use, at which training stage, and under which action representation.
Paper Info
The paper is “Data Pyramid for Embodied Manipulation” by Yifan Ye, Yankai Fu, Yaoxu Lv, Bohan Hou, Jun Cen, Lingdong Kong, Duo Zheng, Tianxing Chen, Jiaming Liu, Ziang Cao, Yunfan Lou, Wei Chow, Xian Sun, Yingshuo Wang, Kuangzhi Ge, Xiaowei Chi, Xidong Zhang, Zhibo Pang, Yiwu Zhong, Sirui Han, Zhihe Lu, Weihao Yuan, Qifeng Chen, Michael Yu Wang, Yao Mu, Ziwei Liu, Jianfei Yang, Ping Luo, and Shanghang Zhang. The authors span PKU, NTU, HKUST, NUS, CUHK, HKU, Duke, UC Berkeley, GBU, NJU, and SJTU. These notes refer to arXiv:2607.24744v1, submitted on July 27, 2026.
This is a data-centric survey and taxonomy. Its scope is the framework connecting data collection, representation, model family, and research capability; it introduces no new policy architecture or benchmark result.
The Two Axes and Four Supporting Dimensions
The pyramid is organized primarily by two axes:
| Dimension | Question |
|---|---|
| Scalability | How efficiently can the source expand under hardware, labor, reset, safety, and marginal-generation costs? |
| Robot alignment | How directly do its observations, representations, and supervision support learning and execution on a physical robot? |
These axes usually pull in opposite directions. A real-robot trajectory contains executable commands and authentic consequences, while every additional hour consumes hardware and operating effort. Web video can grow by orders of magnitude with weak direct supervision for control.
The paper adds four dimensions because scale and alignment alone cannot describe data utility:
- Quality: validity, synchronization, informativeness, consistency, and task relevance;
- Diversity: coverage across tasks, scenes, objects, viewpoints, embodiments, sensors, behaviors, and outcomes;
- Reusability: transfer across robots, tasks, environments, sensing systems, and model families;
- Physical fidelity: contact, friction, compliance, latency, sensor noise, actuation, and object dynamics.
A large dataset can still have narrow state-action coverage. A physically authentic human video can still lack robot-compatible actions. A simulator can offer exact contact labels while approximating the contact physics that produced them. The six dimensions expose these distinctions.
The Five-Layer Data Pyramid
| Layer, apex to base | Supervision retained | Main advantage | Main bottleneck |
|---|---|---|---|
| Real-robot data | observations, robot states, actions, outcomes | direct executability and authentic hardware interaction | cost, resets, safety, platform dependence |
| UMI-style data | wrist-view observations, relative end-effector trajectories, gripper state; sometimes force or touch | portable in-the-wild collection with explicit action structure | tracking, calibration, retargeting, missing robot dynamics |
| Egocentric / exocentric data | human activity, hand-object interaction, task progression; optional pose, gaze, force, EMG, or touch | real-world diversity and natural high-DoF behavior | partial observability and human–robot embodiment gap |
| Simulation data | executable actions, privileged states, contacts, rewards, success signals | parallel, controllable, low-marginal-cost generation | asset coverage and observation/dynamics sim-to-real gaps |
| General data | image, video, language, spatial, 3D, planning, and reasoning supervision | web-scale semantic and cognitive coverage | weak action, contact, and consequence grounding |
The ordering summarizes the overall trade-off across all six dimensions. Individual properties can depart from that order. Simulation, for example, is highly robot-aligned at the action-interface level and less physically faithful than unstructured human video.
1. Real-Robot Data: The Physical Anchor
Real-robot data records the closed loop that matters at deployment:
[ (\text{observation},\ \text{robot state},\ \text{action},\ \text{physical outcome}). ]
It naturally includes sensor noise, controller latency, kinematic limits, actuator response, contact, and hardware failures. The actions are executable on the platform that generated them. These properties make real-robot data the strongest source for final policy grounding and recovery behavior.
Scale has grown from narrow grasping collections to multi-task and multi-embodiment corpora. The survey cites MT-Opt at roughly 800K episodes across 12 tasks, RT-1 at 130K trajectories, RoboMIND at 107K, AgiBot World Beta at one million trajectories and nearly 3,000 hours, RoboMIND 2.0 at more than 310K trajectories across 739 tasks, and Open X-Embodiment at more than two million aggregated trajectories.
Raw trajectory count is an incomplete measure. Variation in initial states, paths, contact sequences, speed, task stage, scene layout, sensor suite, and embodiment often matters more than repeated demonstrations of one behavior. The paper therefore argues for wider in-the-wild coverage and for preserving policy rollouts, human interventions, failures, and recoveries.
2. UMI-Style Data: Action Structure Without a Robot in the Loop
UMI-style systems use a portable handheld gripper or wearable interface equipped with cameras, pose tracking, and gripper sensing. The operator directly performs tasks in ordinary environments while the device records observation and action-like signals. Recent variants add bimanual capture, dexterous hands, 3D sensing, force, and touch.
The key representation is a future end-effector trajectory relative to the current end-effector pose. Relative 6-DoF motion plus a gripper command avoids dependence on a global tracking frame and gives different robots a reusable task-space target. Deployment composes this motion with the robot’s current pose, then uses inverse kinematics, motion planning, or a Cartesian controller to produce joint commands.
This separation is valuable:
- collection remains robot-free and portable;
- the demonstration retains more control structure than ordinary human video;
- embodiment-specific execution is deferred to retargeting and low-level control.
The gap remains substantial for dexterous hands. Wrist motion alone cannot encode finger morphology, contact geometry, compliance, or force. DexUMI, for example, constrains human motion with a wearable exoskeleton and visually replaces the human hand with the target robot hand. Even then, visual plausibility and dynamic feasibility are different tests.
3. Egocentric and Exocentric Data: Human Interaction Priors
Human video occupies the middle of the pyramid. It preserves real objects, real physics, dexterous hand use, tool-use strategies, and long-horizon activity structure at much greater breadth than robot collection.
First-person capture is especially relevant to manipulation, but it brings hand-object occlusion, camera motion, blur, and limited field of view. Synchronized exocentric cameras improve reconstruction and body context while increasing setup, calibration, synchronization, and storage costs. Wearable sensing can add depth, gaze, IMU, EMG, force, or tactile pressure. Post-processing can recover hand pose, object pose, trajectories, action segments, task labels, and contact events.
Robot-oriented use requires another transformation:
[ \text{human observation} \rightarrow \text{hand/object reconstruction} \rightarrow \text{shared or retargeted representation} \rightarrow \text{robot-compatible action}. ]
The paper’s most useful framing for dexterity is to treat human video as structured interaction supervision. It reliably contributes task intent, object affordances, grasp choices, contact order, tool-use strategy, and task decomposition. Robot data must still ground those priors in a particular hand’s joints, forces, friction, sensing, and actuator limits.
4. Simulation Data: Scalable Robot-Oriented Experience
Simulation provides the strongest combination of scalability and explicit robot supervision. It can generate actions, object states, segmentation, contacts, physical parameters, rewards, failures, and success labels in parallel without hardware wear or safety risk. It is useful for policy pretraining, controlled evaluation, curriculum generation, and rare-state coverage.
The central limitation has two parts:
- Observation mismatch: textures, illumination, material appearance, depth noise, calibration, occlusion, tactile response, and force signals differ from reality.
- Interaction mismatch: morphology, coordinate frames, control frequency, latency, compliance, backlash, friction, deformation, torque limits, and multi-body contact remain imperfect.
Scale also does not guarantee behavioral diversity. A generator constrained by a small skill library can produce many trajectories whose key action states remain concentrated in one narrow region.
World models extend simulation from manually constructed physics environments to learned predictive environments. They can support policy optimization, checkpoint evaluation, or synthetic data generation. Their failure mode is different: a generated rollout may look coherent while containing ambiguous actions or physically invalid consequences. Filtering, uncertainty estimation, and calibration against physics or real interaction are therefore essential.
5. General Data: The Cognitive Base
General image, video, language, spatial, 3D, planning, and reasoning data gives embodied models capabilities that robot trajectories rarely cover at sufficient scale:
| Data form | Capability contributed |
|---|---|
| image–text and VQA | semantics, object properties, language grounding, commonsense |
| segmentation and localization | spatial reference and interaction-region grounding |
| 3D data | geometry, pose, depth, and spatial structure |
| video | motion, object-state changes, temporal order, and procedural memory |
| planning and task decomposition | goals, subgoals, step order, and long-horizon reasoning |
| physical and causal QA | plausibility, consequence, and failure reasoning |
| grasp resources | contact and grasp priors that still require a robot-specific generator |
This layer offers enormous diversity and reuse across embodiments. It does not record proprioception, actuator dynamics, contact forces, or executed robot actions. Automatically generated captions, plans, and reasoning traces also introduce hallucinated or physically inconsistent supervision. General data is best understood as a cognitive prior that higher layers ground in interaction.
Data Recipes Are Becoming Broader—and Remain Unsolved
The survey identifies three trends.
First, model families are adding more pyramid layers. The π series moves from real-robot data in π0, to real-robot plus general data in π0.5, then adds egocentric data in π0.7. LingbotVA uses real-robot, UMI, and simulation data; LingbotVA 2.0 reports all five layers.
Second, the reported scale is rising rapidly. Qwen-RobotManip describes an approximately 38,100-hour multi-source corpus, including about 11.4K hours of open robot data and 24,808 hours of robot-compatible trajectories synthesized from 1,933 hours of egocentric video. Xiaomi-Robotics-1 reports more than 100,000 hours of UMI pretraining data followed by roughly 10,000 hours of cross-embodiment post-training.
Third, egocentric data is becoming a major pretraining substrate. Some models mix it with robot and general data, while EgoVLA, Being-H0, H-RDT, VITRA, UniDex, HumanScale, and related systems place human interaction at the center of pretraining.
These reports do not establish a universal recipe. Hours from raw video, reconstructed trajectories, UMI recordings, simulation, and robot execution are not directly comparable. Model architecture, filtering, prompts, action representation, and training stage all confound attribution. Strong robot-only models also remain competitive.
Action Representation Is Part of the Data Recipe
Mixing datasets requires more than putting them into the same file format. The paper separates structural alignment from geometric alignment.
Structural alignment across embodiments
| Strategy | Mechanism | Trade-off |
|---|---|---|
| Embodiment-specific projection | each robot keeps its native action space and uses a dedicated adapter or head | preserves native control; sharing happens inside the backbone |
| Fixed-dimensional padding | all actions use one vector length; inactive dimensions are zeroed and masked | simple batching; aligned shapes do not guarantee aligned meanings |
| Semantic action slots | fixed vector blocks have consistent physical meanings across robots | stronger semantic alignment; requires a carefully designed canonical schema |
The survey highlights Qwen-RobotManip’s 80-dimensional canonical representation: two 29-dimensional arm blocks plus 22 reserved dimensions, with arm joints, end-effector pose, gripper, and dexterous-hand fields assigned explicit slots.
Geometric alignment
End-effector actions may be expressed in:
- a robot/world frame, which interfaces naturally with controllers but inherits mounting and workspace conventions;
- a camera frame, which can align visually similar motions across robots;
- a wrist frame, which separates local hand articulation from global arm motion.
Every dataset should record frame origin and axes, handedness, tool-center point, absolute versus delta mode, rotation parameterization, units, calibration, and controller mode. Shared storage without shared geometric semantics can create contradictory supervision.
Three Model Families Consume the Pyramid Differently
| Model family | Primary target | Role of action-free data | Role of action-labeled data |
|---|---|---|---|
| Embodied brain / VLM | perception, grounding, reasoning, memory, planning | semantics, spatial-temporal understanding, physical priors | affordances, waypoints, task boundaries, next-action and planning supervision |
| VLA | executable action generation | latent actions, motion fields, visual goals, plans, affordances | direct policy learning through discrete tokens, diffusion, or flow matching |
| World Action Model | world evolution and action consequence | broad temporal and physical priors | action-conditioned dynamics, policy evaluation, control, and synthetic rollouts |
For VLAs, discrete action tokens reuse language-model machinery, while continuous diffusion or flow-matching heads better preserve high-frequency and high-dimensional control. Action-free video can provide latent motion tokens, point trajectories, object motion fields, and intermediate plans; robot demonstrations or an action decoder still connect those proxies to execution.
WAMs commonly use a two-stage recipe: large-scale action-free pretraining learns generic dynamics, then action-conditioned post-training grounds the model in interaction. Continuous-action systems use diffusion or flow matching; autoregressive systems can discretize observations and actions into joint token sequences.
Six Open Problems
1. Large-scale tactile data
Vision gives indirect evidence of slip, force, deformation, friction, and grasp stability. Tactile data remains sensor-specific, inconsistent in format, narrow in task coverage, and rarely synchronized with long-horizon trajectories. A standard contact layer is still missing from most embodied datasets.
2. Failure and recovery data
Success-only demonstrations leave policies poorly prepared for their own deployment distribution. Useful failure records need temporal and causal structure: pre-failure context, onset, category, cause, state change, corrective action, and recovery outcome. Failed trajectories can also contain reusable successful segments.
3. Scalable collection across pyramid layers
Wearable systems need lighter, wireless, modular, and less intrusive hardware, along with automatic calibration, on-device hand-object reconstruction, and standardized metadata. Collection should target uncertain, rare, and underrepresented states instead of only increasing raw volume.
4. Cross-embodiment state-action alignment
Cartesian end-effector poses still conflict when datasets use different base, camera, wrist, or world frames. Coordinate conventions, controller semantics, and calibration must become first-class metadata.
5. Egocentric priors for dexterous hands
Human and robot hands differ in topology, DoF, geometry, compliance, sensing, friction, and force limits. Human data is most dependable for interaction structure; morphology-conditioned policies, contact-centric representations, uncertainty-aware retargeting, and robot grounding must convert that structure into feasible execution.
6. Principled data recipes
The field lacks compute-matched ablations that isolate individual sources and categories. Future studies should compare fixed mixtures, curricula, and adaptive sampling under controlled architecture, representation, and compute budgets.
Critical Reading
The paper’s strongest contribution is conceptual compression. It turns a long list of datasets into a decision framework and then connects that framework to model capability. The six-dimensional view prevents “more data” from becoming a content-free strategy.
Three caveats matter:
- The pyramid is a taxonomy whose ordering has not been fitted as an empirical law. It is a category-level synthesis; individual datasets can violate the expected trade-offs.
- Scale numbers are heterogeneous. Trajectories, hours, clips, frames, QA pairs, and synthetic episodes measure different quantities and vary widely in information density.
- Data-mixture claims remain correlational. Modern systems change architecture, action representation, curation, compute, and data at the same time. The marginal value of each pyramid layer is rarely isolated.
The most productive way to use the framework is to ask which capability and failure mode a source covers:
[ \text{semantics} \rightarrow \text{interaction prior} \rightarrow \text{controllable experience} \rightarrow \text{portable action structure} \rightarrow \text{hardware grounding}. ]
This sequence is a practical design heuristic. The training order can vary with the target capability and available supervision.
Takeaways
For building a manipulation foundation model, a reasonable interpretation of the pyramid is:
- use general data to establish perception, language, spatial reasoning, and task structure;
- use egocentric/exocentric data to learn human interaction priors and long-horizon behavior;
- use simulation to scale explicit action-consequence supervision and rare-state coverage;
- use UMI-style data to collect portable real-world demonstrations with end-effector structure;
- use real-robot data to calibrate the final action representation, contact dynamics, failures, and recovery.
The next leap will depend on the interfaces between layers: shared geometry, action semantics, calibrated uncertainty, touch, failure structure, and controlled mixture studies. The data pyramid is valuable because it makes those interfaces visible.