Four Stages of Embodied Model Training: From Semantic Priors to Robot Deployment
Published:
I started with a simple hypothesis: an embodied model passes through four rounds of training—language or vision-language pretraining, robot foundation training, a human-data middle stage, and task-specific fine-tuning. The more training recipes I compared, the less convincing that fixed chronology became. Some learn actions from human data before seeing a robot. Others adapt a video model directly on task demonstrations. Still others acquire a new behavior through a prompt, without another gradient update.
The useful part of the hypothesis survives, with one revision: these are four functional stages, not four mandatory training runs. They describe where semantic knowledge, physical priors, actionable representations, and deployment competence come from. Their boundaries can overlap; their order is not universal.
The interactive map emphasizes shared ingredients. The article explains how different models use those ingredients, what the public evidence supports, and where this framework stops being literal. Use the navigation’s English / 中文 switch for both the text and the map. Sources and version scope were checked on September 5, 2026.
Four stages of embodied-model training
A shared map of capabilities and data—not four mandatory training runs.
Select a model to highlight shared components. Gray means not mapped here, not proven absent.
Semantic foundations
Physical-world priors
Action foundations / grounding
Deployment adaptation
The data continuum
Toward more explicit action alignment—not a quality rankingπ0.6 / π*0.6: the RL highlight refers to π*0.6 / RECAP, not supervised π0.6. Training a controller, running feedback control, and adapting model weights are distinct operations. See the article for version scope and evidence.
Open the interactive map on its own →
1. What the four stages actually mean
| Stage | Name used here | The question it answers |
|---|---|---|
| 1 | Pre-training — semantic foundations | How does the system understand language, objects, scenes, and goals? |
| 2 | Pre-training — physical-world priors | How does it learn interaction, temporal change, and plausible motion at scale? |
| 3 | Pre-training / Mid-training — action foundations and grounding | How does that knowledge become an action representation that can support robot control? |
| 4 | Fine-tuning / Post-training — deployment adaptation | How is a pretrained system adapted to a target embodiment, task, or operating distribution? |
For a conventional VLA, Stage 1 often means inheriting a pretrained VLM. As one scale reference, Qwen3 reports approximately 36 trillion tokens for its pretraining corpus. The unit is tokens—not “trillions of text”—and image/video counts need their own accounting. This is one model family’s disclosure, not a universal budget. Qwen3 technical introduction.
Stage 2 need not output executable actions. Predicting observations or learning latent transitions from action-free video can supply useful temporal structure. But a latent action inferred from video is not automatically an identifiable robot command; observational prediction alone also does not establish causal control.
Stage 3 is about action grounding, not a particular sensor. A broad robot corpus can serve this function, as can human motion mapped into a common action space. Stage 2 and Stage 3 may happen in the same training mixture. In a native sensorimotor model, physical knowledge and action structure can develop together, while the final correspondence to a particular robot is learned later.
Stage 4 concerns a narrower deployment distribution. It may use supervised learning, reinforcement learning, distillation, or combinations of them. Engineering a controller or reducing latency can be necessary for deployment, but these operations do not all constitute model fine-tuning. The map includes this deployment context without pretending that every highlighted box is a gradient-based stage.
2. The data continuum is a different axis
“Egocentric” describes the viewpoint. It does not tell us whether a sample contains raw video, reconstructed hand poses, calibrated motion capture, a robot-compatible command, or actual robot feedback. Those distinctions determine what supervision the sample provides.
I find the following progression useful: web semantics → observed interactions → human motion signals → robot-aligned human actions → robot-native trajectories. This is an alignment spectrum, not a ranking of intrinsic data quality or collection cost. Robot-native data from the wrong task or hardware can still be far from the intended deployment distribution.
A MANUS-type glove, an exoskeleton, or a handheld interface can provide much richer supervision than ordinary video. Yet accurate human joint measurements do not by themselves specify robot control targets. Coordinate frames, kinematics, timing, actuation, contact, and dynamics still matter. Hardware–data co-design can reduce this gap, especially when the capture interface and robot share an action convention; kinematic similarity does not eliminate every embodiment difference.
Consequently, the same collection interface can supply several stages. A large, diverse human dataset may build Stage 2 priors; retargeted motion may support Stage 3 grounding; demonstrations collected for a particular robot and task may support Stage 4 adaptation. Detailed language labels can provide instruction supervision, but their benefit depends on temporal alignment, task coverage, and how they enter the objective—not just annotation granularity.
3. How representative models occupy the framework
The paths below are my analytical mapping, not stage names standardized by the authors. “2 + 3” means overlapping functions; “3 / 4” means grounding and deployment adaptation may share one training step. A skipped stage does not imply a missing capability.
| Model / version scope | Approximate functional path | Main emphasis |
|---|---|---|
| π0.5 | 1 → broad 3 → 4 | Heterogeneous supervised co-training and generalization |
| π0.6 / π*0.6 | 1 → 3 → 4, with experience feeding learning | Separate the supervised base from RECAP’s RL extension |
| π0.7 | 1 → overlapping 2 + 3; task-specific 4 often omitted in reported evaluations | Diverse experience and richer conditioning |
| GR00T N1 / N1.5 | 1 → mixed 2 + 3 → 4 | Real, human, simulated, and synthetic data; robot adaptation |
| LingBot-VLA 2.0 | 1 → mixed 2 + 3 → 4 | VLM-based action learning with robot and ego data |
| LingBot-VA 2.0 | Semantic components → 2 → 3 → 4, with co-training | Native causal video–action representation and efficient deployment |
| Cosmos Policy | Video foundations, 1 + 2 → combined 3 / 4 | Adapt a video model to actions, future states, and value |
| Generalist GEN-0 / GEN-1 | Native physical and action learning, 2 + 3 → 4 | High-fidelity human interaction at scale, then robot adaptation |
| Generalist GEN-1.5 | Native base → physical prompting or few-step 4 | Distinguish in-context behavior from weight updates |
| EgoScale | 1 → action-labeled human 2 → aligned 3 → 4 | Explicit human-to-robot transfer through a middle stage |
| Helix 02 | Different components occupy different stages | Semantic policy, whole-body action, and low-level control |
| Gemini Robotics, 2025 releases | 1 → robot 3 → embodiment/task 4 | Transfer semantic competence into adaptable robot policies |
The π family: supervised learning, RL, and experience reuse are different
π0.5 combines heterogeneous robot and non-robot supervision, including semantic tasks and high-level action descriptions. I place broad robot learning primarily in Stage 3, followed by deployment-oriented adaptation. Its reported recipe is supervised; it should not be used as the example of on-robot RL. π0.5 paper.
π0.6 and π*0.6 must be distinguished. The base is supervised; the starred model uses RECAP. That framework includes offline RL pretraining, task adaptation, and learning from autonomous robot experience and expert corrections. RL therefore does not belong exclusively to a final box. The shared map’s on-robot RL highlight refers specifically to the starred extension. π*0.6 / RECAP.
π0.7 emphasizes generalization through richer conditioning and a broader mixture, including autonomous trajectories and experience produced by RL-trained specialists. Reusing or distilling such trajectories is not the same claim as optimizing the resulting policy with an RL objective. Many reported evaluations avoid task-specific post-training; that is not a guarantee that every evaluated task is absent from pretraining. π0.7 paper, official introduction.
VLA foundations: scale robot learning without losing semantic priors
For GR00T N1 / N1.5, the useful distinction is between the inherited visual-language foundation, the mixed embodied training data, and target-robot adaptation. N1 describes combining human video, real robot data, and synthetic sources; N1.5 develops the data and representation recipe further. A simulator or synthetic trajectory source is not evidence of on-robot RL. GR00T N1, GR00T N1.5.
LingBot-VLA 2.0 reports 50,000 hours of robot trajectories and 10,000 hours of egocentric data, alongside a VLM-based policy and standardized action/state representation. Human trajectories are processed into usable motion supervision; future-feature and geometry objectives enrich the policy. Such auxiliary prediction does not, by itself, make a VLA equivalent to a generative world-action model. Official repository, technical report.
World–action models: the inherited prior and the training boundary both change
LingBot-VA 2.0 builds a semantic visual–action tokenizer and a causal video–action backbone rather than merely retrofitting a generic bidirectional video generator. It also uses pretrained semantic components; “from scratch” does not mean every component lacks prior training. Human–robot co-training and later adaptation connect this representation to control. Its asynchronous, observation-grounded execution is an inference mechanism, not evidence of RL. Technical report.
Cosmos Policy offers a different route: start from a pretrained video model and use a single robot-demonstration post-training stage to learn actions, future observations, and value. Within this article’s framework, action grounding and task adaptation can therefore coincide. Additional rollout data refine prediction and value for planning; the world-model label alone does not specify an RL algorithm. Cosmos Policy paper.
Native sensorimotor learning: Generalist is a company and a distinct hypothesis
Generalist AI’s GEN-0 frames high-fidelity physical interaction as a scalable foundation-training substrate. GEN-1 reports pretraining from scratch on more than half a million hours of human physical data, without robot data in that base, followed by roughly an hour of target-robot data for the reported adaptations. Its introduction also mentions learning from experience, including RL; it is not accurate to summarize the whole program as “no RL.” The public company release does not expose a RECAP-style recipe in equivalent detail. GEN-0, GEN-1.
My interpretation is that this route moves much of physical and action-foundation learning into a common human sensorimotor substrate. It does not establish that human data are literally robot commands, or that a separate semantic model is mandatory. Here, “native” describes the training substrate and modeling choice, not a certification of perfect embodiment alignment.
GEN-1.5 makes another distinction essential: the company reports physical prompting with a 3–12-second demonstration and no gradient updates, alongside adaptation using 1–10 gradient steps. Only the latter is fine-tuning. These are reported capabilities on the release’s evaluation tasks, not a universal guarantee of one-shot robot learning. GEN-1.5.
EgoScale, hierarchical control, and a semantic-first route
EgoScale is particularly close to an explicit middle-stage story: large-scale, action-labeled human pretraining, aligned human–robot play for mid-training, then downstream post-training. Its human pretraining set contains 20,854 hours. This is not simply action-free ego video; the action supervision is central to the transfer argument. EgoScale.
Helix 02 is a warning against treating an entire robot stack as one checkpoint. Its semantic, visuomotor, and low-level systems run at different rates; the low-level controller uses retargeted human motion and simulation training. Simulation RL in that component should not be relabeled as on-robot RL post-training of the semantic policy. Helix 02.
For the 2025 Gemini Robotics releases, the broad story is semantic foundations followed by robot learning and adaptation. The On-Device release describes adapting to new tasks with 50–100 demonstrations. This illustrates targeted deployment learning, not evidence that the entire stack follows a four-run schedule. Gemini Robotics report, On-Device introduction.
4. Two boundaries that prevent misleading comparisons
ACT and Diffusion Policy belong near target-task demonstrations, but not necessarily under literal fine-tuning. Their classic formulations learn task policies through supervised imitation; the policy can be trained from scratch, even if a visual encoder has prior training. Calling all of this “pure post-training” would imply a pretrained robot policy that need not exist. In this map, their location describes deployment scope, not checkpoint history. ACT, Diffusion Policy.
A rollout is data; RL is a learning method. A failed rollout might train a value model, become a correction example for behavior cloning, or enter reward-conditioned learning. A robot recovering during execution might merely be exercising a fixed policy. Likewise, a demonstration in the context window can change behavior without changing weights. To classify a recipe, I would ask what is predicted, what feedback enters the loss, and which parameters are updated—not whether a video shows recovery.
5. What the demo shows—and what the article adds
The demo is deliberately selective. It keeps model names in the selectors and shared categories in the diagram. Highlighting answers “which ingredients are relevant to this route?”, not “what is the exact order, scale, or objective of every training run?” Gray means not mapped in this compact view. The GEN selector covers a family of releases; the π0.6 selector explicitly includes its starred extension.
The article supplies the distinctions the graphic compresses: version scope, supervised versus RL objectives, a reused foundation versus native pretraining, joint versus sequential learning, and prompting versus fine-tuning. Company disclosures and technical papers also offer different levels of reproducibility. Neither the number of highlighted nodes nor training hours alone is a model-quality ranking.
My revised formulation is therefore: semantic foundations → physical priors → action grounding → deployment adaptation, understood as four questions to ask of a system. VLA, world–action, and native sensorimotor approaches differ less in whether these questions exist than in where they answer them, what supervision they can scale, and how much target-robot learning remains.