[Paper Notes] HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents
Published:
TL;DR
HY-Embodied-0.5 is Tencent Robotics X and HY Vision Team’s attempt to build an embodied VLM foundation model before attaching a robot action head. The paper’s main claim is that real-world agents need more than general image-language understanding: they need fine-grained spatial perception, temporal/trajectory reasoning, affordance grounding, and planning-oriented reasoning. The system therefore combines a modality-adaptive Mixture-of-Transformers (MoT) architecture, visual latent tokens, embodied/spatial data construction, iterative RL + rejection-sampling post-training, and large-to-small on-policy distillation.
My read: this paper is best treated as a backbone paper for embodied agents, not a complete robot policy paper. The strongest technical story is the path from embodied VLM to robot-ready VLA: first build a compact MoT-2B model that can reason about spatial and embodied tasks, then reuse it as the perception/reasoning backbone for downstream real-robot control.
Paper Info
The paper is “HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents” by Tencent Robotics X and HY Vision Team, including Xumin Yu, Zuyan Liu, Ziyi Wang, He Zhang, Yongming Rao, Fangfu Liu, Yani Zhang, Ruowen Zhao, Oran Wang, Yves Liang, Haitao Lin, Minghui Wang, Yubo Dong, Kevin Cheng, Bolin Ni, Rui Huang, Han Hu, Zhengyou Zhang, Linus, and Shunyu Yao. It appears on arXiv as arXiv:2604.07430, submitted April 8, 2026. Code and models are released at Tencent-Hunyuan/HY-Embodied, with the MoT-2B weights linked from the project repository.
The model family has two main variants:
- HY-Embodied-0.5 MoT-2B: an edge-oriented model with about 2B activated / 4B total parameters.
- HY-Embodied-0.5 MoE-A32B: a larger reasoning model with about 32B activated / 407B total parameters.
Core Problem
General VLMs are good at describing images, answering broad visual questions, and using internet-scale semantics. Embodied agents need a different mix of abilities. A robot policy or planning system has to know where objects are, how far they are, which part is graspable, what trajectory is plausible, what state will come next, and how an instruction maps into a physical sequence.
HY-Embodied-0.5 frames this as a foundation-model gap. Action outputs are only the final interface. The backbone itself needs stronger fine-grained visual perception, 3D/spatial grounding, and embodied reasoning for prediction, interaction, and planning. This is why the paper spends most of its effort on VLM architecture, embodied/spatial data, and post-training, before showing a downstream VLA experiment.
Architecture
The compact model keeps the usual VLM shape: a visual encoder produces visual tokens, and an LLM processes visual and text tokens together. The difference is that HY-Embodied-0.5 changes how visual tokens are handled after the encoder.
The visual encoder is HY-ViT 2.0, a native-resolution ViT with about 400M parameters. It supports arbitrary-resolution inputs and is distilled from a stronger internal ViT. The paper also trains a larger ViT that turns each 8 x 8 image patch into a discrete code from a 2K codebook; those codes are later used as supervision for the visual branch.
The central architectural choice is Mixture-of-Transformers (MoT). Instead of forcing visual and text tokens through fully shared transformer parameters, the model duplicates the QKV and FFN parameters for a vision branch while keeping the language branch separate. Visual tokens use the vision-specific parameters; text tokens use the original language parameters.
This design targets a practical problem: heavy visual training can improve perception while damaging language capability, especially in small models. MoT gives the model more visual capacity without turning the compact model into a slow dense model. The paper reports that MoT converges faster than a dense transformer baseline and adds negligible inference overhead because decoding dominates runtime.
The attention pattern is also modality-specific. Visual tokens use more bidirectional/full attention because image patches do not have the left-to-right causal structure of language. Text tokens retain causal attention. This makes the vision branch more like a visual modeling module while preserving the LLM-style generation path.
Visual Latent Tokens
HY-Embodied-0.5 appends a learnable visual latent token after each image or video frame. The paper treats this token as a bridge between visual full attention and language causal attention. During pre-training, the latent token is supervised to match a global feature from the teacher ViT, so it is encouraged to carry image-level semantics instead of staying at local patch information.
The training objective combines three losses during large-scale pre-training:
\[ L_{\mathrm{total}} = L_{\mathrm{llm}} + L_{\mathrm{vision}} + L_{\mathrm{global}} \]
The visual branch predicts the teacher ViT’s next discrete visual code:
\[ L_{\mathrm{vision}} = -\frac{1}{N_v}\sum_{i=1}^{N_v}\log p_i(z_i) \]
The latent token is aligned to the teacher ViT global feature using negative cosine similarity:
\[ L_{\mathrm{global}} = - \frac{f_{\mathrm{latent}}^\top f_{\mathrm{teacher}}} {|f_{\mathrm{latent}}||f_{\mathrm{teacher}}|} \]
After pre-training and mid-training, the extra visual/global objectives are removed; later stages use standard autoregressive language loss. The design is useful because the model gets visual supervision when it is learning representations, then returns to a clean generation objective for reasoning and instruction following.
Data Recipe
The data construction is the other major contribution. The paper does not rely on generic image-text pairs alone. It mixes visual perception, embodied-centric, spatial-centric, and general understanding data.
The large-scale pre-training stage uses more than 600B tokens: 389B general understanding tokens and 236B embodied/perception tokens. Inside the embodied/perception portion, spatial and robotics data account for 43%, with the rest coming from visual perception data. The mid-training stage then uses about 30M higher-quality instances mixed at a 12:5:3 ratio across general understanding, embodied, and spatial data.
The visual perception data includes:
- 62M Omni-Detection samples for 2D/3D detection and object grounding.
- 36M depth-estimation samples for absolute and relative depth.
- 5M segmentation samples based on filtered high-quality masks.
- 11M pointing/counting samples to reduce enumeration errors and spatial hallucinations.
The embodied-centric data is organized into grounding, affordance, trajectory, understanding, planning, and reasoning. This structure matters. It turns “embodied data” from a vague label into a curriculum: locate the relevant thing, infer what can be done to it, predict or evaluate a path, understand task state, then plan the next actions.
The spatial-centric data targets correspondence, geometry, configuration, measurement, and dynamics. This is the part that makes the model more than a robot benchmark specialist. It trains capabilities like 2D-3D correspondence, metric object size estimation, relative direction, room area estimation, camera motion, and object movement.
Post-Training
The post-training pipeline is built around long-chain embodied reasoning. It starts with about 100K cold-start CoT instances, constructed through a human-model collaborative pipeline and filtered for reasoning quality, correctness, and repetition.
For RL, the paper avoids a fixed training set. Each RL round samples from a large candidate pool using the current model. Examples that are always solved are too easy; examples that always fail are too hard. The retained samples are those with partial success, which sit near the current capability frontier. Each RL stage uses 50K newly selected samples, balanced across perception, prediction, interaction, and planning.
The reward design is task-aware. Grounding uses IoU, Hungarian matching, point distance, or Chamfer distance. Trajectory tasks use DTW or Frechet-style path similarity. Counting and multiple-choice tasks use exact or partial matching. Continuous estimates get regression-style rewards. Free-form reasoning falls back to an LLM judge.
The RL objective is GRPO-style. For each input, the model samples a group of G = 16 responses, normalizes rewards inside the group, and updates with a clipped policy-ratio objective. The group-relative form matters because embodied tasks have different reward scales; a point-localization score and a planning score should not be treated as directly comparable raw numbers.
The paper then alternates RL with rejection-sampling fine-tuning. RL expands the frontier; rejection sampling selects successful high-quality reasoning traces and turns them into supervised data. In practice, around 1M candidate examples are filtered into about 300K high-quality traces for the next SFT stage.
On-Policy Distillation
The large MoE-A32B model is stronger, but deployment favors the compact MoT-2B model. HY-Embodied-0.5 therefore uses large-to-small on-policy distillation. The student first rolls out its own response:
\[ y \sim \pi_s(\cdot \mid x) \]
Then the teacher is evaluated on the same student-generated prefixes. The student minimizes token-level KL to the teacher:
\[ L_{\mathrm{OPD}} = \mathbb{E}{x,y\sim\pi_s} \left[ \frac{1}{|y|} \sum_t \mathrm{KL} \left( \pi_t(\cdot \mid x,y{<t}) | \pi_s(\cdot \mid x,y_{<t}) \right) \right] \]
This is more interesting than ordinary response imitation. Offline distillation trains the student on teacher trajectories; on-policy distillation trains the student on states it actually visits during decoding. For embodied reasoning, this is important because many errors happen in intermediate spatial or planning steps before the final answer appears.
Evaluation
HY-Embodied-0.5 is evaluated on 22 benchmarks covering visual perception, embodied understanding, and spatial understanding. The compact MoT-2B model reaches the best score on 16 of 22 benchmarks and second place on 4 more among compared sub-7B embodied/general VLMs. The paper reports an average score of 58.0%, outperforming Qwen3-VL-4B by 10.2 points and RoboBrain2.5-4B by 8.6 points in that evaluation setup.
The larger MoE-A32B model reaches 67.0 average over the same benchmark suite, ahead of Gemini 3.0 Pro at 63.6, Seed 2.0 at 66.2, Qwen 3.5 A17B at 66.1, and Kimi K2.5 at 61.1 in the paper’s reported comparison.
The most useful interpretation goes beyond “higher score.” The benchmark categories line up with the design choices: perception improvements support detection, depth, segmentation, and counting; spatial-centric data supports 3D and multi-view reasoning; embodied-centric data supports affordance, trajectory, and planning tasks. The reported gains are most meaningful where these capabilities are physically relevant.
Robot Control
The robot section extends HY-Embodied-0.5 MoT-2B into a VLA model by adding an action expert similar to the \(\pi_0\) / \(\pi_{0.5}\) design. The system first fine-tunes on 5K hours of UMI data. Because this stage uses UMI data, the model has not yet seen the specific robot embodiment used in later deployment.
The model is then SFT-trained on real-robot data for three tasks, with 300-700 demonstrations per task. The setup uses a dual-arm Xtrainer with head-mounted and wrist-mounted cameras. Evaluation uses 20 real-robot trials per task with randomized object poses.
The reported success rates are:
| Task | HY-Embodied-0.5 VLA | \(\pi_0\) | \(\pi_{0.5}\) |
|---|---|---|---|
| Precision Plug-in Packing | 85% | 80% | 85% |
| Tableware Stacking | 80% | 60% | 85% |
| Mug Hanging | 75% | 45% | 50% |
The mug hanging result is the clearest signal: it is the hardest of the three tasks and shows the largest margin. The paper’s claim is that the embodied VLM backbone, UMI pre-training, and MoT architecture transfer into manipulation after task-specific SFT.
Limitations
The paper is broad, so some details are inevitably compressed. The VLA section is convincing as a downstream signal, but it is shorter than the backbone sections and does not yet isolate every component of the robot stack. The reported robot tasks also use SFT with task-specific demonstrations; the paper does not establish zero-shot robot control.
The benchmark suite is large, yet embodied VLM evaluation remains unstable as a field. Some baselines are reported in best-of thinking/non-thinking mode, and the paper notes repetitive thinking problems for some Qwen3.5 variants. This makes exact leaderboard interpretation less important than the consistent pattern across perception, spatial, and embodied tasks.
Finally, HY-Embodied-0.5 is still mostly a VLM foundation model. Bridging language and action is explicitly listed as future work. The newer Hy-Embodied-0.5-VLA report appears to continue that direction, but this paper’s main contribution is the embodied perception/reasoning backbone.
Takeaways
HY-Embodied-0.5 is useful because it treats embodiment as a foundation-model design problem. The model architecture protects language ability while giving vision its own capacity; latent tokens summarize visual inputs for language reasoning; data construction targets physical-world competencies; post-training turns partial successes into stronger reasoning traces; and on-policy distillation transfers the large model’s behavior into a deployable compact model.
For robotics, the reusable idea is clear: build the VLA stack on a backbone that already understands space, affordance, trajectory, and planning. The action head matters, but the “brain” under it matters too.