[Paper Notes] BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning (arXiv 2025)
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
BFM-Zero is a humanoid-control foundation model built with off-policy unsupervised RL (instead of standard PPO-style task-specific training). It learns a shared latent task space that can be prompted for:
- zero-shot motion tracking
- zero-shot goal reaching
- zero-shot reward optimization
- few-shot latent-space adaptation (without finetuning network weights)
The paper’s main contribution is not just a single algorithmic trick, but a full sim-to-real recipe combining:
- Forward-Backward (FB) unsupervised RL representations
- motion-data regularization (FB-CPR lineage)
- asymmetric history-based learning
- domain randomization
- safety/feasibility reward regularization
They demonstrate the system on a real Unitree G1 humanoid, including recovery from large perturbations and promptable behavior composition.
Paper Info
- Title: BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning
- Authors: Yitang Li, Zhengyi Luo, Tonghe Zhang, et al.
- Affiliations: Carnegie Mellon University, Meta
- Venue: arXiv preprint (submitted 2025-11-06)
- arXiv: 2511.04131
- Project page (paper first page): BFM-Zero Website
1. Motivation and Problem Setting
The paper targets a core gap in humanoid control:
- Many strong humanoid systems are task-specific (especially tracking).
- Many rely on on-policy RL (e.g., PPO) with explicit rewards.
- It is hard to get large-scale humanoid action-label datasets for pure behavior cloning.
The authors ask whether off-policy unsupervised RL can train a reusable, promptable behavioral foundation model for humanoids that supports multiple downstream tasks without retraining.
They formulate real-world humanoid control as a POMDP and train in simulation with:
- privileged state available in sim
- partial observations for the actor (plus history)
- a motion dataset of unlabeled trajectories used as behavioral regularization
2. Core Idea: A Promptable Behavioral Foundation Model
BFM-Zero learns:
- a shared latent task space
Z - a promptable policy conditioned on latent vector
z
Different task types are mapped into the same latent space:
- Goal reaching: prompt with a latent derived from a target state
- Reward optimization: infer a latent from reward-weighted state embeddings
- Tracking: generate a sequence of latent prompts from a motion trajectory
This gives a unified interface for humanoid behaviors instead of training separate policies per task.
3. Method Overview
3.1 Built on Forward-Backward (FB) Unsupervised RL
The method builds on Forward-Backward representations and FB-CPR:
B(s)encodes states into a latent/task-related representationF(s, a, z)behaves like a latent-conditioned successor-feature-style quantityzdefines a task/objective in latent space
The paper emphasizes that this latent space is structured enough to support:
- explainable prompting
- zero-shot task execution
- interpolation between skills
- few-shot optimization in latent space
3.2 Key Sim-to-Real Design Choices (Most Important in Practice)
The paper highlights several design choices needed to make unsupervised RL work on real humanoids:
- Asymmetric learning: actor uses observation history, critics use privileged information
- Domain randomization: masses, friction, offsets, perturbations, sensor noise
- Reward regularization: auxiliary penalties for safety / physically feasible behavior
- Large-scale off-policy training: many parallel environments, replay buffer, high UTD ratio
This is the strongest practical lesson from the paper: the success comes from the combination of representation learning and systems-level sim-to-real engineering.
3.3 Training Objective (High Level)
BFM-Zero combines multiple components during training:
- an FB objective for learning long-horizon latent dynamics / representations
- an auxiliary critic for safety and physical constraints
- a discriminator/style critic to bias behaviors toward motion-data realism
- a policy objective that balances task value, style, and regularization terms
The result is a policy that is both:
- broad enough to support different prompts
- constrained enough to remain stable and human-like on hardware
4. Zero-Shot Inference Modes (Why This Paper Is Interesting)
The same pre-trained model supports multiple downstream uses:
4.1 Zero-shot Tracking
Given a motion trajectory, BFM-Zero derives a sequence of latent prompts and tracks the motion without retraining.
4.2 Zero-shot Goal Reaching
A target pose/state is embedded into the latent space and used as a prompt. The paper shows smooth transitions and reasonable behavior even for difficult or partially infeasible targets.
4.3 Zero-shot Reward Optimization
A reward function can be converted into a latent prompt via replay-buffer samples and state embeddings. This enables:
- locomotion commands
- arm-raise commands
- crouching / sitting-like behaviors
- combined rewards (composed skills)
This is an unusually clean interface: the same model can be prompted by rewards, goals, or motions.
5. Experiments and Results
5.1 Training / Setup
- Robot: Unitree G1 humanoid
- Simulation: IsaacLab for training (paper reports simulation at
200 Hz, control at50 Hz) - Behavior dataset: retargeted LAFAN1 motions (40 several-minute motions)
- Also evaluated: Mujoco sim transfer and a Booster T1 humanoid in appendix
5.2 Simulation Validation (Zero-shot)
The paper evaluates:
- tracking
- reward optimization
- pose/goal reaching
Key reported observations:
- Domain-randomized deployable BFM-Zero performs somewhat worse than a privileged no-DR version, but remains strong.
- The paper reports drops of about 2.47% / 25.86% / 10.65% across tracking / reward / pose-reaching compared with the idealized privileged setting.
- Sim-to-sim transfer to Mujoco shows relatively small degradation (reported variations under ~7%).
- The model also generalizes to out-of-distribution AMASS motions/poses (evaluated in Mujoco).
5.3 Real-Robot Results (Main Highlight)
The real-robot section demonstrates:
- tracking of diverse motions (including dynamic behaviors)
- goal reaching with smooth transitions
- reward optimization for locomotion / posture / arm control
- disturbance rejection and recovery (pushes, kicks, being dragged/falling)
A particularly notable claim is that recovery looks natural/human-like rather than brittle or overly aggressive.
5.4 Few-Shot Adaptation Without Finetuning Weights
The paper adapts behavior by optimizing in latent prompt space:
- Single-pose adaptation: with a 4 kg payload, optimized latent prompt improves single-leg standing from failure (<5s collision) to >15s balance.
- Trajectory adaptation: under friction shift, latent-sequence optimization improves tracking error by about 29.1%.
This is a strong demonstration of prompt-level adaptation for control.
6. Latent Space Structure (Interpretability / Compositionality)
The authors visualize the latent space (t-SNE) and show:
- semantically similar motions cluster together
- different task types occupy structured regions
- interpolating latent vectors yields meaningful intermediate behaviors
This supports their “promptable BFM” framing: the latent space is not just a hidden internal variable, but a usable interface.
7. Strengths
- Clear and ambitious objective: a promptable humanoid behavioral foundation model
- Strong real-world focus, not simulation-only
- Unified interface across tracking / goals / rewards
- Practical sim-to-real recipe (asymmetric learning + DR + regularization)
- Compelling prompt-space adaptation results without network finetuning
- Good qualitative emphasis on robustness and natural recovery
8. Limitations / Open Questions
The paper explicitly notes several limitations (Discussion):
- Behavior scope depends on the training motion data distribution
- More work is needed on scaling laws (data size / architecture / performance)
- Sim-to-real gap is reduced but not solved; stronger online adaptation may be needed
- Fast adaptation and finetuning are only preliminarily explored
My additional practical questions:
- How robust is reward-to-latent inference when replay-buffer samples shift under stronger domain randomization?
- How much of the real-world quality comes from the discriminator/style prior vs. the FB latent structure itself?
- What is the failure boundary for more contact-rich manipulation-like humanoid tasks?
9. Takeaways for Robotics Research
- Off-policy unsupervised RL for real humanoids is more viable than many people assume.
- A reusable humanoid controller may benefit from a latent prompt interface instead of task-specific policies.
- Sim-to-real success here is heavily driven by engineering choices, not only the base algorithm.
- Prompt-space optimization is a promising middle ground between zero-shot execution and full policy finetuning.
10. Notes for Future Reading
If I revisit this paper, I would look more closely at:
- Appendix details on architecture/data-size scaling
- reward inference stability and dataset choice
- how this compares empirically to newer humanoid RL pipelines and VLA-style humanoid systems