[Paper Notes] Motus2: A Self-Evolving General World Model for Dexterous Manipulation
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
Motus2 turns a shared video–action backbone into three control interfaces: a policy proposes action chunks, a simulator predicts their visual consequences, and an evaluator estimates task progress. Best-of-N planning uses the evaluator to choose an action at test time; model-based reinforcement learning (MBRL) feeds the same signal back into the policy. The cleanest idea is its supervision routing: successful demonstrations teach actions, while failed and suboptimal trajectories teach dynamics and value without becoming imitation targets.
The strongest empirical result concerns representation and domain transfer. Egocentric pretraining raises average real-robot success from 0% to 51%, and robot-domain mid-training raises it further to 84% across five dexterous tasks. Evidence for self-evolution is promising but smaller: MBRL improves the average from 65.0% to 72.5% on two tasks, while Best-of-N planning contributes another 2.5 points. With 20 trials per task, that planning gain corresponds to one additional success across 40 rollouts.
My read is that Motus2 is a strong systems paper about closing the loop among action generation, prediction, and evaluation. Its data curriculum and causal interface design are more convincing today than the broad claim of autonomous self-evolution.
Paper Info
- Title: Motus2: A Self-Evolving General World Model for Dexterous Manipulation
- Authors: Hongzhe Bi, Zihao Zhou, Yihang Tang, Jingrui Pang, Shuhe Huang, Haitian Liu, Runqing Wang, Shuai Huang, Yichen Wang, Yiming Cheng, Ruowen Zhao, Zhenghua Li, Hengkai Tan, Xiaolong Liu, Jinhui Wan, Jiabao Liu, Min Zhao, Fan Bao, Jun Zhu
- Affiliations: GensPI; Tsinghua University; Beihang University; Beijing Institute of Technology
- Date: 2026-08-31
- Venue: arXiv preprint
- Links: arXiv:2608.30237 · Project page
At the time of writing, the project page lists the code and model as coming soon.
1. The Problem: A Policy Has No Built-In Critic
Large behavior-cloning policies learn a mapping from observations and language to actions. This works well when demonstrations are abundant and execution remains close to the training distribution. It leaves two structural gaps. First, the policy does not explicitly predict what its candidate action will cause. Second, imitation provides no internal criterion for deciding whether a predicted outcome advances the task.
Dexterous manipulation makes these gaps especially costly. Hands occlude the object, contact state is visually ambiguous, and small timing or pose errors can change the grasp mode. Robot demonstrations with aligned vision, arm motion, finger motion, and tactile sensing are also expensive. Motus2 addresses these constraints along two axes:
- model scaling: connect policy, simulation, and evaluation inside a shared world model;
- data scaling: learn broad interaction priors from human egocentric video, then ground them in robot trajectories.
The high-level loop is:
flowchart TD
C["Language + real observation history"] --> P["Policy: propose action chunks"]
P --> S["Simulator: predict future visual states"]
S --> V["Evaluator: estimate task progress"]
V --> Q{"Use the score"}
Q -->|"Test time"| B["Select best candidate and execute"]
Q -->|"Post-training"| U["Update the action distribution"]
B --> R["Append the next real observation"]
R --> C
2. One Backbone, Three Conditional Interfaces
Let (c_t) contain the language instruction, current proprioception, and selected visual history. Let (A_t) be an executable action chunk, (Z_t) its latent future observations, and (Y_t) a discretized task-progress value. Motus2 factorizes their joint distribution in action-first order:
[ p_\theta(A_t,Z_t,Y_t\mid c_t) =\underbrace{\pi_\theta(A_t\mid c_t)}{\text{policy}} \underbrace{p^{\mathrm{wm}}\theta(Z_t\mid c_t,A_t)}{\text{simulator}} \underbrace{p^{\mathrm{vm}}\theta(Y_t\mid c_t,A_t,Z_t)}_{\text{evaluator}}. ]
The evaluator scores a branch using the expected value of its categorical progress distribution:
[ V_\theta(c_t,A_t,Z_t) =\mathbb{E}{Y_t\sim p^{\mathrm{vm}}\theta}[Y_t]. ]
These are conditional modes of a shared video–action transformer initialized from Wan 2.2-TI2V-5B. Ordinary control queries only the action factor, so the robot does not generate a video rollout at every control step. Planning and MBRL activate the full policy–simulator–evaluator chain.
Action-first masking prevents future leakage
Joint human video–action pretraining allows bidirectional interaction between video and action tokens within a chunk. Robot-domain mid-training switches to an action-first mask. Inside each chunk:
[ A_j \rightarrow Z_j \rightarrow U_j, ]
where (U_j) is a read-only value query. Action tokens cannot read the corresponding future-video or value tokens. Future-video tokens can condition on the action, and the value query can read both. Causality is also preserved across chunks.
This mask matters because a jointly trained model could otherwise predict an action after observing the future that the action is supposed to cause. The action-first layout turns the shared transformer into a valid causal interface for deployment.
Route each trajectory to the supervision it deserves
The model predicts flow-matching velocity fields for video and action plus a categorical value distribution. Loss gates select among three modes:
| Mode | Action target | Future-video target | Value target | Appropriate data |
|---|---|---|---|---|
| Policy | Yes | Yes | No | Curated successful demonstrations |
| Simulation | No; action is clean context | Yes | No | Successful, failed, suboptimal, or task-irrelevant transitions |
| Evaluation | No | No | Yes | Segments with progress labels |
This is one of the paper’s most reusable ideas. A failed trajectory still contains a valid physical transition: given the executed action, the observed consequence really happened. It also supplies negative evidence about task progress. Motus2 extracts both signals while preventing the policy from imitating the failed action.
3. The Human-to-Robot Data Curriculum
Motus2 reports approximately 130,000 raw hours of egocentric data before filtering, segmentation, and annotation:
| Data group | Approximate raw hours | Purpose |
|---|---|---|
| Monocular egocentric data | 112,500 | Broad visual, semantic, and interaction coverage |
| Stereo egocentric data | 17,400 | Binocular geometry and more accurate 3D hand poses |
| Robot + human–robot alignment data | More than 100 | Embodiment grounding during mid-training |
Training proceeds in three stages. Stage 1 trains the video pathway for 500K low-resolution and 340K high-resolution monocular steps. Stage 2 performs 450K steps of synchronized stereo video–action pretraining. Mid-training then introduces robot trajectories, alignment data, the action-first mask, and the policy/simulation/evaluation mixture.
Human observations are standardized into a 134-dimensional bimanual representation containing wrist poses and 20 non-wrist 3D keypoints per hand. An inverse-kinematics retargeter maps MediaPipe-format human keypoints into the 20-DoF Wuji hand space for pretraining. Target-robot post-training uses each robot hand’s native joint angles.
The paper also fits a stereo data-scaling trend using nested 2K, 4K, 10K, and 20K-hour subsets:
[ \mathcal{L}^{*}_{\mathrm{val}}(D)\approx 0.101-0.005\ln D. ]
Larger subsets monotonically reduce held-out human-action MSE. This is useful directional evidence, although four points and an action-prediction metric are not enough to establish a general robot-control scaling law. The table of named stereo sources sums to about 17.4K hours while the scaling experiment reaches 20K; the difference may come from rounding, overlap, or a distinct subset accounting rule, but the paper does not spell it out.
4. What “Self-Evolving” Means Here
Progress-based value learning
For a segment of length (\Delta t) beginning at time (t) in a successful trajectory of length (T), Motus2 uses the relative-progress target
[ r_t=\frac{\Delta t}{T-t}. ]
Failed and task-irrelevant segments receive the negative counterpart:
[ r_t=-\frac{\Delta t}{T-t}. ]
The values are discretized into 201 bins. This gives the evaluator a local measure of whether an action segment advances or obstructs the task.
The labeling assumption is intentionally simple. Every segment from a successful trajectory receives positive progress, while failed trajectories receive negative progress. Real failures can contain useful intermediate actions, and successful runs can contain inefficient corrections, so these labels are a coarse proxy for state-dependent value.
Best-of-N planning
At inference, the policy samples (N) action chunks. The simulator predicts one future branch for each candidate, and the evaluator ranks those branches. The robot executes the highest-valued chunk, observes the real next state, and replans. This receding-horizon design keeps imagined rollouts short and regularly anchors the model to real observations.
MBRL with DiffusionNFT
Planning changes which proposal is selected; MBRL changes the distribution that produces proposals. For each offline trajectory prefix, the training pipeline generates eight policy candidates plus one ground-truth action–future pair. Candidate scores are normalized within the group into weights (\hat r_i\in[0,1]). High-value samples move the online flow field toward their actions, low-value samples move it away, and the ground-truth pair acts as a positive anchor. An EMA reference policy limits drift.
Only action-related parameters are updated during this phase; the video backbone and evaluator remain frozen. The imagined horizon is one action chunk. The authors implement the rollout, value scoring, reference construction, and optimization stages as an asynchronous Ray pipeline.
This supports a precise interpretation of self-evolution: Motus2 improves its action distribution using consequences and values generated by its own frozen simulator–evaluator pair. It does not yet demonstrate an open-ended robot loop that autonomously collects new real-world experience, verifies model errors, updates all components, and continues across tasks. Model bias can therefore reinforce policy bias, especially beyond the short rollout horizon.
5. Memory and Touch Address Two Forms of Partial Observability
The default policy uses a bounded sliding-window KV cache. It stores recent real observations, evicts old ones, and rebases temporal RoPE coordinates. This keeps memory and per-step attention bounded, but early evidence disappears.
Two long-context variants explore the trade-off:
- Global autoregression retains every previous clean visual latent. It preserves full history while cache size and attention cost grow with episode length.
- Hybrid memory retains initial anchor frames and recent frames at full resolution, then compresses older observations into persistent memory tokens.
For contact, a separate lightweight tactile expert refines short action sub-chunks at 30 Hz. The 5B backbone denoises a 48-action chunk to an intermediate noise level once. The expert reuses its detached layer-wise KV cache and applies the latest 90 Hz tactile window immediately before each six-action sub-chunk executes. An auxiliary future-force prediction loss regularizes contact evolution during training; deployment outputs only refined actions.
The tactile module is an efficient late-fusion design. It also qualifies the strongest version of the “single shared model” claim: policy, simulator, and evaluator share the main backbone, while high-rate touch is handled by an additional 30-layer, width-128 transformer expert.
6. Experiments
Human priors and robot-domain grounding provide the largest gain
Each real-robot task uses 20 rollouts under matched task-specific SFT data and observation interfaces.
| Method / initialization | Place Ball | Multi-Finger | Attach Eraser | Screw Bulb | Put Phone | Average |
|---|---|---|---|---|---|---|
| (\pi_{0.5}) | 0% | 0% | 0% | 0% | 0% | 0% |
| Wan-SFT | 0% | 0% | 0% | 0% | 0% | 0% |
| Egocentric Pretrain-SFT | 60% | 35% | 90% | 55% | 15% | 51% |
| Motus2 Midtrain-SFT | 100% | 70% | 100% | 90% | 60% | 84% |
The controlled Wan → egocentric pretraining → robot mid-training comparison is the paper’s strongest evidence. The (\pi_{0.5}) result is harder to interpret as a general model comparison: zero success across all five high-DoF tasks may expose a severe embodiment or action-interface mismatch, even though target-task data and observations are matched.
MBRL helps more than test-time planning in the reported study
| Method | Put Phone | Multi-Finger | Average |
|---|---|---|---|
| Motus2 | 60% | 70% | 65.0% |
| + Planning | 65% | 70% | 67.5% |
| + MBRL | 65% | 80% | 72.5% |
| + MBRL + Planning | 70% | 80% | 75.0% |
Across 40 rollouts, planning adds one success, MBRL adds three, and their combination adds four relative to the base policy. The direction is consistent with the proposed loop. More tasks, independent training runs, confidence intervals, and tests under distribution shift would be needed to establish a robust gain.
Full history beats compressed memory
Global autoregression reaches 78% average success in simulation and 57.5% on the real robot for Find Square and Press Button. Hybrid memory reaches 52% and 25%, respectively. This is a substantial gap. It says that the evaluated compression mechanism loses task-critical evidence; it does not yet show that full-history attention is the scalable final answer.
Tactile refinement improves contact-rich tasks
On Pull Out Paper Cup and Tear Paper, tactile refinement raises average success from 60.0% to 72.5%. The gain is 10 points on cup extraction and 15 points on paper tearing. These two tasks support the role of touch in contact-sensitive control, while broader evaluation across objects, sensors, and embodiments remains open.
7. Strengths and Limitations
Strengths
The architecture gives policy, simulation, and evaluation a coherent causal order inside the same backbone. Supervision routing makes practical use of failed interactions without teaching failed behavior. The data curriculum provides a concrete path from abundant human observation to scarce embodiment-specific control. The paper also studies memory and touch as first-class sources of partial observability instead of treating vision-only action prediction as the whole problem.
Limitations
- Real-robot evaluation uses only 20 trials per task and reports no confidence intervals or independent training seeds.
- MBRL and planning are evaluated on two tasks with a one-chunk imagined horizon.
- The value labels equate success-trajectory segments with positive progress and failure-trajectory segments with negative progress, which misses recoverable mistakes and useful sub-behaviors inside failures.
- The simulator and evaluator are frozen during policy optimization, so their errors may become increasingly off-policy as the action distribution changes.
- The stereo scaling result uses four data points and human-action MSE, with no direct link to downstream success scaling.
- Full-history autoregression performs well but grows in cost with episode length.
- Tactile transfer is limited by sensor noise and the morphological gap between human and robot hands. A glove deforms even without external contact, and different hand geometries require embodiment-specific sensing hardware.
- Code and checkpoints are not yet available, leaving inference speed, compute cost, and independent reproducibility unresolved.
8. Takeaway
Motus2 offers a compelling blueprint for a robot world model with three usable interfaces:
- learn interaction priors from large human video–action data;
- impose an action-first causal structure during robot grounding;
- preserve failed experience as dynamics and value supervision;
- use simulated consequences for candidate selection and action-policy updates;
- add explicit memory and touch when current vision cannot recover the physical state.
The immediate lesson is broader than the name “self-evolving.” World models become useful for control when action generation, consequence prediction, and outcome evaluation share a compatible representation and training interface. Motus2 demonstrates that integration clearly. The next decisive step is a longer autonomous loop where real interaction corrects the simulator and evaluator as the policy evolves.