[Paper Notes] SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model
Published:
Three hours, including learning the dynamics
SuperMimic reports surpassing SONIC’s released motion-tracking checkpoint after three hours on one H100. The question that made me look more closely was what those three hours purchase: a policy update on top of an already trained world model, or the whole learning process? The reported time covers simulator data collection, world-model fitting, and policy/critic training in one alternating loop. Checkpoint evaluations are performed afterward and excluded. This is the training-loop budget; it should not be read as a measurement of every step of preparing a new robot and motion dataset. §4.1
The next question is more interesting. If a simulator still supplies the dynamics data, why can learning a second model save so much computation? SuperMimic combines cheaper predicted trajectories with gradients through those trajectories, while sharing the learned dynamics across the motion library.
These notes follow SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model, by Qiayuan Liao, Yiyang Shao, Junfeng Long, Kevin Zakka, Roberto Horowitz, Pieter Abbeel, and Koushil Sreenath, UC Berkeley. arXiv:2610.28480v1 was submitted on October 20, 2026 and lists acceptance as a CoRL 2026 Oral. Sources are the paper and official project page; the measurements below are author-reported.
Two kinds of motion data do different jobs
The reference library specifies what the robot should do. SuperMimic uses G1-retargeted BONES-SEED motions, filtered to remove unsuitable terrain/object interactions and infeasible trajectories. The retained set contains 129,514 clips totaling 264.56 hours. A reference supplies the policy’s goal and the target used by the tracking reward. §4.1; Appendix E
World-model supervision comes from what happens when the current policy acts in MuJoCo Warp. The simulator executes a joint-target command, with exploration noise during collection, and records the resulting state transition. A useful shorthand for one dynamics training example is $(s_t,a_t^{\mathrm{exec}},\xi,s_{t+1})$, where $\xi$ contains episode parameters such as friction, joint calibration offsets, and torso center-of-mass offset. Reward, observation, and termination information support the other model components.
This distinction matters. A reference pose sequence alone does not specify which control command produced each transition. The collected data supplies that action–consequence relationship, including imperfect tracking and failures. Earlier policies’ transitions remain usable because the body’s dynamics do not depend on which policy collected them. The replay buffer retains recent experience as the controller improves. §3.1–3.2
Here “world model” means a surrogate of the robot’s state dynamics and task-relevant quantities. The dynamics network is an MLP with three hidden layers of width 2048; the task is grounded in simulator states and controls. Appendix C
What alternates, and what stays differentiable
Each training iteration has the following sequence:
- Run the current policy in simulation and add transitions to the replay buffer.
- Fit the world model on eight-step replay windows, feeding its predicted state into the next prediction while using recorded actions.
- Start imagined rollouts from replay states, hold the world-model parameters fixed, and improve the policy through 32 predicted steps. Fit the critic using the imagined transitions.
The next iteration updates the world model again. Freezing its parameters during the policy update still allows derivatives through its state and action inputs. Otherwise the policy would lose the very gradient the method needs. The physical simulator supplies training targets without participating in backpropagation. §3; Algorithm A.1
Writing the imagined dynamics as
\[\hat s_{t+1}=f_\omega(\hat s_t,a_t,\xi),\qquad a_t=\pi_\theta(\hat{\bar o}_t+\eta_t,g_t),\]the model provides a predicted noise-free observation $\hat{\bar o}_t$, and sensor noise $\eta_t$ is added for policy training. The policy maximizes the paper’s short-horizon objective:
\[\hat J(\theta)=\mathbb{E}\left[ \sum_{t=0}^{H-1}\gamma^t\hat c_t\hat r_{t+1} +\gamma^H\hat c_H V_\psi(\hat s_H,g_H,\xi) \right],\qquad H=32.\]The survival weight follows $\hat c_0=1$ and $\hat c_{t+1}=\hat c_t(1-\hat d_{t+1})$, using the predicted termination probability. This makes termination affect the objective smoothly. The critic supplies value beyond the rollout horizon. At the 50 Hz control rate, 32 steps cover 0.64 seconds: the model handles a short explicit trajectory, and the critic represents its longer-term consequences. Eq. 2; §3.3
Where the speed comes from
A cheaper forward calculation. The simulator resolves rigid-body dynamics and contact at each step. A learned network approximates the resulting transition with batched neural computation. On an H100, the paper measures more than 10× higher rollout throughput at the 2k–16k batch sizes used in practice. The comparison counts one 50 Hz control step, implemented by MuJoCo Warp as four 5 ms substeps; the world model uses TF32. This is a rollout-throughput result, with model fitting and policy backpropagation accounted for separately in the full training time. Figure 5(a)
A more informative policy update. PPO estimates improvement from sampled actions and their advantages while treating the transition function as a black box. A differentiable dynamics model supplies local derivatives of the predicted next state with respect to the action. Backpropagation through time connects policy parameters to actions, predicted states, and return. For intuition, the model can indicate how a small change to a joint command would affect a later tracking error. That direction is an estimate under the learned dynamics.
The paper isolates this second contribution with PPO-WM, which also learns from world-model rollouts. On LAFAN1 walk/run/sprint motions, SuperMimic reaches PPO-WM’s best training return with approximately 64× fewer imagined steps. This is the result I find most useful for understanding the method: both optimizers already have access to inexpensive imagined experience, yet their learning efficiency differs substantially. Its scope is this mechanism ablation, and its metric is imagined steps to a specified training return. It is not a measured 64× reduction in simulator interactions across the full BONES-SEED dataset. Figure 5(b)
The 10× throughput result and 64× step-efficiency result have different denominators and experimental settings. Multiplying them would not establish the total training speedup.
Shared dynamics explain why this can remain useful as the library grows. Walking, rolling, and jumping expose different parts of the same body’s dynamics. Across the authors’ 1–264-hour data sweep, the normalized eight-step state prediction error stays within 0.007–0.011 while compute to a target return grows slowly. These errors are measured online on policy rollouts, so the evidence concerns the encountered training distribution. My reading is that the cost of learning the body’s response can be reused across many motion goals; the sweep does not establish unrestricted extrapolation to new contacts or embodiments. Figure 3
The results and their measurement boundaries
| Controller | Reported training hardware and time | Failed motions out of 129,514 |
|---|---|---|
| SONIC released checkpoint | 128 L40 GPUs, 7 days | 179 |
| SuperMimic | 1 H100 GPU, 3 hours | 25 |
| SuperMimic | 2 H100 GPUs, 9.8 hours | 0 |
The three-hour run also exceeds SONIC’s normalized return. SONIC’s reported budget is about 21,000 GPU-hours; the corresponding SuperMimic checkpoint costs three GPU-hours. Hardware differs, and SONIC trained on 611 hours of motion compared with SuperMimic’s 264.56 hours. These are published pipeline costs, not a controlled same-hardware measure of algorithmic efficiency. §4.1
A separate LAFAN1 comparison matches motions, observations, rewards, termination, domain randomization, network sizes, and evaluation protocol. SuperMimic reaches an evaluation return of 0.82 in 1.2 hours on one GPU. PPO-Sim does not reach that value within 18 hours, including its eight-GPU configuration, whose final return is 0.72. This supports the efficiency claim under a more closely matched setup, while remaining a comparison against the specified PPO recipe. Figure 4(a)
The large benchmark evaluates each motion once from its first frame, with deterministic physics, foot friction 1.0, and no observation noise or domain randomization. Failure uses SuperMimic’s termination rule. The evaluated motions belong to both methods’ training sets, so zero failures measures training-motion coverage. It does not mean zero tracking error or success on unseen motions. The real G1 demonstrations separately establish transfer of one tracker across motions including a side roll and flips. Appendix F; §4.4
The approximation that still needs checking
A model can predict states reasonably well while giving imperfect local derivatives. Differentiability makes an update available; its usefulness depends on the learned approximation in the states the policy visits. Continued simulator collection helps keep that approximation current, and short rollouts limit how far the policy differentiates through accumulated prediction error.
The critic has a measurable role here. In the reported horizon sweep, training without it fails at horizons of 16 steps or fewer. With a critic, 16 steps match the critic-free 32-step setting, while the default 32-step setting reaches the target return fastest. Extending the rollout to 128 steps increases cost and lowers final return in that sweep. More imagined future is therefore not automatically better. §4.3; Figure 4
I would prioritize this approach for a controller with a dense tracking reward, a shared body, and many reference motions. For manipulation or uneven terrain, my first experiment would test model prediction and policy improvement around the new contacts under a matched compute budget. The paper learns dynamics entirely from simulation and trains on flat ground; terrain and object interaction remain outside its demonstrated scope. That is where the proposed reuse of dynamics still needs evidence. §6