[Paper Notes] Robust and Generalized Humanoid Motion Tracking
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
RGMT asks a practical question about humanoid motion tracking: what should a controller do when its reference motion is noisy, contact-inconsistent, or temporarily incompatible with the robot’s current dynamics? Its answer is dynamics-conditioned command aggregation. A causal Transformer summarizes ten recent proprioceptive observations into a dynamics embedding. That embedding becomes the query of a cross-attention module over a 21-frame local reference window, allowing the policy to emphasize command tokens that fit the current closed-loop state and suppress unreliable ones.
The same PPO policy also learns fall recovery. Fifteen percent of training environments begin from randomized unstable poses, receive an early upward-assistance curriculum, and remain alive for a three-second recovery window. This exposes the tracker to a wider contact-state distribution and substantially improves crawling, kneeling, sitting, and breakdance-style tracking.
The complete system uses about 3.5 hours of curated motion data, trains end to end without teacher-student distillation, and deploys on a 29-DoF Unitree G1. In the paper’s MuJoCo evaluation, success reaches 98.3% on MoCap motions, 94.6% on video-derived motions, and 90.1% on ground-interaction motions, compared with GMT’s 84.6%, 72.4%, and 48.9%. The core contribution is therefore a control interface that treats a reference trajectory as uncertain contextual evidence, interpreted through the robot’s recent physical state.
Paper Info
“Robust and Generalized Humanoid Motion Tracking” is by Yubiao Ma, Han Yu, Jiayin Xie, Changtai Lv, Qiang Luo, Chi Zhang, Yunpeng Yin, Boyang Xing, Xuemei Ren, and Dongdong Zheng, with affiliations at Beijing Institute of Technology and Humanoid Robotics (Shanghai) Co., Ltd. This note covers arXiv:2601.23080v1, posted on January 30, 2026. The project page contains hardware demonstrations. The current release is an eight-page preprint.
RGMT is a separate work from GMT, created by a different author team. GMT is a baseline in this paper, not an earlier version maintained by the RGMT authors.
1. The Reference Is an Imperfect Command
General humanoid tracking systems often train as though every reference frame were equally trustworthy. Real inputs violate that assumption. Retargeted MoCap may contain foot sliding, penetration, or inconsistent contacts. Video reconstruction adds pose error and temporal jitter. Online teleoperation adds drift, latency, and operator inconsistency. Motion matching can introduce abrupt transitions when switching between clips.
These local defects become dangerous in closed loop. A command that is only slightly wrong kinematically can demand an impossible contact transition. The robot deviates, the next command becomes less compatible with its actual state, and error compounds into a fall. RGMT therefore changes the controller’s question from “How closely can I reproduce this frame?” to “Which parts of the nearby reference remain useful from my current dynamical state?”
flowchart TD
A["Recent proprioceptive history"] --> B["Causal history encoder"]
B --> C["Dynamics embedding as query"]
D["Past, current, and future reference tokens"] --> E["Command token encoder"]
C --> F["Dynamics-conditioned cross-attention"]
E --> F
F --> G["Command embedding"]
H["Current observation"] --> I["Actor"]
G --> I
I --> J["Residual joint target"]
J --> K["PD control on Unitree G1"]
This architecture gives the policy a learned mechanism for phase correction and command filtering. The paper describes the aggregation as operating under physical-feasibility constraints. Those constraints are implicit in training experience and closed-loop state; the attention layer does not solve an explicit dynamics optimization problem.
2. Control Formulation
The deployable observation combines projected gravity, base angular velocity, joint state, and the previous action:
[ o_t=\left[g_t^{\mathrm{proj}},\;\omega_t,\;q_t-q_0,\;\dot q_t,\;a_{t-1}\right]. ]
The reference command at each step is
[ g_t=\left[v_t^{\mathrm{ref}},\;\omega_t^{\mathrm{ref}},\; g_t^{\mathrm{ref}},\;q_t^{\mathrm{ref}}\right]\in\mathbb{R}^{38}, ]
containing body-frame base linear and angular velocity, reference gravity direction, and 29 reference joint positions. The actor receives noisy deployable observations. An asymmetric critic additionally sees reference root height, body-link poses, and base linear velocity during training.
The actor predicts a residual joint-position action (a_t\in\mathbb{R}^{29}). The low-level target is anchored at the reference pose:
[ q_t^{\mathrm{tar}}=q_t^{\mathrm{ref}}+a_t, ]
and a joint-space PD controller produces torque:
[ \tau_t=K_p(q_t^{\mathrm{tar}}-q_t)-K_d\dot q_t. ]
The residual action gives the reference pose a strong prior while leaving room for balance and contact corrections. It also narrows exploration to offsets around a meaningful command.
3. Dynamics-Conditioned Command Aggregation
Encoding recent dynamics
RGMT uses the most recent ten proprioceptive observations. Each 93-dimensional observation is projected to a 128-dimensional token, receives sinusoidal position encoding, and passes through a lightweight causal Transformer. A causal mask ensures that each temporal token uses only current and earlier observations. Element-wise max pooling produces the dynamics embedding (h_t):
[ h_t[j]=\max_{\tau\in{t-K,\ldots,t}}\bar H_\tau[j], \qquad K=9. ]
This history supplies information absent from one instantaneous state: recent control response, phase lag, instability growth, and the direction in which the body is already moving.
Querying a local command window
The reference window includes ten commands before and ten commands after the current index:
[ g_{t-L:t+L}=\left[g_{t-L},\ldots,g_t,\ldots,g_{t+L}\right], \qquad L=10. ]
The history embedding becomes a query, while encoded command tokens provide keys and values:
[ q_t=\mathrm{MLP}_{\mathrm{dyn}}(h_t), ]
[ \widetilde Z =\mathrm{MLP}{\mathrm{cmd}}(g{t-L:t+L})+P^{\mathrm{cmd}}, ]
[ u_t=\mathrm{CrossAttn}(Q=q_t,\;K=\widetilde Z,\;V=\widetilde Z). ]
The resulting (u_t) is fused with the current observation and passed to the actor. Because the query depends on recent robot dynamics, attention weights can shift across the command window as the physical execution advances, lags, or encounters an inconsistent reference segment.
The comparison with GMT is revealing. GMT encodes an approximately two-second future motion window with a CNN and uses a Motion Mixture-of-Experts to increase model capacity. RGMT focuses on selective command interpretation: its reference aggregation explicitly depends on the robot’s recent execution history.
4. Compact Data and Single-Stage Training
The training corpus comes from LAFAN1 and a selected AMASS subset, retargeted with General Motion Retargeting. The authors remove infeasible, redundant, and low-quality sequences, leaving about 3.5 hours of motion. Their claim is qualitative but important: diverse clean supervision can be more useful than a much larger corpus containing duplicated or physically inconsistent segments.
The policy is trained with PPO in Isaac Gym using 5,680 parallel environments on one RTX 4090. Tracking rewards cover keypoint alignment, relative pose consistency, and keypoint velocity. Regularizers penalize rapid action changes, joint-limit violations, and undesired contacts. Training is single-stage and end to end; deployment does not require privileged state or policy distillation.
The 3.5-hour figure describes reference-motion duration, not GPU training time or the number of simulation transitions. It should therefore be read as data efficiency, not a direct measure of total compute efficiency.
5. Fall Recovery as Contact-Distribution Expansion
RGMT integrates recovery into the tracking policy through a simple curriculum:
- With probability 0.15, an environment resets the robot into a randomized unstable pose.
- Recovery environments initially receive an upward force sampled from ([0,200]), helping exploration reach recoverable states.
- The assistance is linearly annealed until the final policy stands up under its own control.
- Instability normally terminates a rollout, but recovery environments receive a three-second grace period to stand and re-stabilize.
This mechanism has two effects. It teaches autonomous recovery after an external push, and it exposes the controller to ground contacts, transitions, and low-body configurations that ordinary upright tracking rarely visits. The latter effect explains why recovery training also improves crawling, kneeling, sitting, and breakdance-style tracking.
6. Main Results
All three compared methods are evaluated in MuJoCo. Success means completing a rollout without the root height deviating from the reference by more than 0.2 m. MPJPE measures root-relative 3D joint-position error in millimeters.
| Method | MoCap success | Video-derived success | Ground-interaction success |
|---|---|---|---|
| GMT | 84.6% | 72.4% | 48.9% |
| Any2Track | 89.2% | 54.3% | 41.2% |
| RGMT | 98.3% | 94.6% | 90.1% |
| Method | MoCap MPJPE | Video-derived MPJPE | Ground-interaction MPJPE |
|---|---|---|---|
| GMT | 65.15 mm | 96.47 mm | 146.95 mm |
| Any2Track | 56.96 mm | 112.16 mm | 209.57 mm |
| RGMT | 41.12 mm | 46.56 mm | 54.92 mm |
Architecture ablations support the proposed mechanism. Replacing cross-attention with self-attention reduces success to 76.7% on video-derived motion and 73.2% on ground interaction. Replacing the causal history encoder with a CNN is less damaging, reaching 91.9% and 81.5%. The command encoder is therefore the larger contributor under distribution shift, while causal dynamics history provides a complementary gain.
Recovery training has little effect on ordinary MoCap and video success, but changes ground-interaction success from 70.5% to 90.1% and reduces MPJPE from 96.75 mm to 54.92 mm. This is one of the paper’s clearest ablations because it isolates the benefit of expanded contact experience.
7. Noise Robustness and Real-World Deployment
The authors perturb reference base velocity, angular velocity, gravity direction, and joint position on a Charleston dance clip. GMT and Any2Track degrade rapidly beyond 200% of the base noise specification. RGMT remains stable in the reported experiment up to 1500%, with errors increasing more gradually. Cross-attention removal causes the largest degradation at high noise, supporting its role as the command-filtering component.
“1500% noise” is a scaled synthetic stress test defined by the paper’s perturbation ranges. It is useful as a relative robustness curve and does not represent a universal physical noise threshold.
On the physical Unitree G1, the policy demonstrates four input modes:
- fixed MoCap and breakdance-style ground-contact references;
- video-derived motion reconstructed from public videos;
- real-time full-body control from PICO VR trackers or a motion-capture suit;
- joystick-driven stylized locomotion through an upstream motion-matching system.
The fall-recovery demonstration applies an external push, after which the same policy stands and resumes tracking without a manual reset. The joystick example is also informative: discrete motion-matching switches create nonsmooth command transitions, providing a practical test of the reference-filtering interface.
8. Strengths, Caveats, and Research Takeaways
RGMT has a clean architectural hypothesis. Command reliability depends on the robot’s current dynamics, so command aggregation should use dynamics as its query. The design is small, causal on the state-history side, compatible with real-time control, and trained in one stage. Integrating recovery into the same policy improves both operational continuity and contact-rich tracking.
The evidence has several boundaries. The baseline comparison uses released checkpoints on a common evaluation platform, yet the methods differ in training pipeline, action representation, and robot configuration. The paper provides strong simulation tables and qualitative hardware demonstrations, while large-scale quantitative real-robot comparisons are absent. Its attention weights also offer a learned selection mechanism without a formal guarantee that rejected commands are physically infeasible.
The controller uses root-relative references and does not include global localization. Long-horizon world-frame position and heading can therefore drift. The authors identify global localization and deeper integration with upstream motion generation and planning as future work.
The broader lesson extends beyond humanoids: when an upstream generator supplies imperfect trajectories, the low-level controller benefits from treating them as contextual proposals. Recent closed-loop state supplies the evidence needed to decide which parts of that proposal remain executable.