[Paper Notes] LESSMIMIC: Long-Horizon Humanoid Interaction with Unified Distance Field Representations
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
LESSMIMIC proposes a unified representation for humanoid-object interaction based on distance fields (DFs). Instead of conditioning a humanoid policy on motion references or crafting task-specific rewards per skill, the paper represents interaction through:
- distance to the object surface
- distance-field gradients
- velocity components decomposed into surface-normal and tangential directions
The result is a single whole-body policy that can generalize across object geometry and scale, recover from failures, and compose multiple skills over long horizons. The paper’s central argument is that a good interaction representation matters as much as the control algorithm.
Paper Info
- Title: LESSMIMIC: Long-Horizon Humanoid Interaction with Unified Distance Field Representations
- Authors: Yutang Lin, Jieming Cui, Yixuan Li, Baoxiong Jia, Yixin Zhu, Siyuan Huang
- Affiliations: Peking University, BIGAI, Beijing Institute of Technology
- Project page: lessmimic.github.io
- arXiv: 2602.21723
1. Motivation
The paper targets a familiar limitation in humanoid manipulation:
- reference-based methods can produce high-quality motions, but they are tightly coupled to the geometry and trajectories seen in demonstrations
- reference-free methods are more flexible, but usually rely on task-specific rewards and end up as isolated single-skill policies
So the real question becomes: what interaction representation would let a humanoid:
- act without reference motions at inference time
- generalize to new object shapes and scales
- compose different interaction skills inside one policy
- recover when the world deviates from the nominal script
The authors argue that distance fields provide exactly that interface.
2. Core Idea
The key move in LESSMIMIC is to represent humanoid-object interaction in a geometry-aware but motion-agnostic way.
For each humanoid link position relative to the object, the distance field provides:
- the distance to the nearest surface
- the local surface gradient / normal
- a way to decompose link velocity into:
- normal motion: approach / push / apply force
- tangential motion: slide / traverse along the surface
This representation is meant to capture the structure of interaction rather than memorizing a particular reference trajectory.
That is the paper’s main conceptual contribution. Instead of telling the humanoid “follow this motion,” it tells the humanoid “reason about how your body is moving relative to the object’s geometry.”
3. Method Overview
3.1 DF-based interaction representation
At the representation level, the policy observes geometric cues derived from the object’s distance field. The paper emphasizes that absolute coordinates alone are not enough, because they do not tell the agent whether it is approaching, pressing, sliding, or disengaging from the object.
So the method uses:
- DF values
- DF gradients
- velocity decomposition into normal and tangential parts
These interaction features over time are encoded by a VAE into an interaction latent z_t, which smooths the signal and provides a compact policy input.
3.2 Three-stage training pipeline
The training pipeline has three stages:
Interaction skill pre-training
A teacherpi_mimictracks retargeted human motions with physics-aware residual compensation. A studentpi_baseis then trained via DAgger-style behavior cloning, but crucially without access to reference motions at inference.Discriminative post-training
The student is fine-tuned with RL under geometry randomization. Instead of reference tracking rewards, the policy is guided by Adversarial Interaction Priors (AIP), which regularize interaction validity in the DF latent space.Visual-motor distillation
A MoCap-conditioned full policy is distilled into a vision-based policy that uses egocentric depth only, so the system can deploy without motion-capture infrastructure.
This is a clean design: mimic for physical feasibility, RL for geometric generalization, and distillation for deployability.
4. Why Distance Fields Help
The value of DFs here is not just that they encode geometry. The representation has three useful properties:
- it is local, so it stays meaningful across different object shapes
- it is continuous and differentiable, so it provides smooth geometric feedback
- it is task-agnostic, so the same interface can support pushing, pickup, carrying, and sitting interactions
That last point is the most important one. LESSMIMIC is trying to build a shared language for heterogeneous humanoid-object interactions.
5. Main Results
5.1 Generalization across object scale and shape
The paper evaluates four interaction tasks under object scale variation:
- PickUp
- SitStand
- Push
- Carry
The training scale is 1.0x, while evaluation ranges from 0.4x to 1.6x. For pickup, the paper also varies object shape across boxes, cylinders, and spheres.
Key takeaways:
- reference-based baselines degrade sharply once object scale shifts away from the reference motion setup
- LESSMIMIC remains much more stable across scales
- the MoCap-conditioned full model reports 80-100% success on several scale-generalization settings for PickUp and SitStand
- the policy generalizes in the real world from trained box-like objects to a soccer ball, which is a good qualitative sign that the policy is using geometry rather than memorized motion patterns
This is the strongest result in the paper: the representation seems to support geometry generalization better than motion-conditioned baselines.
5.2 Long-horizon skill composition
The paper then tests whether one policy can handle multiple heterogeneous tasks in sequence without environment resets.
They evaluate randomly ordered task compositions of length:
N = 5N = 10N = 15N = 25N = 40
The full MoCap-based LESSMIMIC policy reports:
- 61.7% success at
N=5 - 38.1% at
N=10 - 23.5% at
N=15 - 9.0% at
N=25 - 2.1% at
N=40
Those numbers are not huge at the longest horizon, but they matter because the ablated variants collapse to nearly zero much earlier. The result suggests the unified DF representation actually helps preserve cross-skill consistency over long execution chains.
5.3 Failure recovery
One of the paper’s qualitative claims is that the policy can recover from perturbations. For example, after an object is dropped, the humanoid can re-initiate pickup from the object’s new location instead of failing permanently because a reference trajectory was broken.
This makes sense given the representation design: if the policy is grounded in current geometry rather than a fixed motion target, it has a better chance to re-plan implicitly through closed-loop control.
5.4 Real-world deployment
The paper evaluates both a MoCap-based and a vision-based policy on a real humanoid platform.
Reported results include:
- MoCap-based
- PickUp
22 cm^3:10/10 - PickUp
60 cm^3:8/10 - SitStand
12 cm:8/10 - SitStand
46 cm:10/10
- PickUp
- Vision-based
- PickUp
22 cm^3:8/10 - PickUp
60 cm^3:7/10
- PickUp
The vision model is weaker than the MoCap-conditioned one, but still reasonably effective, which supports the claim that the DF-based interaction logic can be distilled into depth-based deployment.
6. Ablation Insights
The ablations are useful because they show the system is not succeeding from one trick alone.
- Removing AIP significantly hurts robustness, implying the adversarial interaction prior is important for geometry-consistent interaction.
- Removing synthetic physicalization hurts contact-rich tasks, especially carrying, showing that physically valid teacher trajectories matter.
- Removing geometry randomization causes severe overfitting outside the training scale.
- Removing RL post-training leaves behavior cloning insufficient for strong generalization.
- Replacing the Transformer with an MLP hurts performance, especially on tasks with longer temporal dependencies.
Overall, the paper argues that the full stack is necessary: representation, physically grounded pre-training, adversarial post-training, and sufficient model capacity.
7. Strengths
- Clear problem framing around representation rather than only reward design or imitation quality.
- The DF formulation is intuitive and task-general.
- Strong evidence for scale/shape generalization relative to motion-tracking baselines.
- Long-horizon skill composition is a meaningful benchmark, not just isolated single-task success.
- The paper includes a plausible path from privileged MoCap training to egocentric-depth deployment.
8. Limitations and Open Questions
- The vision-based model still shows a noticeable gap from the MoCap-conditioned version.
- The long-horizon success rate drops substantially by
N=25andN=40, so this is still far from robust open-ended execution. - The task set, while heterogeneous, is still fairly structured; more cluttered or highly deformable interactions would be a harder test.
- The method still depends on retargeted human interaction data and a mimic teacher during pre-training.
- It is not fully clear how far the DF abstraction can scale once interactions involve more complex articulated or deformable objects.
9. Takeaways
My main takeaway is that LESSMIMIC makes a compelling case that interaction representation is a bottleneck in humanoid manipulation. A policy that reasons through distance-field geometry can be both more adaptive than motion-tracking systems and more unified than task-specific reference-free controllers.
The paper does not solve long-horizon humanoid interaction in a complete sense. But it gives a strong recipe:
- use a geometry-centered interaction representation
- pre-train from physically valid demonstrations
- post-train with geometry randomization and interaction priors
- distill into vision for practical deployment
That feels like a solid direction for building humanoids that can actually chain contact-rich behaviors together in the real world.