[Paper Notes] DeformGen: Dynamics-Based Topology Augmentation for Deformable Manipulation Policy Learning
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
DeformGen is a data augmentation framework for deformable object manipulation. It starts from sparse demonstrations, generates new physically plausible deformable object states through localized physical disturbances and forward simulation, then transfers the original manipulation trajectory to each new state using deformation-field warping. The goal is to turn one or a few demonstrations into many usable training episodes for rope, cloth, toy, and other deformable manipulation tasks.
My read: the paper is useful because it states the failure mode of rigid-object demonstration augmentation precisely. Rigid augmentation assumes equivariance: if the object pose changes by an SE(3) transform, the end-effector trajectory can receive the same transform and remain valid. Deformable objects break this assumption. Their state is high-dimensional, physically constrained, and non-equivariant under local deformation. DeformGen therefore augments both sides together: the object topology and the behavior attached to that topology.
Paper and Resources
The paper is “DeformGen: Dynamics-Based Topology Augmentation for Deformable Manipulation Policy Learning” by Zili Lin, Wenyao Zhang, Yuyang Zhang, Zekun Qi, Junyan Lin, Hanxin Zhu, Jiaolong Yang, Zhibo Chen, Yao Mu, Xiaokang Yang, Xin Jin, and Wenjun Zeng. The affiliations include Shanghai Jiao Tong University, Eastern Institute of Technology, Ningbo, Tsinghua University, The Hong Kong Polytechnic University, University of Science and Technology of China, Zhongguancun Academy, and Microsoft Research.
The paper is available as arXiv:2606.25939. The project page is zili2002.github.io/DeformGen, and the code is at Zili2002/DeformGen.
Why Rigid Augmentation Fails
For rigid manipulation, a single demonstration can often be reused by changing the object pose and applying a matching rigid transform to the robot trajectory. This works because distances between material points are preserved, and the relative pose between gripper and object can be kept consistent.
Deformable manipulation has two different problems.
First, the object state is not a 6-DoF pose. The paper models a deformable object as \(N\) particles:
\[ s = (p_1, \ldots, p_N) \in \mathbb{R}^{3N}. \]
Only a small subset \(S_{\mathrm{real}} \subset \mathbb{R}^{3N}\) corresponds to physically plausible configurations. A random perturbation in particle space can easily create disconnected ropes, impossible cloth folds, self-intersections, or shapes with unrealistic internal stress. Low-dimensional rigid pose perturbations stay valid, but they cover only spatial diversity, not shape or topological diversity.
Second, trajectory transfer is non-equivariant. When a rope bends or a cloth drapes, nearby material points move differently. A global rigid transform of the end-effector trajectory may misalign the grasp pose, miss the local geometry, or fail to compensate for the changed global shape. For deformables, valid data augmentation must synthesize a new object state and a new trajectory that matches that state.
State Augmentation: Simulate Into Plausible Topologies
DeformGen’s first component is Dynamic Topological Transformation. Instead of directly editing the particle positions, it starts from a known valid state \(s_0 \in S_{\mathrm{real}}\), applies localized physical disturbances, and forward-simulates the object dynamics:
\[ s_{\mathrm{aug}} = \Phi_{\mathrm{sim}}(s_0, f, \Delta t). \]
Here \(f\) is a localized force field and \(\Phi_{\mathrm{sim}}\) is the simulator rollout. The key assumption is conditional: a good simulator approximately preserves physical plausibility when it evolves from a valid state, while it cannot reliably repair an invalid state produced by arbitrary geometric perturbation.
This is the core design choice. DeformGen does not ask the simulator to fix broken shapes after the fact. It uses the simulator as the generator itself. Localized forces can produce bending, twisting, folding, draping, and compression while the internal constraints, contacts, and settling dynamics remain coupled through simulation.
In the implementation, the simulator does not expose a direct external-force API. The authors implement disturbances by commanding an xArm7 gripper to execute randomized Cartesian perturbations while contacting the object. Rope and toy use 180 random steps; cloth uses 260. After perturbation, the object is stabilized for 30 to 40 simulation steps.
Trajectory Augmentation: Warp the Demonstration Through the Deformation Field
Generating a new object state is only half the problem. The source demonstration trajectory must also be adapted. DeformGen uses Deformation-Field Warping.
Let \(p^{\mathrm{orig}}, p^{\mathrm{def}} \in \mathbb{R}^{N \times 3}\) be the source and deformed point clouds. Each particle displacement is:
\[ \delta_i = p^{\mathrm{def}}_i - p^{\mathrm{orig}}_i. \]
For an end-effector waypoint \(x_t\), DeformGen finds its \(K\) nearest source particles and interpolates their displacements with inverse-distance weights:
\[ d(x_t) = \sum_j \tilde{w}{t,j}\delta{\mathrm{nn}_j(x_t)}. \]
The warped waypoint is:
\[ x_t^{\mathrm{warp}} = x_t + \alpha_t d(x_t). \]
The scalar \(\alpha_t) is a decay function. It lets the trajectory strongly follow local deformation near the grasp phase while gradually reverting toward the original path when useful for the manipulation phase.
Orientation is adapted by estimating a local Jacobian from the original KNN neighborhood to the deformed neighborhood. The induced rotation is projected onto \(SO(3)\), then interpolated with the original orientation using SLERP:
\[ R_t^{\mathrm{warp}} = \mathrm{SLERP}(R_t, R’_t, \alpha_t). \]
The practical detail is important: the grasp pose uses small \(K\), because grasp alignment depends on nearby geometry. The manipulation phase uses all object points, because the motion must compensate for the global deformation of the object.
Policy Training Setup
The experiments use Real2Sim-Eval with PhysTwin for soft-body dynamics and rendering. The robot is an xArm7 with two RGB cameras: third-person and wrist-mounted, both at 848 by 480 and 30 Hz. The policy outputs an 8D action: end-effector position, quaternion orientation, and gripper opening.
The paper evaluates three tasks:
| Task | Goal | Success criterion |
|---|---|---|
| Rope routing | Thread a rope through a clip | enough rope intersections with upper and lower clip planes |
| Toy packing | Put a stuffed toy into a container | enough object points inside a scaled oriented bounding box |
| Cloth folding | Fold cloth into a triangle | projected mask matches a fitted triangle by contour and IoU criteria |
For each task, the authors collect one teleoperation source demonstration, generate more than 1,200 distinct object states, synthesize trajectories, and keep successful rollouts. They train ACT, Diffusion Policy, SmolVLA, and \(\pi_0\) with LoRA, then evaluate on held-out object states unseen during training.
Baselines and Ablations
The comparison is structured to separate state augmentation from trajectory transfer:
| Method | State augmentation | Trajectory transfer |
|---|---|---|
| 1 Src. | none | source demo only |
| SoftMimicGen* | rigid state perturbation | deformation-field warping |
| DeformGen* | topological state augmentation | local rigid trajectory transfer |
| DeformGen | topological state augmentation | deformation-field warping |
This setup isolates two questions. Comparing SoftMimicGen* with DeformGen tests whether topology-diverse states help. Comparing DeformGen* with DeformGen tests whether deformation-field warping improves over local rigid transfer on the same augmented states.
Results
The headline result is that DeformGen improves policy learning in most settings. In Table 2, the single-source baseline is close to zero success on most tasks. Full DeformGen achieves the best average success for three of four policy architectures:
- ACT: 59.00% average success with DeformGen, compared with 1.33% for one source demo.
- SmolVLA: 56.50% with DeformGen, compared with 2.50% for one source demo.
- \(\pi_0\): 56.67% with DeformGen, compared with 2.33% for one source demo.
Diffusion Policy is the exception where DeformGen* slightly exceeds full DeformGen on average, but the overall trend still supports the value of topological augmentation. The strongest numbers appear in rope routing, where DeformGen reaches 90.50% with ACT, 92.00% with SmolVLA, and 99.00% with \(\pi_0\).
The state coverage analysis supports the mechanism. Rigid augmentation clusters near the source state with near-zero non-rigid residual. DeformGen spreads broadly in the unified state PCA and shows large non-rigid residuals after Procrustes alignment. This indicates that it is creating genuine deformable shape variation, not simply more rigid poses.
The synthetic data scaling study also matters. For ACT, average success increases from 19.50% at 100 synthetic trajectories to 61.50% at 750. For SmolVLA, it increases from 36.83% to 63.17%. This suggests the generated data is useful at scale, beyond acting as a small regularizer.
Limitations
The paper is simulation-only. Real2Sim-Eval and PhysTwin provide high-fidelity dynamics and rendering, but the authors do not show real-robot deployment. Sim-to-real transfer could require additional adaptation for material properties, contact dynamics, and visual appearance.
The tasks are limited to single-arm manipulation with xArm7. Rope, stuffed toy, and cloth cover 1D, quasi-rigid 3D, and 2D sheet-like deformables, but they do not cover bimanual folding, cable routing in clutter, dough or clay shaping, or surgical tissue manipulation.
Trajectory synthesis is also not guaranteed. The success rates for generating valid trajectories vary: rope is easy at 99.5%, toy is 60.3%, and cloth is 39.5%. Large topology changes, unstable contact, and kinematic constraints can make warped trajectories fail. The paper partially addresses this by filtering and by showing that policies can generalize to some synthesis-failure states, while the generation process still has clear reliability limits.
Takeaways
DeformGen’s main contribution is a clear augmentation principle for deformable manipulation: generate new states through dynamics, then transfer trajectories through deformation fields. This is a stronger recipe than adapting rigid-object SE(3) augmentation to a setting where the object state lives in \(\mathbb{R}^{3N}\).
For robot learning, the practical message is direct. If the task involves deformable objects, collecting one demonstration and rigidly moving it around will mostly create spatial variety. To create useful training diversity, the augmentation process must change the object’s topology while preserving physical plausibility, and the robot trajectory must move with the object’s local and global deformation.