[Paper Notes] DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A generated USB hub can look convincing while its cable moves like a rigid extension of the case. DiagGen tests such assets in a physics simulator, observes their response to controlled interactions, and sends the evidence back to the generation stage responsible for the suspected fault. Starting from one image, it builds geometry and material specifications, then runs a generate–simulate–diagnose–refine loop.
The useful idea is the repair decision: an implausibly stiff cable may require a new material assignment, a better part mask, or a corrected mesh. DiagGen makes those alternatives explicit. In the paper’s 40-object experiment, diagnostics requests 14 revisions. Static specification scores improve, while paired interaction judgments favor nine revisions, four baselines, and one tie. The method provides a practical way to debug generated assets; the results also show why each repair needs another interaction test.
Paper and sources
DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation, by Guanxiong Chen, Yiduo Qu, Qianjun Xia, Pengyu Jing, Yixian Cheng, Bole Ma, Pengzhi Yang, Bingyang Zhou, Ziming Li, Shashwat Suri, Gongbo Sun, Chao Liu, Peter Yichen Chen, Ziqiu Zeng, and Fan Shi. The first three authors share equal technical contribution. Affiliations span UBC, Cambridge, NUS, FAU, and Wisconsin–Madison.
These notes follow the eight-page preprint arXiv:2609.23103v1, submitted September 19, 2026. See the PDF and project page for the original figures and videos. The cited version does not state a conference acceptance. Results below are author-reported; the interpretation and proposed evaluation priorities are my reading of the paper.
1. Part boundaries determine which physics can be assigned
The USB hub example explains why physical repair reaches upstream into perception. In the initial asset, poor segmentation prevents the cable from becoming a distinct deformable part. Downstream processing consequently assigns one material to the whole object. During a grasp-and-lift test, the cable keeps an overly stiff bend as the case rises. After segmentation repair, the cable and case can receive different material behavior: the cable stays attached, bends under gravity, and lets the connector hang downward (Section IV-C and Fig. 5).
This dependency shapes the generation pipeline. Image cleanup produces a reference image and a description emphasizing rest shape and material cues. SAM3 supplies part masks, and OmniPart uses the masks and description to reconstruct separate part meshes. Material inference assigns mechanical properties to the parts. Mesh processing establishes metric scale and joins the parts into a connected, monolithic mesh for Genesis. Diagnostics tests that candidate before final export.
The paper uses volumetric meshes and Genesis’s linear elastic material model to represent objects containing both soft and relatively rigid regions. The material specification includes Young’s modulus $E$, Poisson’s ratio $\nu$, density $\rho$, friction, and representation choices such as fill mode. Connectivity is just as consequential as these values: a tetrahedralized OmniPart toy separates during a lift when its parts lack an elastic connection; DiagGen’s monolithized version remains one object under the same action.
For me, this is the strongest motivation for the workflow. A usable deformable asset couples semantic parts, geometry, scale, connectivity, and material response. A failure in one layer can appear as a material problem during simulation.
2. The diagnostic agent chooses a physical question to test
The agent receives indexed parts, geometry summaries, and material specifications, then proposes up to four semantic anchors. Each anchor identifies a region with informative behavior, such as a handle, rim, flap, or compliant body, and states a physical hypothesis. A grasp box specifies the intended contact region. Preview renders let the agent correct the target and motion direction before running the interaction.
Each approved anchor starts a fresh episode from a resting state. The end effector approaches, grasps the selected region, applies a prescribed displacement, releases it, and allows the object to settle. The protocol is shared across revisions; the anchor, displacement, duration, and observation schedule depend on the asset. If the evidence is insufficient, the agent can pause, inspect, and run further probes.
The observations include synchronized RGB triptychs from top and oblique views, overlays linking visible regions to parts, geometry and action metadata, step summaries, and runtime logs. Deformation under contact and recovery after release provide different evidence. The metadata helps distinguish a material mismatch from an incorrect target, bad scale, missing geometry, or a simulator initialization failure.
The diagnostic agent returns observations, a suspected cause, and a recommended repair route. An orchestrator considers the revision history and decides whether to accept the asset, launch the repair, or halt. This is an inference-time workflow built around model reasoning and tool calls. The paper’s central mechanism does not depend on a newly trained end-to-end asset generator or a differentiable reconstruction objective.
3. Repair reruns the responsible stage and its downstream stages
Section III-C defines three destinations for repair. Problems in the diagnostic episode itself, such as a misplaced grasp target, stay within diagnostics.
| Repair destination | Evidence that motivates it | Artifacts preserved before rebuilding |
|---|---|---|
| Segmentation | Distinct functional regions were merged, or a continuous region was fragmented | Cleaned image reference |
| Material inference | Geometry is coherent, but representation or mechanical response is implausible | Reference, masks, and OmniPart meshes |
| Mesh processing | Scale, topology, or discretization prevents meaningful contact or deformation | Earlier artifacts, including representation and material assignments |
The selected stage and every later stage run again to produce a complete candidate. Versioned outputs preserve which image, masks, geometry, materials, and diagnostic evidence produced each revision. The exported bundle contains the asset and its supporting manifest, material specification, and diagnostic record.
Table II gives concrete examples of the information passed through this loop. For a juice box, Genesis rejects initialization because local self-clearance is below the assigned shell-thickness threshold. The material stage reduces shell thicknesses and replaces a uniform 3.0 GPa stiffness with 0.8, 0.7, 0.6, and 0.3 GPa for the body, top, seam, and flap. The rebuilt candidate initializes and completes its probes. These are generated settings that resolve the reported failure; the paper does not establish them as measured properties of the photographed carton.
A medical bag exposes a different problem. Its surviving pouch looks stable during interaction, but handles, straps, and hardware are absent from the simulated object. Mesh processing increases fidelity for the missing parts and restores all five part labels in a connected mesh with 13,318 tetrahedra. Stable motion of the surviving geometry would have missed that failure.
4. Static consistency and dynamic plausibility answer different questions
Part–Material Specification Consistency (PMSC) scores whether the asset’s static specification agrees with the image and common material priors. Two independent VLM evaluators inspect an anonymous candidate; they do not see its revision status, diagnostic history, paired alternative, or each other’s judgment. One assigns a whole-asset score $H\in{1,2,3,4,5}$. The other sets $F=1$ when a major concern undermines the representation or specification, and $F=0$ otherwise. Equation (1) gives
\[S=\max\left(1,\;H-F-\mathbf{1}[T>40{,}000]\right),\]where $T$ is the tetrahedron count. The 40,000 threshold is a heuristic computation budget; exceeding it costs one point. PMSC therefore combines semantic and material consistency with a mesh-cost penalty. It is not a force or deformation error measured against a real object. The reported experiment applies this rubric retrospectively to existing independent reviews.
Paired Asset Interaction Plausibility (PAIP) compares the baseline and revision in motion. Before examining the revision, the evaluation freezes one probe from the baseline diagnostic evidence, including its target part, anchor, and action. Both candidates execute that probe under the same simulator, controller, and camera settings, with each grounding the target on its own parts. Two VLM judges each compare A versus B and B versus A, producing four judgments for the final win/loss/tie outcome. Swapping presentation order helps mitigate position bias.
The repair experiment uses GPT-5.6 Luna at Max reasoning for generation and diagnostics, and GPT-5.6 Terra at Max reasoning for PAIP judging. Using another model variant and hiding revision information reduce some evaluation biases. They still leave an evaluation based on visual plausibility and model priors, without calibrated physical ground truth.
5. What the experiments actually establish
The fixed evaluation set contains 40 objects from phone images and prior work. Each run permits at most one repair to keep the experiment tractable. The before/after comparisons concern the 14 assets selected for revision.
| Evaluation | Scope | Reported result |
|---|---|---|
| Seeded-fault diagnosis | Three selected objects, one known fault per object | Correct issue and repair destination in all three cases |
| PMSC before/after | 14 revised assets; five-point scale | Mean 2.14 → 3.43; 10 improve, 4 tie, 0 decline |
| PMSC without revision | 26 assets accepted on the first attempt | Mean 3.12 |
| PAIP before/after | The same 14 revised assets | Revised wins 9; baseline wins 4; 1 tie |
The seeded faults isolate the three repair routes: a curling iron’s flexible cord receives barrel-like stiffness; a massager’s segmentation omits most of its metal coil; kitchen tongs have their metric scale changed from 0.28 m to 2.8 m. GPT-5.5 at high reasoning diagnoses these cases without being told the injected issue or its owning stage. The three successes are useful qualitative evidence for routing, with a deliberately small and unambiguous test set.
The dynamic results deserve as much attention as the static gain. The revised plunger cup deforms more plausibly relative to its handle, although the baseline recovers more cleanly after release. The baseline plush kitten supports slightly better garment selection and a similar response with less force. For the sport watch, the baseline buckle and keeper respond more clearly; the revised target barely moves. A better specification score can coexist with a worse response to a particular interaction.
The robotic case studies use Genesis’s implicit FEM solver with SAP contact and an agent that iteratively proposes gripper poses and apertures, simulates them, and revises the trajectory. The demonstrations cover a toy with a keyring, kitchen tongs and a cookie, and a USB hub. A scene reconstruction example also integrates a DiagGen asset into Agentic Real2Sim using Genesis. These studies demonstrate simulation usability and qualitative behavior. The paper leaves quantitative effects on policy learning and real-world manipulation success for future work.
6. What I would reuse, and what I would measure next
I would reuse the combination of targeted interaction tests, explicit repair destinations, and versioned artifacts in an asset-generation system. It gives a failed rollout a concrete next action and makes regressions inspectable. For a downstream task, I would preserve its critical probes and require repaired assets to pass them again before entering a training set.
The hardest unresolved issue is attribution. Bad segmentation and bad materials can produce similar symptoms, a limitation the authors explicitly acknowledge. Three isolated faults do not establish reliable diagnosis when several stages are wrong at once. The one-repair budget also leaves the behavior of longer revision loops largely untested, and large-scale parallel generation has not been evaluated.
If the goal is diverse, plausible objects for simulation, this workflow looks useful. If the goal is a quantitatively accurate twin of a particular object, I would add measured forces and deformations to the evaluation. My next experiment would hold out real interaction trajectories, compare generated responses against those observations, and check whether the repaired assets improve the policy’s target task. That would connect the paper’s plausibility gains to the physical errors that matter in deployment.