[Paper Notes] DexAnyTwist: Learning General Dexterous Twisting with Hybrid Manipulation System Identification
Published:
DexAnyTwist asks a useful scaling question for dexterous manipulation: why can adding more object diversity make a universal policy worse? Its answer is that a broad twisting dataset is a hybrid dynamical system. A screwdriver, valve, bottle cap, nut, and fragile light bulb impose different contact formation, torque transmission, friction, scale, and slip constraints. A single continuous policy receives incompatible gradients and settles on a mediocre average action pattern.
The paper’s central contribution is DexSifter, an iterative expert-discovery procedure. It trains a policy over all objects, assigns the reliably solved objects to that expert, trains a new expert on the unsolved residual set, and finally learns a router that selects one expert at inference. This turns general twisting from one giant function-fitting problem into partitioning and controlling locally consistent manipulation regimes.
Paper Info
The paper is “DexAnyTwist: Learning General Dexterous Twisting with Hybrid Manipulation System Identification” by Xing Liu, Yunlong Dong, Jun Wan, Linan Deng, Feng Hua, Yi Shen, Min Yu, Guijun Ma, Cheng Cheng, Haitao Song, Han Ding, and Ye Yuan. It appeared in National Science Review in June 2026. The official paper page and project page provide the article, demonstrations, and data.
Why General Twisting Is a Hybrid-System Problem
Twisting looks like a single skill at the task label level, yet its contact mechanics change sharply across objects. A small nut asks for fingertip coordination inside a narrow workspace. A valve may allow a stable power grasp. A light bulb asks for torque with controlled contact force. Screw and bulb tasks can also slip or tilt as the movable component loosens.
The authors model this using latent manipulation modes (m_o\in{1,\ldots,K}):
[ P(\mathbf{s}{t+1}\mid\mathbf{s}_t,\mathbf{a}_t,m_o)=P_k(\mathbf{s}{t+1}\mid\mathbf{s}_t,\mathbf{a}_t). ]
Each (P_k) represents a locally consistent regime. The mode is not manually labeled from physical properties. DexAnyTwist discovers a practical partition through learning performance: objects an expert handles consistently are treated as one subsystem; the unresolved cases expose a residual regime that needs a new expert.
This explains the paper’s counterintuitive scaling result. With the same architecture, optimizer, and training setup, increasing the object library from 10 to 290 instances sharply reduces the consecutive-success metric for a monolithic policy. More data expands coverage, but it also increases dynamic conflict.
Training World and Task Definition
The authors build a self-contained library of more than 300 simulated objects in 10 categories, including bottles, valves, screws, and bolts. Every object is modeled as a stationary base plus a twistable part.
There are two physics templates. Valve-like tasks use one revolute joint about the vertical axis. Screw- and bulb-like tasks add two revolute joints at the bottom along the (x) and (y) axes. Those additional degrees of freedom represent loosening, pose change, and relative motion that can make an object slide out of the hand. The benchmark therefore spans both straightforward rotation and contact-rich rotation with grasp-stability requirements.
The reported metrics separate short-horizon completion from sustained control:
- CSC (Consecutive Success Count): average number of sequential goals completed in one continuous episode; this is the primary long-horizon metric.
- SR (Success Rate): fraction of attempts that finish the goal.
- TTR (Time to Reach): time required to reach a goal.
DexSifter: Forward Partition, Backward Refinement, Hard Routing
DexSifter has three stages.
First, it performs forward partitioning. An initial expert is trained on the full object distribution and evaluated by empirical CSC on every object. The objects it reliably solves become that expert’s assigned subset. The remaining objects form the residual set. A fresh expert is trained on this residual set while prior experts stay frozen. Repeating the process produces a library of complementary experts.
Second, it applies backward refinement. Each expert is further optimized only on its assigned subset. This removes gradients from unrelated contact regimes after the partition is established.
Third, a learned gating network receives hand-object interaction features and routes to a single expert:
[ j^=\arg\max_{j\in{1,\ldots,K}}p_{\boldsymbol\phi,j}, \qquad \mathbf{a}=\pi_{j^}(\mathbf{x}). ]
The route selects one expert instead of averaging their actions. That detail matters: blending two policies that expect different contact modes can create exactly the incoherent action that the decomposition was designed to avoid.
Policy Inputs and Architecture
DexAnyTwist observes proprioceptive hand states, object states, and an object point cloud. A recurrent backbone encodes the temporal interaction state, while a PointNet encoder extracts geometry-aware object features. A routing module maps these features to the expert policy library.
The point cloud is doing more than visual decoration. Similar rotational objectives can require different finger placements for different geometry, and the routing decision needs a representation that connects shape with the current contact state. The recurrent component likewise lets an expert react to interaction history instead of treating every frame as an independent grasp pose.
Results: Decompose the Long Tail
DexAnyTwist’s result is strongest in the heterogeneous, difficult portion of the distribution. On the most challenging setting, the paper reports an 18.5% improvement in success rate and a 114% improvement in CSC over the baselines. On the hard subset, the method maintains 69.5% SR, while the compared baselines fall to 18.1% SR.
The system is also transferred directly from simulation to real hardware. High-level target joint positions run at 10 Hz, and the low-level PD motor controller runs at 1 kHz. Training initially uses object-joint positions, which are convenient simulator states but unavailable on physical objects. The authors therefore distill a deployment student with curriculum learning: Gaussian noise on those joint-position inputs is gradually increased until the final policy operates without that privileged signal.
The real-world evaluation includes six representative objects. The demonstrations show twisting for a screwdriver, valve, spray bottle, nut, and fragile bulb-like objects. The point is zero-shot transfer across unfamiliar geometries, with no object-specific policy fine-tuning.
Emergent Manipulation Primitives
One attractive qualitative result is that the learned behavior becomes interpretable. Fingertip-density visualizations show the thumb acting as the main torque-producing digit, while the index and middle fingers tend to form localized contact regions that behave like virtual pivots. This division resembles a human twist strategy: one digit drives rotation and other digits stabilize the object and constrain its axis.
That observation should be read as an empirical property, not a hand-coded prior. The method does not prescribe a thumb-actuator / finger-pivot controller. The roles emerge after learning policies whose partitions better match the underlying contact regimes.
What I Take from the Paper
DexAnyTwist makes a broader point about embodied-policy scaling. Dataset diversity is valuable only when the learner has a mechanism to represent incompatible physical modes. For contact-rich skills, more objects can create a fractured optimization landscape before they create a universal policy.
DexSifter offers a simple operational test for that situation: train on all data, identify what is already stable, isolate the residual failure modes, specialize, then learn a state-and-geometry-aware router. It uses control performance to build a useful latent decomposition instead of recovering symbolic equations for the hybrid system. This occupies a practical middle ground between a monolithic end-to-end policy and fully specified contact-mode modeling.
The remaining question is routing under broader real-world uncertainty. The paper’s hard routing is deliberately mode-consistent, yet a wrong mode decision can be costly around contact transitions. Extending the approach with uncertainty-aware routing, recovery behaviors, or tactile observations would be a natural next step for deployment in cluttered everyday settings.