[Paper Notes] End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy
Published:
This paper is about a very practical bottleneck in dexterous VLA work: high-quality arm-hand demonstrations are hard to collect because full-DoF teleoperation overloads the operator. The proposed answer is Shared Autonomy. A human controls the arm’s coarse end-effector motion through VR, while a tactile-aware DexGrasp-VLA policy controls the dexterous hand as a Copilot. The collected trajectories then fine-tune an end-to-end arm-hand VLA policy.
My read: the interesting part is the division of labor. The human supplies scene understanding, target choice, approach strategy, and pre-grasp positioning. The autonomous hand policy supplies contact-rich grasp closure, tactile adaptation, and force modulation. This converts full arm-hand teleoperation into a semi-autonomous data engine, then uses Arm-Hand Feature Enhancement and Corrective Teleoperation to turn that data into a stronger policy.
Paper Info
The paper is “End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collection” by Yu Cui, Yujian Zhang, Lina Tao, Yang Li, Xinyu Yi, and Zhibin Li from ByteDance Seed. The arXiv version is 2511.00139, and the project page is dexvla-seed.github.io/dex-vla.
Problem
Dexterous arm-hand manipulation has two coupled difficulties. The arm handles macro motion: reaching, avoiding obstacles, positioning the hand, and aligning pre-grasp poses. The hand handles micro motion: compliant multi-finger closure, contact adaptation, force control, and slip recovery. A monolithic teleoperation setup asks the operator to control both at once. The paper reports that even 20-30 minutes of continuous full-DoF teleoperation can be tiring, especially for untrained operators.
Automated planning has the opposite failure mode. It can create many state-action pairs, but its data distribution reflects its solvers and constraints. The resulting motions may be stiff, physically valid, and diverse while still missing the small human strategies that make dexterous manipulation work.
The paper’s formulation is a compromise:
- human operator: VR-guided 6-DoF arm end-effector motion;
- DexGrasp-VLA Copilot: autonomous 12-DoF dexterous hand control;
- final policy: end-to-end arm-hand VLA trained from synchronized shared-autonomy data.
Both the hand-only policy and the final arm-hand policy are fine-tuned from (\pi_0) using LeRobot.
DexGrasp-VLA As Hand Copilot
The paper uses VLA policies in two places. The first is DexGrasp-VLA, a hand-only tactile-aware VLA policy. Its job is to control the fingers during data collection, so the operator can focus on moving the arm.
DexGrasp-VLA is bootstrapped in two stages. First, the authors train a vision-free LSTM grasping policy from proprioception and tactile feedback. The dataset combines 68 parameterized force-adaptive control demonstrations and 150 human teleoperation demonstrations. The force-adaptive controller follows:
[ q_c(i)=q_m(i)+q(0)e^{-k f_z(i)}. ]
When contact is weak, the command closes the hand quickly. As fingertip normal force increases, the closure slows and tightens into a stable grasp. The LSTM sees a temporal window of proprioception and tactile readings:
[ x_t=[s^{hand}_t,f^{hand}_t]\in\mathbb{R}^{39}, \quad s_t=[q^{hand}_t,\tau^{hand}_t]\in\mathbb{R}^{24}, \quad f^{hand}_t\in\mathbb{R}^{15}. ]
It predicts a 12-DoF hand action with behavior cloning and MSE loss. This creates a compact reactive expert for tactile force adaptation.
Second, the LSTM expert collects data for a multimodal hand VLA policy. DexGrasp-VLA consumes:
[ o^{hand}_t=[I^{hand}_t,l_t,q^{hand}_t,z^{tac-f}_t,z^{tac-s}_t]. ]
Here (I^{hand}t) is the eye-in-hand image, (l_t) is the language instruction, (q^{hand}_t) is the hand state, (z^{tac-f}_t) is per-fingertip resultant force, and (z^{tac-s}_t) is a spatial tactile embedding. The raw tactile tensor is (F{raw}\in\mathbb{R}^{10\times12\times3}) per fingertip. The paper compresses it two ways:
- resultant force vectors (f^{tac-f}_t\in\mathbb{R}^{5\times3}), giving explicit net force magnitude and direction;
- convolutional autoencoder tactile latents (f^{tac-s}_t\in\mathbb{R}^{5\times128}), preserving spatial contact patterns.
This combination is important because force magnitude alone cannot describe where contact sits on the fingertip. Spatial tactile latents help the policy detect slippage, off-center contact, and shifts in contact distribution.
Shared Autonomy Data Collection
Shared autonomy combines VR arm teleoperation with autonomous hand control. The VR system uses relative motion mapping: once the user engages the controller, the initial VR controller pose (T_{VR,0}) and robot end-effector pose (T_{robot,0}) define the mapping:
[ T_{robot,t}=T_{robot,0}(T_{VR,0}^{-1}T_{VR,t}). ]
The target pose is converted to joint commands through a velocity-level QP IK solver using PlaCo and Pinocchio. The paper reports OpenXR support at 90 fps, less than 100 ms latency, and synchronized data saving at 30 Hz.
During collection, the human drives the arm while DexGrasp-VLA produces hand actions:
[ D^{uni}={(o^{uni}t,a^{arm}_t,a^{hand}_t)}{t=1}^{T}, \quad a^{arm}t\sim p{teleop}, \quad a^{hand}t\sim\pi{hand}(\cdot|o^{hand}_t). ]
The combined observation contains multi-view RGB, language, arm joint states, and hand joint states. The result is a synchronized arm-hand dataset where the arm motion has human spatial intent and the hand motion has tactile-adaptive execution.
In the appendix, shared autonomy collects 110 trajectories/hour/person for the main dataset, compared with 90 trajectories/hour/person for full teleoperation. Corrective collection is 100/hour/person versus 80/hour/person. The authors frame this as roughly a 25% collection-rate gain, with lower operator fatigue and a faster policy iteration loop.
End-to-End Arm-Hand VLA
The final policy (\pi_{uni}) controls the full arm-hand system end to end:
[ \pi_{uni}(A^{uni}_t|o^{uni}_t). ]
The key architectural addition is Arm-Hand Feature Enhancement. The base (\pi_0) model produces a shared task representation (z^{share}_t) from visual, language, and proprioceptive inputs. The enhancement module passes this shared representation through two dedicated MLPs:
[ z^{arm}t=E{arm}(z^{share}t), \quad z^{hand}_t=E{hand}(z^{share}_t). ]
Auxiliary heads predict arm and hand sub-actions from the limb-specific features. The main action head uses the fused representation:
[ z^{fused}_t=[z^{share}_t,z^{arm}_t,z^{hand}_t]. ]
The loss combines the main flow-matching action loss with auxiliary arm and hand losses:
[ L_{total}=L_{main}+\lambda(L_{hand}+L_{arm}). ]
The motivation is straightforward. Arm motion is smooth, spatial, and longer horizon. Hand motion is contact-rich, local, and force-sensitive. A single undifferentiated latent representation can mix these demands poorly. The extra branches encourage the model to preserve shared task context while learning limb-specific dynamics.
Corrective Teleoperation
The paper also builds a corrective human-in-the-loop system. During deployment, successful autonomous rollouts are recorded. When the policy fails, the operator intervenes through the same shared autonomy interface, recovers the task, and creates a corrective trajectory. At iteration (k):
[ D^{(k)}=D^{(k)}{success}\cup D^{(k)}{corrective}. ]
The next policy is fine-tuned from (\pi_0) using the original shared-autonomy dataset plus the curated deployment data:
[ \pi^{(k+1)}_{uni}=SFT(\pi_0;D^{uni}\cup D^{(k)}). ]
The important design choice is targeted correction. Human effort is concentrated on failure states: bad orientations, corner positions, unseen object shapes, and adversarial configurations. This creates a small but high-value data flywheel.
Experiments
The hardware is a UR3e arm with a 12-DoF five-finger Xhand, each fingertip carrying 120 tri-axial force channels, plus two fixed RealSense D435i cameras and one wrist-mounted RealSense D405. The paper also validates the Arm-Hand Feature Enhancement module on an RY-H2 hand. Policies are evaluated at 30 Hz.
Datasets are deliberately small:
- LSTM pretraining: 150 human teleoperation demonstrations plus 68 force-adaptive autonomous trajectories.
- DexGrasp-VLA hand policy: 180 successful grasping trajectories in clutter, covering 60 objects.
- End-to-end arm-hand policy: 100 shared-autonomy demonstrations over 20 common household objects.
- Corrective datasets: 50 orientation-recovery trajectories and 50 corner-case trajectories.
The hand-only DexGrasp-VLA policy is evaluated in cluttered tabletop scenes with more than 50 objects. It reaches 95.5% overall success in a handheld setup, where the human moves the mounted hand and the policy decides finger-level grasping.
The final end-to-end arm-hand VLA is tested on a pick-and-place task across 50 objects. It reaches 88.7% average success, with 91.7% on seen objects and 85.6% on unseen objects.
Ablations
The tactile ablation is one of the most useful parts of the paper. Under a protocol with 3 seconds of visible grasping followed by 10 seconds of full visual occlusion:
| Hand policy | Average success |
|---|---|
| (\pi_{hand-origin}) from (\pi_0) | 21% |
| force tactile only (f^{tac-f}) | 70% |
| force + spatial tactile (f^{tac-f}+f^{tac-s}) | 90% |
This supports the paper’s tactile design: resultant force gives coarse contact magnitude and direction, while spatial tactile embedding gives contact distribution.
The Arm-Hand Feature Enhancement ablation is also strong:
| Policy | Xhand | RY-H2 | Xhand with camera occlusion |
|---|---|---|---|
| baseline (\pi_{uni-origin}) | 88% | 71% | 19% |
| (\pi_{uni-enhance}) | 95% | 81% | 58% |
The occlusion result is the clearest signal. The enhanced model preserves more limb-specific structure and relies less on a single global visual representation.
The paper also reports a negative result that is worth keeping. Adding the same tactile features directly into the unified arm-hand policy lowered performance from 95% to 82% in the tested setting. The authors interpret this as a phase-alignment problem: tactile signals are useful during grasping, but can become noisy during arm reaching, when incidental contacts or no-contact periods dominate. This argues for selective tactile routing or phase-aware gating in future models.
Corrective teleoperation improves targeted failures. The first corrective round with 50 orientation trajectories handles orientation failures better. The second round with 50 corner-case trajectories improves corner placements. The appendix extends the same correction idea to long-horizon gripper tasks, with reported success rates of 65%, 90%, and 70% across three tasks, and to peg-in-hole assembly, where adding 20 recovery trajectories improves success from 70% to 90%.
Strengths and Limitations
The strongest idea is the data collection strategy. Shared autonomy uses humans where they are efficient: semantic understanding, object choice, approach, and spatial positioning. It uses an autonomous tactile VLA where humans are overloaded: force-adaptive multi-finger closure. This is a good pragmatic path for collecting real robot data without asking operators to micromanage every finger.
The Arm-Hand Feature Enhancement module is also conceptually useful. Arm and hand actions are coupled, but their control regimes differ. Explicitly giving the model shared, arm-specific, and hand-specific features is a lightweight way to encode that difference during fine-tuning.
The main limitation is task scope. The current system focuses on grasping and pick-and-place as the core testbed. It has not yet demonstrated complex in-hand reorientation, tool use, or broader long-horizon dexterous manipulation with the five-finger hand. Tactile integration remains unresolved at the full arm-hand policy level; simple uniform fusion hurts performance in their tested setup. Corrective teleoperation is effective and still depends on human intervention, so the longer-term scaling path needs autonomous failure detection and recovery.
Takeaway
This paper is best read as a systems recipe for dexterous VLA data. The main contribution is not just a new policy architecture. It is a full loop:
- train a tactile hand Copilot;
- use it to reduce human burden during shared-autonomy data collection;
- fine-tune an arm-hand VLA with limb-aware feature enhancement;
- improve the policy through corrective deployment data.
For my taxonomy, I would label it Shared Autonomy / Tactile VLA Copilot / Arm-Hand Feature Enhancement / Corrective Teleoperation / Dexterous VLA Fine-Tuning. The reusable lesson is that dexterous VLA scaling may need better data interfaces before it needs larger models. A human should guide intent and arm motion; a tactile Copilot should handle contact; the final policy should learn the coordination pattern from synchronized data.