[Paper Notes] DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
DeCAL is a dexterous vision-language-action model built for manipulation where contact changes quickly and the fingers often hide the object. Its central design joins three Transformer experts: an understanding expert reads language, cameras, and touch; a generation expert imagines future visual and tactile states; an action expert turns those representations and robot state into an action chunk.
Two details make the model physically grounded. Contact-aware gating raises or lowers the contribution of tactile features at each step, so touch matters strongly during contact and does not constantly perturb visual grounding. Visuo-tactile latent co-imagination predicts future visual latents together with future tactile signals, including raw tactile images, deformation maps, and 6-DoF forces. The action expert can therefore condition on an anticipated interaction state instead of reacting only to the current frame.
On six real-robot tasks, DeCAL averages 70.8% full-task success (SR) and 83.35% progress success (PSR) over 20 trials per task; the abstract reports these as 71% and 83.4%. It reaches 100% on Wipe Vase, 80% on Erase Whiteboard, 65% on Assemble Parts, 80% on Twist Cap, 60% on Pipetting, and 40% on Screw Light Bulb. On Twist Cap with an unseen object, it keeps 75% success. The system runs at 30 Hz, and the appendix reports 0.27 s per 50-step action chunk on one RTX 4090.
My read is that DeCAL’s strongest contribution is the separation of when touch should enter the policy from what physical evolution the policy should anticipate. The paper also exposes the current costs: accurate tactile sensing is assumed, demonstrations have no fingertip force feedback for the operator, and the model is trained without large-scale visuo-tactile pretraining.
Paper Info
- Title: DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
- Authors: Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang
- Affiliations: Peking University; Beijing Academy of Artificial Intelligence
- Date: arXiv:2609.09119v1, September 8, 2026
- Links: arXiv abstract · project page · code
1. Why Vision Alone Breaks at Contact
A vision-only policy can identify a bottle or a socket, yet a dexterous task is decided by details that cameras observe poorly: whether a fingertip has actually touched the object, how force is changing, whether the object is slipping, and how deformation evolves after a small motion. The hand itself creates occlusion, and a tiny pose error can change the contact mode.
Touch supplies those missing physical signals, but simply concatenating tactile tokens with visual-language tokens has its own failure mode. Contact is intermittent. During free-space motion, noisy tactile features can distract the visual representation and alter grasp decisions. DeCAL therefore treats fusion as a state-dependent decision and models the next interaction state explicitly.
The robot platform has two 6-DoF UR5 arms and two 22-DoF SharpaWave five-finger hands. Three Intel RealSense D435 cameras provide one egocentric view and two wrist views. Each fingertip has a 320 x 240 vision-based tactile sensor. The tactile stream is represented in three forms: the raw sensor image, a deformation-depth map, and a 6-DoF net-force vector (three forces plus three torques).
2. One MoT Model, Three Specialists
DeCAL uses a Mixture-of-Transformers (MoT) architecture with directional information flow:
flowchart LR
I["Language + multi-view RGB + tactile + robot state"] --> U["Understanding expert\nQwen3-VL backbone"]
U --> G["Generation expert\nfuture visual/tactile latents"]
G --> A["Action expert\nfactorized flow matching"]
A --> O["50-step arm + hand action chunk"]
The training objective asks one policy to model future visuo-tactile latents and future actions conditioned on the current observation and language instruction:
\[\max_{\theta}\;\mathbb{E}_{(o_t,l,a_{t+1:t+H},z_{t+H})\sim\mathcal D} \left[\log \pi_{\theta}(a_{t+1:t+H},z_{t+H}\mid o_t,l)\right].\]The three experts have different jobs.
Understanding expert. Qwen3-VL encodes the instruction and the three camera views. Tactile features enter through a dedicated cross-attention layer: visual-language embeddings provide queries, and tactile tokens provide keys and values. The resulting features are added back to the joint self-attention stream.
Generation expert. Given the current and recent history, this expert predicts future visual and tactile latents in parallel. Parallel decoding matters because pixel-level autoregressive generation would be too slow for high-frequency control. The expert uses Qwen3 and receives the current frame plus the previous 15 frames, about 0.5 seconds at 30 Hz.
Action expert. It predicts actions with flow matching, conditioned on the understanding and generation representations and the robot state. The action space contains both arm and hand motion. DeCAL uses Factorized Flow Matching: arm and hand actions start from separate noise distributions and separate token sequences, are jointly processed for coordination, and are decoded into their respective trajectories. This preserves a shared coordination pathway while giving the two motion types distinct denoising processes.
The attention mask enforces the causal direction Understanding -> Generation -> Action. Tokens inside the understanding and generation experts can attend bidirectionally. Action tokens can attend to the robot state and preceding expert blocks, so an action sees semantic context and imagined future contact without allowing later modules to rewrite earlier representations.
3. Adaptive Visuo-Tactile Fusion
DeCAL encodes the deformation map from each fingertip with a shared ResNet-based encoder. Spatial pooling and a cross-finger Transformer produce two forms of tactile representation: local tokens for detailed contact patterns and one global token summarizing the hand’s contact state.
At every Transformer block, local tactile tokens participate in cross-attention. A lightweight gate computes tactile weights from the global tactile token:
\[\widetilde X = X + \sigma \odot \operatorname{CrossAttn}(Q_{vl},K_{local},V_{local}), \qquad \sigma = \operatorname{Gate}(z_{global}).\]Here, $X$ is the current embedding, $Q_{vl}$ is the visual-language query, and $K_{local},V_{local}$ are the local tactile keys and values. When the fingertips are not interacting with anything, the gate stays low and the policy remains primarily visual. During meaningful contact, the gate increases and tactile evidence has a larger effect on action generation.
The paper’s episode visualizations on Assemble Parts and Twist Cap show this intended behavior. The gate does not represent a fixed global preference for touch; it tracks the interaction phase. That distinction is useful for tasks such as insertion, where vision locates the socket but contact tells the policy whether the plug is aligned and advancing.
4. Latent Co-Imagination of Future Contact
The generation expert predicts two coupled futures.
For touch, a force encoder maps each fingertip’s 6-DoF force/torque signal to an embedding. The model predicts a tactile latent that can be decoded into three targets: future raw tactile images, future deformation maps, and future 6-DoF force vectors. These targets cover different scales of interaction: appearance of the sensor surface, local contact geometry, and global wrench dynamics.
For vision, Cosmos VAE encodes each camera image into a latent grid. A 32 x 32 grid is compressed to 4 x 4 with an 8 x 8 convolution before future tokens are decoded in parallel. During training, the visual latent is regressed to the Cosmos target. Tactile latents are trained indirectly through reconstruction of the future raw, deformation, and force signals. At inference, DeCAL retains the predicted latents as context for the action expert and omits the pixel-space reconstruction decoders.
The appendix writes the full loss as
\[\mathcal L_{total} = \lambda_v\mathcal L_{visual} + \lambda_t\mathcal L_{tactile} + \mathcal L_{action},\]with
\[\mathcal L_{tactile} = \lambda_R\mathcal L_{img}(\widehat R,R) + \lambda_M\mathcal L_{def}(\widehat M,M) + \lambda_F\mathcal L_{force}(\widehat F,F).\]For action prediction, an expert action chunk $a_{t:t+k}$ is interpolated with Gaussian noise $\epsilon$:
\[\widehat a^{\tau}_{t:t+k}=(1-\tau)\epsilon+\tau a_{t:t+k}, \qquad \tau\sim\operatorname{Beta}(1.5,1.0),\]and the learned velocity field is trained toward $a_{t:t+k}-\epsilon$. This gives the action expert a continuous transport objective while the visual and tactile heads supply forward-looking physical context.
5. Data and Evaluation
The authors collect 100 high-quality teleoperated demonstrations for each of six tasks and evaluate each method with 20 trials:
| Task | Main contact challenge | DeCAL SR | DeCAL PSR |
|---|---|---|---|
| Wipe Vase | Hold a vase while wiping its curved surface | 100.0% | 100.0% |
| Erase Whiteboard | Maintain eraser contact over a surface | 80.0% | 86.7% |
| Assemble Parts | Align and insert a plug into a socket | 65.0% | 85.0% |
| Twist Cap | Stabilize a bottle and unscrew a cap | 80.0% | 93.3% |
| Pipetting | Press a plunger and hand over the pipette | 60.0% | 86.3% |
| Screw Light Bulb | Grasp, insert, and rotate a bulb | 40.0% | 48.8% |
SR requires the entire task to succeed. PSR averages completion over task stages, so it shows partial progress when a long-horizon task fails late. DeCAL’s average is 70.8% SR and 83.35% PSR. The strongest baseline averaged about 55.8% SR from the table, so the gap is roughly 15 percentage points.
The comparison includes GR00T N1.6, InternVLA-A1, ViTacFormer, DECO, and a tactile-augmented InternVLA-A1t. Vision-only policies are competitive on simpler visual stages but lose reliability in sustained contact. Tactile specialists improve physical feedback yet lack DeCAL’s unified future dynamics. The naive InternVLA-A1t result also shows that adding tactile inputs to a VLM is insufficient by itself: tactile signals can disturb visual grounding when fusion has no adaptive control.
6. What the Ablations Say
The ablation on Assemble Parts and Twist Cap removes one component at a time:
| Factorized FM | Tactile gating | Visual generation | Tactile generation | Assemble Parts | Twist Cap |
|---|---|---|---|---|---|
| No | Yes | Yes | Yes | 25% | 35% |
| Yes | Yes | No | No | 20% | 30% |
| Yes | Yes | No | Yes | 35% | 45% |
| Yes | Yes | Yes | No | 55% | 60% |
| Yes | No | Yes | Yes | 50% | 70% |
| Yes | Yes | Yes | Yes | 65% | 80% |
The largest single drop in this small ablation comes from removing Factorized Flow Matching on Assemble Parts. The authors observe inconsistent arm-hand coordination when one denoising process handles the full action space. Removing both latent-generation paths also causes a large decline, consistent with the claim that future motion and future contact state help the policy advance through insertion and twisting. Removing the gate lowers performance by letting tactile features interfere with visual-language features at weak-contact stages.
The generation metrics support the same mechanism. On Assemble Parts, DeCAL improves over InternVLA-A1 from Cosmos cosine similarity 0.913 to 0.946 and lowers LPIPS from 0.243 to 0.222. On Twist Cap, cosine similarity rises from 0.908 to 0.927 and LPIPS falls from 0.257 to 0.245.
7. Generalization and Failure Modes
The OOD evaluation uses Twist Cap under four shifts: a new background, a cluttered scene, dimmer lighting, and a novel cup with different shape, diameter, and height. Figure 9 reports DeCAL at 60%, 70%, 70%, and 75% SR respectively. The appendix’s stage-wise Table 6 gives 65% for unseen lighting while the figure gives 70%, so that one number should be read with care. The 75% unseen-object result is the most revealing: the policy must change grasp height, contact position, and wrist pose, so appearance augmentation alone is unlikely to explain the transfer.
The paper reports two main failure categories. Bimanual tasks such as Pipetting can fail when arm motion, hand alignment, and handover timing drift apart. A limited head-camera field of view can also let an object leave the visible workspace during large motions. The authors suggest wider-angle or fisheye cameras and active perception as possible remedies.
There are three additional limits in the method itself. Tactile sensor noise, calibration errors, and model drift can degrade long or high-load operation. The teleoperator receives no fingertip force feedback, which can reduce contact regulation and demonstration quality. Finally, the current model does not use large-scale visuo-tactile pretraining, leaving open whether the same architecture will scale across more embodiments and contact regimes.
8. Takeaways
DeCAL gives tactile input two separate roles. The gate decides how much the current policy should trust touch; latent co-imagination gives the policy a representation of what touch and vision are likely to become next. Factorized flow matching then turns that context into coordinated arm and hand motion.
For future dexterous VLA systems, the paper suggests a practical recipe: preserve the semantic prior of a pretrained VLA, expose contact through structured tactile representations, make fusion conditional on interaction state, and train future tactile dynamics alongside action prediction. The unresolved engineering questions are equally concrete: better force feedback during data collection, more reliable tactile calibration, larger pretraining corpora, and active sensing when the camera loses the object.