[Paper Notes] VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
VT-MUSE learns a compact state representation from a history of synchronized external RGB and bilateral optical-tactile observations. Its central premise is that contact-rich manipulation depends on how contact evolves over time. A single tactile frame can reveal local deformation; a sequence reveals whether contact is increasing, slipping, aligning, or stabilizing. Scene vision provides the global geometry that tactile sensing cannot see.
The framework separates representation learning from task-specific policy learning. Stage I jointly adapts separate visual and tactile ViT encoders using synchronous cross-modal alignment, adjacent-state temporal contrast, and consistency under random visual masking. Stage II freezes those encoders, masks the recent visual tail, retains the complete tactile history, and trains a conditional variational latent model to reconstruct the missing RGB frames and predict tactile depth changes. The deployable conditional prior becomes a frozen multimodal memory for a lightweight action-chunking Transformer policy, injected through gated cross-attention.
VT-MUSE reaches 55.25% average success on four simulated UniVTAC tasks, above ACT+UniVTAC at 39.00%. On four physical Flexiv tasks, it succeeds in 76 of 80 trials (95.00%), while ACT with current vision and touch succeeds in 25 of 80 trials (31.25%). The ablations strongly support temporal and dual-modality supervision. The generalization claim remains bounded by four simulation tasks, four real tasks, one optical tactile gripper, discrete task IDs, and at most seven representation-pretraining tasks.
Paper Info
“VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation” is by Congsheng Xu, Qiaochu Yang, Fangyuan Shi, Yifan Han, Baijun Chen, Yiming Wang, Haonan Zhao, Daolin Ma, Xiaokang Yang, and Hesheng Wang, with affiliations at Shanghai Jiao Tong University and Xense Robotics. This note covers arXiv:2608.21290v1, submitted August 21, 2026. VT-MUSE expands to Multimodal Unified SEquential representation learning.
1. Why Visuotactile History Matters
Vision and touch observe different scales of the same interaction. External RGB describes object pose, scene layout, target geometry, and long-range motion. Optical tactile images expose local contact geometry after the gripper touches an object or surface. Tactile feedback remains informative when the end effector or manipulated object occludes the important visual region.
Many visuotactile systems encode the two modalities independently and fuse their current features during policy training. That design gives the controller touch, but leaves two relationships weakly supervised: the fine correspondence between visual change and contact change, and the temporal evolution of the contact itself. These relationships matter in insertion, compliant wiping, key withdrawal, and button pressing, where similar-looking camera frames can hide different force or alignment states.
VT-MUSE shifts representation learning toward a synchronized temporal window. For target time (t), the model constructs strided histories of length (L):
[ \mathcal I_t=(I_{t-(L-1)s},\ldots,I_t), \qquad \mathcal T_t=(T_{t-(L-1)s},\ldots,T_t), ]
where (I) is external RGB, (T=(T^L,T^R)) contains the two optical tactile views, and (s) is a fixed temporal stride. The encoder receives a masked visual history (\overline{\mathcal I}_t), the full tactile history, and a discrete task identity (c):
[ f_t^{VT}=E_{VT}(\overline{\mathcal I}_t,\mathcal T_t,c). ]
This is a perceptual state encoder. It does not predict actions during representation pretraining.
2. The Complete VT-MUSE Pipeline
flowchart TD
A["Synchronized RGB + bilateral tactile histories"] --> B["Stage I: jointly adapt modality encoders"]
B --> C["Cross-modal alignment + temporal contrast + mask consistency"]
C --> D["Freeze visual and tactile ViTs"]
D --> E["Stage II: mask recent vision, keep full tactile history"]
E --> F["Interleaved temporal Transformer memory"]
F --> G["Conditional variational latent state"]
G --> H["Reconstruct RGB tail + predict tactile depth flow"]
H --> I["Freeze deployable conditional-prior encoder"]
I --> J["Gated cross-attention into action-chunking policy"]
J --> K["Predict K robot actions"]
The pipeline contains two forms of separation. The visual and tactile encoders remain modality-specific, with separate parameters. Their training is coordinated by cross-modal objectives. The representation encoder is also trained before downstream policies and then frozen, allowing pooled multi-task sensory data to produce a reusable memory while each task-specific policy uses a smaller action-labeled dataset.
The word unified therefore describes the learned sequential state and its supervision. It does not mean that raw camera and tactile pixels pass through one shared backbone.
3. Stage I: Align Modalities, Time, and Missing Views
Stage I initializes two pretrained ViTs, one for external RGB and one for tactile RGB. Left and right tactile images are spatially concatenated and resized into a three-channel observation. Each token receives modality, temporal-slot, and relative-time embeddings. A learnable task token retrieves task-conditioned context and injects it into both streams.
Three objectives adapt the encoders to robot interaction data.
Synchronous Cross-Modal Alignment
Visual and tactile tokens from the same temporal slot form a positive pair; observations from other batch samples act as negatives. A symmetric InfoNCE objective aligns both directions:
[ \mathcal L_{\text{cross}} =\frac{1}{L}\sum_{i=1}^{L}\frac{1}{2} \left(\ell_{v\rightarrow h}^{(i)}+\ell_{h\rightarrow v}^{(i)}\right). ]
This objective asks the encoders to recognize which scene state and contact state occurred together.
Adjacent-State Temporal Contrast
For each temporal slot, the normalized visual and tactile features are averaged into a multimodal state (s_i). Neighboring states from the same trajectory receive a contrastive temporal objective:
[ \mathcal L_{\text{temp}} =\frac{1}{L-1}\sum_{i=1}^{L-1} \mathcal L_{\mathrm{NCE}}(s_i,s_{i+1}). ]
It organizes nearby interaction states even when their individual RGB or tactile frames contain ambiguity.
Mask-Invariant Consistency
The same history receives two independently sampled visual masks. Their multimodal states are encouraged to remain consistent:
[ \mathcal L_{\text{cons}} =1-\frac{1}{BL}\sum_{b=1}^{B}\sum_{i=1}^{L} \cos!\left(s_{b,i}^{(1)},s_{b,i}^{(2)}\right). ]
The complete Stage-I loss is
[ \mathcal L_{\text{Stage 1}} =\lambda_{\text{cross}}\mathcal L_{\text{cross}} +\lambda_{\text{temp}}\mathcal L_{\text{temp}} +\lambda_{\text{cons}}\mathcal L_{\text{cons}}. ]
Only the final three Transformer blocks of each pretrained ViT are updated. This preserves much of the pretrained visual knowledge while adapting the upper representation layers to synchronized robot interaction.
4. Stage II: Infer the Visual State from Contact History
Stage II freezes both modality encoders. The final (K) visual tokens in the history are replaced with learnable mask tokens; the complete tactile sequence remains visible. Visual and tactile tokens are interleaved in time and processed by a temporal Transformer, producing memory (M).
The final (K) tactile tokens serve as queries over this memory. Their retrieved context is combined with the tactile queries and task context to form a compact latent interaction state. This asymmetry encodes a useful deployment assumption: recent vision may be missing or occluded near contact, while tactile sensing directly observes the local interaction.
The latent model has two distributions:
- A conditional prior uses only information available at deployment.
- A privileged posterior additionally observes the ground-truth visual tokens hidden from the prior.
The posterior teaches the deployable prior during representation training and is removed at inference. Two decoders force the latent to retain complementary information. One reconstructs the masked recent RGB frames; the other predicts bilateral tactile depth differences, called tactile depth flow. The Stage-II objective is
[ \mathcal L_{\text{Stage 2}} =\lambda_I\mathcal L_{\text{rgb}} +\lambda_D\mathcal L_{\text{depth}} +\beta\mathcal L_{\text{KL}}. ]
The RGB and depth terms use mean-squared error. The KL term regularizes the privileged posterior toward the conditional prior. RGB reconstruction emphasizes global scene state; tactile depth-flow prediction emphasizes local contact geometry and its change.
5. Turning the Representation into Policy Memory
After Stage II, the conditional-prior encoder is frozen. A two-layer adapter with layer normalization and GeLU projects its output into the hidden space of a four-layer Transformer policy. Two intermediate policy layers retrieve the projected representation through cross-attention:
[ X’{\text{act}} =X{\text{act}} +g\,\mathrm{CrossAttn}(X_{\text{act}},U_{VD},U_{VD}), ]
where (X_{\text{act}}) are policy action tokens, (U_{VD}) is the projected VT-MUSE memory, and (g) is a learned scalar gate. Its initialization gives the auxiliary memory only a small influence at the start of policy training.
The policy also receives the current visual, tactile, and proprioceptive observations directly. VT-MUSE memory adds historical interaction context; it does not replace immediate feedback. The output is an action chunk,
[ \widehat A_t= \pi(q_t,I_t,T_t,f_t^{VT}) =[\hat a_t,\ldots,\hat a_{t+K-1}]. ]
Training uses an (L_1) action-prediction loss plus KL regularization of the action posterior toward a unit Gaussian:
[ \mathcal L_{\text{policy}} =\mathcal L_{\text{act}} +\lambda_{\text{act}}\mathcal L_{\text{act-KL}}. ]
Despite its multimodal name, this policy is not a language-conditioned VLA in the reported experiments. Multi-task representation learning uses a discrete task ID, and the downstream policies are task-specific.
6. Data and Evaluation Setup
Simulation uses four UniVTAC tasks: Lift Bottle, Pull-out Key, Insert Hole, and Insert HDMI. The authors collect 500 trajectories per task, producing 2,000 sensory trajectories for representation learning. Each task-specific downstream policy uses the benchmark’s 50 action-labeled demonstrations. Encoder pretraining takes about 25 hours on eight NVIDIA A800 GPUs; each frozen-encoder policy takes about 30 minutes on one A800.
The physical platform is a Flexiv Rizon 4s arm with an XenseGripper. Four tasks cover different contact structures: Insert Tube, Wipe Board, Pull-out Drawer, and Press Toaster. The dataset contains 50 demonstrations per task, or 200 trajectories total. The same trajectories supply sensory sequences for representation learning and action labels for policy learning. Every method receives 20 evaluation trials per task.
This split matters when interpreting transfer. Simulation demonstrates pooled representation pretraining followed by smaller task-specific policy learning. The real-robot experiment trains representation and policies from the same 50 demonstrations per task, so its result demonstrates strong sample reuse and architectural benefit within the evaluated task set, not transfer from an independently collected pretraining corpus.
7. Simulation Results—and a Reporting Discrepancy
| Method | Lift Bottle | Pull-out Key | Insert Hole | Insert HDMI | Four-task average |
|---|---|---|---|---|---|
| ACT, vision only | 42 | 28 | 19 | 15 | 26.00 |
| ACT + UniVTAC | 71 | 45 | 23 | 17 | 39.00 |
| ViTaL pretraining | 72 | 47 | 25 | 6 | 37.50 |
| FTP-(\pi_{0.5}) | 77 | 30 | 47 | — | — |
| VT-MUSE | 84 | 38 | 68 | 31 | 55.25 |
VT-MUSE improves the complete four-task average by 16.25 absolute points over ACT+UniVTAC, the strongest baseline with results on all four tasks. ViTaL remains stronger on Pull-out Key, so VT-MUSE does not dominate every contact pattern.
The paper contains a small numerical inconsistency worth preserving in the record. On the three tasks shared with FTP-(\pi_{0.5}), VT-MUSE reports task scores of 84, 38, and 68. Their arithmetic mean is 63.33%, matching the prose in Section IV-B. Table I and Figure 1 print 62.33%, while the abstract describes an 11-point improvement over FTP’s 51.33%. The per-task values imply a 12.00-point improvement. The four-task average of 55.25% is internally consistent.
8. Physical-Robot Results
| Method | Insert Tube | Wipe Board | Pull-out Drawer | Press Toaster | Aggregate |
|---|---|---|---|---|---|
| ACT, vision only | 1/20 | 3/20 | 13/20 | 4/20 | 26.25% |
| ACT, vision + touch | 1/20 | 5/20 | 15/20 | 4/20 | 31.25% |
| VT-MUSE | 19/20 | 19/20 | 20/20 | 18/20 | 95.00% |
Adding the current tactile observation directly to ACT yields a modest five-point gain. VT-MUSE adds 63.75 points over that visuotactile ACT baseline. This gap supports the paper’s main argument: access to touch and a learned history of cross-modal interaction are different capabilities.
The magnitude also invites caution. The evaluation has 20 trials per task, one robot and gripper, four tasks, and two locally trained ACT baselines. The tactile hardware and tasks are closely matched to VT-MUSE’s design. The paper appropriately frames 95% as validation under these conditions, without claiming unrestricted real-world generalization.
9. What the Ablations Show
| Representation variant | Four-task success |
|---|---|
| Stage I only | 27.25% |
| Without visual reconstruction | 37.00% |
| Without tactile depth-flow prediction | 38.75% |
| Without temporal contrastive loss | 42.25% |
| Full VT-MUSE | 55.25% |
Stage-I cross-modal alignment alone is insufficient. Adding the full Stage-II conditional latent model creates a much more useful control representation. Removing either reconstruction target loses 16.5–18.25 points, showing that global visual state and local tactile dynamics contribute complementary supervision. Removing temporal contrast while keeping the temporal Transformer loses 13 points, isolating the value of temporal supervision from architecture size.
Pixel-level reconstruction quality is not a reliable proxy for control value. The variant without tactile prediction achieves the best RGB PSNR and SSIM, yet its policy success is only 38.75%. Joint training accepts a small trade-off in isolated RGB fidelity and preserves more task-relevant multimodal information.
Observation length is sharply non-monotonic:
| Window length (L) | Average success |
|---|---|
| 3 | 36.50% |
| 4 | 40.00% |
| 5 | 55.25% |
| 6 | 41.25% |
More history can introduce redundant or outdated interaction states. VT-MUSE uses (L=5), while keeping sampling stride fixed. The paper identifies adaptive sampling and variable-length memory retrieval as future work.
10. Strengths and Limitations
VT-MUSE’s strongest design choice is its clean interface. The encoder learns a task-conditioned sequential perceptual state from pooled sensory data; the policy consumes that state as optional memory through a gate. The privileged-posterior construction gives the deployable prior a concrete missing-vision target, and the tactile depth-flow loss asks it to retain contact evolution with direct relevance to manipulation.
Several boundaries remain. The representation uses a discrete task ID, so zero-shot language grounding and unseen-task inference are not evaluated. Pretraining scales only from four to seven tasks; the t-SNE analysis shows organized clusters but cannot establish a scaling law, especially because task identity is an encoder input. The real dataset reuses the same demonstrations for representation and policy training. Sensor generalization across different optical tactile hardware is not tested.
The missing-vision setup masks the recent visual tail during representation learning, yet downstream policy execution still receives the current camera and tactile observations. The experiments therefore show an auxiliary memory that is robust to incomplete historical vision, not a robot policy that operates without current vision. Fixed temporal stride and fixed window length can also miss fast contact events or retain stale observations.
Finally, the simulation baselines are not uniform across task subsets, and the real experiment lacks a stronger pretrained visuotactile policy baseline. Together with the aggregate-number discrepancy, this makes per-task results and exact evaluation scopes more informative than the headline percentage alone.
Takeaway
VT-MUSE treats touch as a structured temporal signal linked to changes in the visible scene. Stage I aligns the modalities and nearby interaction states. Stage II learns a deployable latent state by reconstructing hidden recent vision and predicting tactile depth flow. A gated policy retrieves this state as memory while retaining its immediate sensory-action pathway.
The result is a compelling recipe for contact-rich imitation learning: pool sensory histories for representation learning, preserve both global geometry and local contact dynamics, freeze the encoder, and let small task policies query the memory. The next test is scale—more tasks, new tactile sensors, independent pretraining corpora, language-conditioned policies, and adaptive temporal sampling.