[Paper Notes] Temporal Visuo-Tactile Learning for Dexterous Grasp Stability
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A hand has closed around an object, but the table is still supporting its weight. Temporal Visuo-Tactile Learning for Dexterous Grasp Stability asks whether the preceding three seconds of contact formation can predict what happens after lifting. A temporal multimodal classifier estimates stability, and the robot uses its score to choose between lifting and releasing the object for another sampled grasp.
The study collects 10,000 trials on 200 objects with four Digit 360 fingertips. Object-disjoint cross-validation reaches 83.55% classification accuracy, compared with 79.56% without touch. On 20 additional unseen objects, the tactile gate succeeds on 82.0% of executed lifts, versus 71.5% for the non-tactile gate. Counting episodes that end without a lift lowers these figures to 59.2% and 54.8%. The gain in lift reliability comes with additional grasp attempts and abstentions.
Paper and source version
Ken Nakahara, Aleksei Buvailik, Prokhor Kotov, and Roberto Calandra, LASR Lab, TU Dresden. These notes follow arXiv:2610.10283v1, submitted October 7, 2026, including its appendices. See the 12-page PDF and official project and dataset page. The source is an arXiv preprint; experimental values below are author-reported unless explicitly calculated.
1. Predict the lift outcome from contact formation
The platform combines a seven-DoF xArm7, a four-fingered 16-DoF Tilburg Hand, four Digit 360 sensors, and an external RealSense D435i camera. Each fingertip records tactile images, audio, acceleration, and pressure. The predictor consumes a window starting at grasp initiation and ending before lift onset:
\[p_\theta=f_\theta(s_{t:t+T})=P(o=1\mid s_{t:t+T}),\qquad T=3\ \mathrm{s}.\]The label $o=1$ means that the robot lifts the object by 100 mm and holds it for three seconds without dropping it. The paper trains with binary cross-entropy, which can be written as
\[\mathcal L_{\mathrm{BCE}}=-\frac{1}{N}\sum_{n=1}^{N} \left[o_n\log p_n+(1-o_n)\log(1-p_n)\right].\]The observation window contains how contact develops during closure. It excludes post-lift observations, which would reveal the target outcome. The model learns from the resulting interaction without explicit contact-force or friction estimates. Its output is a stability score; reaching motions and hand targets are sampled by a separate procedure.
For data collection, SAM 3 segments the object, and depth plus camera calibration supplies its location. Reaching pose and hand joint targets are randomized, the hand closes through 50 interpolated position commands at a nominal 50 Hz, and every trial proceeds to a lift. This unconditional lift produces labels for both good and bad candidates. Object height supplies most labels automatically, with proprioceptive fallback under failed detection and manual correction of some labels. The dataset contains 47.1% stable and 52.9% unstable trials.
2. Encode each fingertip over time, then fuse the hand
The input branches preserve different sampling structures:
| Branch | Resampled model input | Encoder |
|---|---|---|
| External RGB | 16 frames | Pretrained VideoMAEv2-Base plus temporal Transformer |
| Proprioception | 100 steps of 16 joint angles and six-dimensional end-effector pose | Sequence Transformer |
| Each tactile camera | 16 frames | VideoMAEv2-Base plus temporal Transformer, shared across fingers |
| Each fingertip’s audio | 224 steps of 256-bin log-magnitude spectra | Sequence Transformer |
| Each fingertip’s IMU / pressure | 224 steps of three-axis acceleration / scalar pressure | Sequence Transformers |
Visual and tactile images are resized to $256\times256$ and cropped to $224\times224$, keeping crop coordinates consistent across frames in a clip. Training includes both a center crop and a random crop of each sample; evaluation uses center crops.
Within a fingertip, camera, audio, IMU, and pressure features are resampled to a common temporal grid and fused by a shared modality Transformer. Factorized Transformer blocks then exchange information over time and fingers. Small finger-specific residual adapters capture differences between fingertips, and temporal pooling produces one token per finger.
The final stage joins six sensing tokens—external vision, proprioception, and four fingertips—with 16 learnable fusion tokens in a 512-dimensional space. Three Transformer layers with eight attention heads mix them. Mask-aware pooling over the updated sensing tokens feeds a two-layer MLP and sigmoid. Masks accommodate unavailable streams.
The separation is useful: the tactile branch can represent both the evolution of one contact and how several fingers support the same object. It also avoids requiring one independently trained visual backbone per finger. The final fusion stage can still attend to individual fingertips alongside the external scene.
All models train for 100 epochs with AdamW, batch size 4, initial learning rate $10^{-6}$, weight decay $10^{-4}$, cosine scheduling, and a 0.03 warmup ratio. Offline evaluation uses the final checkpoint, a 0.5 classification threshold, and five-fold object-disjoint cross-validation. Every input ablation is trained and evaluated with its modified inputs, so these experiments measure what can be learned from a restricted signal.
3. The useful tactile signal is mostly optical in this setup
The VideoMAEv2-based modality results are:
| Inputs | Mean classification accuracy |
|---|---|
| Vision | 79.66% |
| Proprioception | 79.99% |
| Touch | 82.64% |
| Vision + proprioception | 79.56% |
| Proprioception + touch | 82.34% |
| Vision + touch | 83.52% |
| Vision + proprioception + touch | 83.55% |
Touch accounts for a 3.99-percentage-point improvement over vision plus proprioception. Adding the other modalities to touch alone gives a smaller 0.91-point gain, with a paired test across five folds reported at $p=0.035$. Proprioception adds almost nothing to vision plus touch in this comparison.
Appendix II narrows down which Digit 360 streams matter. With vision and proprioception retained, tactile-camera-only input reaches 83.41 ± 1.30%, close to 83.55 ± 1.58% with all fingertip streams. Audio-only, IMU-only, and pressure-only variants of the touch branch reach 79.60%, 79.94%, and 79.93%, respectively. These are near the 79.56% non-tactile baseline. The result favors optical tactile information under this architecture; the authors leave open whether the other signals need more suitable encoders.
Raw tactile RGB also slightly outperforms absolute differences from the first frame: 83.55% versus 82.58%. Subtracting the background changes the statistics seen by a video backbone pretrained on RGB and can discard stable appearance cues. The experiment gives a concrete reason to test raw inputs before committing to contact-only preprocessing.
4. Separate temporal evidence from backbone capacity
With all sensing branches enabled, a static ResNet-50 plus MLP model reaches 79.79%, a ResNet-50 plus temporal Transformer reaches 81.42%, and VideoMAEv2 plus Transformer reaches 83.55%. That comparison changes architecture, capacity, and pretraining together. The controlled input ablations provide a cleaner test of temporal information.
Keeping the VideoMAEv2-based architecture fixed, the authors repeat the final observation throughout selected streams, preserving the final grasp configuration while removing variation within the window:
| Temporal input condition, all-modality model | Accuracy |
|---|---|
| Original sequence | 83.55% |
| Repeat final visual observation | 83.20% |
| Repeat final proprioceptive observation | 83.57% |
| Repeat final tactile observations | 81.48% |
| Repeat all final observations | 80.05% |
| Jointly permute RGB and tactile-camera frames | 81.93% |
Removing tactile variation costs 2.07 points; removing visual or proprioceptive variation individually changes little. Joint image permutation preserves the frames and alignment between image streams but disrupts their temporal order, reducing accuracy by 1.62 points. This supports information in the dynamics and ordering of contact formation beyond the final contact image. Because the permutation jointly changes vision and tactile-camera order, its effect cannot be assigned exclusively to tactile order.
Spatial detail matters too. The authors downsample tactile images and restore them to the original encoder size. A $1\times1$ image, retaining only mean RGB, yields 79.40%; $28\times28$ reaches 82.51%; both $112\times112$ and $224\times224$ reach 83.55%. Much of the benefit appears at modest spatial resolution. Since the images are enlarged again before encoding, this experiment does not demonstrate reduced model inference cost.
Finger selection changes the sensor inputs while every grasp still uses the full four-fingered hand. All-finger input reaches 83.55%, versus 83.41% for thumb plus middle finger and 82.55% for thumb alone. These findings concern observability under this grasp distribution, not the physical sufficiency of a two-finger hand. The appendix’s contact-count analysis is also observational: trials with more contacts can differ in object difficulty and grasp geometry.
5. A lift gate changes both reliability and coverage
At deployment, both learned gates use a probability threshold of 0.95, fixed before testing. A rejected candidate causes the hand to open, a small end-effector displacement, and another sampled grasp. One initial grasp plus up to five retries is allowed. Six rejections end the episode with a no-lift reset.
Each method is evaluated until it has executed 200 lifts: ten per object across 20 unseen objects, excluded from the training set. The random gate accepts each candidate with probability 0.25, approximating the tactile gate’s acceptance rate, and uses the same retry protocol.
| Method | Successful lifts / executed lifts | Grasp candidates | No-lift resets | Success counting no-lift episodes as failures |
|---|---|---|---|---|
| Direct lift | 99/200 = 49.5% | 200 | 0 | 49.5% |
| Random gate | 95/200 = 47.5% | 764 | 49 | 38.2% |
| Vision + proprioception gate | 143/200 = 71.5% | 690 | 61 | 54.8% |
| Vision + proprioception + touch gate | 164/200 = 82.0% | 779 | 77 | 59.2% |
For the tactile gate, the conservative episode denominator is $200+77=277$, giving $164/277\approx59.2\%$. Its 10.5-point advantage over the non-tactile gate among executed lifts becomes 4.4 points under the episode-level metric. Calculated from the reported counts, the tactile gate uses about 3.90 candidates per executed lift, versus 3.45 without touch. These counts expose the retry burden; the paper does not establish a corresponding wall-clock throughput advantage.
The tactile-versus-non-tactile lift-success comparison has an object-level paired-test $p=0.003$. Random gating does not detectably improve on direct lifting ($p=0.48$), so extra attempts and abstention alone do not explain the learned gates’ benefit. On the random gate’s independently accepted lifts, the tactile score separates candidates strongly: success is 77/89 = 86.5% above its threshold and 18/111 = 16.2% below it.
One detail in Section VI-B needs care: the prose says the episode metric preserves the same ordering as lift success, but its reported numbers place direct lift above the random gate at the episode level. The table here follows the numerical results. Also, a 0.95 model threshold is an operating choice; the observed lift success does not establish 95% probability calibration.
6. What I would carry into a grasping system
The useful component is a pre-lift critic trained on the dynamics of contact formation. It could screen candidates from an existing planner, provided the sensing and grasp distribution remain suitable. The current system’s retries are random local adjustments; the predictor neither scores explicit candidate actions nor learns which correction will improve a rejected grasp.
The three heaviest test objects still achieve only 40–60% lift success. Pre-lift contact can look plausible while failing to support the object’s weight, and mass is not an explicit input. Learning to predict the consequences of a short load-bearing probe, or conditioning the model on proposed adjustments, would address a different and useful part of the control problem.
For a reproduction, I would keep the object-disjoint split, the repeat/permutation controls, and both deployment denominators. A reliable decision to lift and a productive grasping system are related goals, but candidate count, abstention, and elapsed time determine whether a more selective gate is useful in practice.