[Paper Notes] Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
Tactile sim-to-real is hard because raw tactile images are full of sensor-specific optical artifacts (reflections, lighting, camera noise). Tacmap sidesteps this by defining a shared geometric representation – the penetration depth map (deform map) – that can be computed analytically in simulation and learned from real tactile images via a translation network. Both domains meet in this “common geometric space,” enabling zero-shot sim-to-real transfer of RL policies for contact-rich tasks like in-hand rotation. The approach is fast enough for massive parallel training in Isaac Lab.
Paper Info
- Title: Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map
- Authors: Lei Su, Zhijie Peng, Renyuan Ren, Shengping Mao, Juan Du, Kaifeng Zhang, Xuezhou Zhu
- Affiliations: Sharpa, HKUST, NVIDIA
- arXiv: 2602.21625
- Paper type: tactile simulation / sim-to-real transfer / dexterous manipulation
1. Problem and Motivation
Vision-based tactile sensors (VBTS) like GelSight and DIGIT capture rich contact information via a camera observing elastomer deformation. Training policies that use these sensors requires millions of interactions, making simulation essential. But current tactile simulation faces a dilemma:
- Analytical methods (e.g., TACTO): fast depth-buffer rendering, but oversimplifies elastomer physics – large sim-to-real gap.
- Empirical methods (e.g., Taxim): supervised by real data, but poor generalization to novel geometries.
- Physics-based methods (e.g., FEM): high fidelity, but computationally prohibitive for large-scale RL.
An additional blind spot: most simulators assume flat sensor surfaces. Curved fingertips (common in anthropomorphic hands) cause projection distortions that existing tools handle poorly.
2. Method
2.1 Core Insight: Deform Map as Common Geometric Space
Raw tactile images differ wildly between sim and real due to optics. But the underlying deformation geometry is the same physical quantity in both domains. Tacmap defines a unified representation: the penetration depth map (deform map) – a pixel-wise map of how deeply an object penetrates the sensor elastomer.
- In simulation: compute the deform map analytically via ray-casting.
- In the real world: train a neural network to translate raw tactile images into deform maps.
Both domains produce the same type of output, eliminating the need to simulate complex optical effects.
2.2 Simulation: Geometric Rendering Pipeline
The sensor geometry is defined by two surfaces:
- Undeformed sensor surface $S_u$: the physical resting shape of the elastomer.
- Virtual sensing surface $S_s$: positioned at a fixed offset exterior to the sensor, defining the interaction zone.
For each pixel $(u,v)$ on an $H \times W$ grid over $S_s$, a ray is cast along the surface normal toward the sensor interior. The deformation value is:
\[d(u,v) = \max(0, z_s - \max(z_u, z_o))\]where $z_s$ is the ray origin on $S_s$, $z_u$ is the undeformed surface coordinate, and $z_o$ is the first intersection with the object mesh. This naturally handles curved fingertips by computing in local normal-projection space rather than assuming a flat plane.
2.3 Real World: Automated Data Collection + Translation
An automated 3-axis motion stage presses geometric indenters into the real sensor under controlled conditions. The indenter’s precise 3D pose is recorded, and ground-truth deform maps are computed using the same geometric projection logic as simulation. This yields a paired dataset $\mathcal{D} = {(I_{\text{raw}}^{(i)}, M_{\text{gt}}^{(i)})}$.
A ResNet-based encoder-decoder is trained to map raw tactile images to deform maps: $\hat{M} = \Phi(I_{\text{raw}})$, minimizing pixel-wise MSE against the kinematically-derived ground truth.
2.4 Three Tactile Information Streams
Tacmap provides three synchronized signals:
- Net Force $F$: from physics engine in sim; from a trained regression network in real.
- Contact Position $P$: from contact sensor in sim; from centroid of deform map in real.
- Deform Map $M$ (the main contribution): dense, pixel-wise penetration depth.
3. Implementation
Integrated into both Isaac Lab and MuJoCo:
- A Multi-Mesh Ray Caster pre-defines tactile sensing points and directions on the sensor surface.
- GPU-accelerated ray-casting computes penetration depth in parallel across thousands of environments.
- In Isaac Lab: uses the Raycaster API for massive parallelism. In MuJoCo: uses
mj_rayfunction.
The tactile sensing resolution is decoupled from physics collision mesh resolution, so high-fidelity tactile feedback doesn’t compromise physics solver stability.
4. Experiments and Results
Sim-to-Real Fidelity
Tested with cylindrical and square indenters on the SharpaWave hand’s tactile fingertips:
| Object | Contact Position Error | Deform Depth Error | Net Force $L_2$ Error | Deform IoU |
|---|---|---|---|---|
| Square | 0.66 mm | 18.53% | 0.28 N | 88.21% |
| Cylinder | 0.96 mm | 14.71% | 0.61 N | 85.67% |
The simulated and real deform maps show “remarkable structural similarity” across compression sequences. Force alignment between sim and real is highly correlated.
Computational Efficiency
- GPU memory: near-linear growth from 16 to 8192 parallel environments (ray-casting is much lighter than FEM).
- Rendering throughput: negligible degradation of overall simulation speed even with thousands of concurrent environments.
- Trainable on a single consumer-grade GPU.
Zero-Shot Sim-to-Real Transfer: In-Hand Rotation
- Policy trained with PPO exclusively in simulation, using the Tacmap stream as observation.
- Deployed directly on the physical SharpaWave hand without any real-world fine-tuning.
- Successfully achieves smooth, continuous in-hand rotation of a spherical object.
- The policy interprets real-world tactile images (translated to deform maps) and performs proactive finger coordination to prevent slips.
5. Strengths
- Elegant core idea: abstract away sensor-specific optics, align sim and real in a shared geometric space. Simple and effective.
- Geometry-agnostic: works for both flat and curved sensor surfaces via normal-projection space, unlike most existing simulators.
- Computationally efficient: ray-casting is orders of magnitude cheaper than FEM while maintaining sufficient physical fidelity for policy transfer.
- Practical validation: zero-shot transfer of a contact-rich RL policy (in-hand rotation) is a strong demonstration.
- Dual-engine support: works in both Isaac Lab (for massive parallel RL) and MuJoCo.
6. Limitations
- No shear/tangential force modeling: Tacmap only captures normal penetration depth. Shear strain and lateral forces (critical for slip prediction) are not represented.
- Single downstream task: only in-hand rotation is demonstrated for sim-to-real transfer. More diverse tasks (assembly, insertion) would strengthen the claims.
- Ray-casting scales with mesh complexity: as object meshes become more detailed, ray-casting overhead grows. Advanced acceleration structures are mentioned as future work.
- Translation network generalization: the real-world image-to-deform translation network is trained on a limited set of geometric indenters. Generalization to arbitrary object shapes in the wild is not extensively tested.
7. Takeaways
- The deform map is a clever abstraction layer. By standardizing both sim and real into penetration depth, you sidestep the hardest part of tactile sim-to-real (reproducing optical phenomena) and focus on what actually matters for control: contact geometry.
- This is the third paper from Sharpa in my recent reading (after DexEMG and SaTA). Together they form a coherent stack: Tacmap for tactile sim-to-real, SaTA for spatially-anchored tactile policy learning, and DexEMG for lightweight teleoperation. All targeting the SharpaWave dexterous hand.
- The approach is complementary to SaTA: Tacmap provides the sim-to-real bridge for training tactile policies in simulation, while SaTA provides the representation for using tactile data effectively at deployment. Combining them could enable sim-trained policies with spatially-anchored tactile reasoning.
- The main open question is shear force. Normal penetration depth captures a lot, but tasks requiring slip detection or delicate force modulation need tangential information. The authors acknowledge this as a key direction.