A Map of Tactile Simulation for Vision-Based Tactile Robotics
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
After reading Taccel, Taxim, FOTS, TacEx, TacSL, IPC/GIPC/UIPC, and several adjacent tactile simulation papers, my current mental model is that tactile simulation is not one monolithic simulator. It is a layered stack:
- Contact physics: how bodies touch, deform, stick, slip, and avoid penetration.
- Elastomer simulation: how the gel/membrane deforms under contact.
- Sensor rendering: how deformation becomes RGB, depth, normal maps, marker motion, or force fields.
- Robot simulator integration: whether the tactile model lives inside Isaac Sim, Isaac Gym, MuJoCo, PyBullet, Warp, or a custom backend.
- Learning interface: whether the simulator is mainly for rendering, RL, differentiable optimization, or sim-to-real transfer.
Under this lens, Taccel is the most specialized high-performance backend for large-scale vision-based tactile robotics. TacEx is the most direct Isaac Sim / Isaac Lab integration route. TacSL is the strongest Isaac-family route for high-throughput tactile policy learning. Taxim and FOTS mostly live at the tactile image / marker output layer. IPC/GIPC/UIPC are not tactile simulators by themselves, but they are the contact mechanics engine behind several modern soft tactile methods.
Why This Map
The central question is practical: if I want to do vision-based tactile simulation for dexterous manipulation, especially around Isaac Sim / Isaac Lab, which papers are actually the core ones?
The short answer is: yes, the cluster is fairly compact, but it helps to add a few older or adjacent baselines. My recommended reading set is:
- Taccel for scalable high-performance VBTS robotics.
- TacEx for Isaac Sim / Isaac Lab integration with GelSight-style tactile sensors.
- TacSL for Isaac/GPU-based tactile policy learning.
- Taxim and FOTS for optical tactile RGB and marker motion.
- IPC / GIPC / UIPC / TacIPC for robust soft-contact mechanics.
- TACTO as the classic fast open-source RGB/depth tactile simulator baseline.
- DiffTactile, Xu et al. CoRL 2022, Tacchi / Tacchi 2.0, and Tactile Gym 2.0 as useful side branches for differentiability, low-cost elastomer simulation, and tactile RL benchmarks.
The Layered View
The most useful way to compare these papers is to ask where each one sits in the tactile simulation stack.
flowchart TD
A["Robot and Scene Simulator<br/>Isaac Sim, Isaac Gym, MuJoCo, PyBullet, Warp, custom"] --> B["Contact Mechanics<br/>Penalty, FEM, MPM, IPC, GIPC, UIPC"]
B --> C["Elastomer / Gel Deformation<br/>height map, mesh deformation, force field, marker displacement"]
C --> D["Sensor Output<br/>RGB, depth, normals, marker flow, force distribution"]
D --> E["Learning / Evaluation<br/>RL, sim-to-real, differentiable optimization, perception"]
Different papers choose different shortcuts. TACTO prioritizes fast RGB/depth rendering and delegates contact dynamics to external physics engines. Taxim calibrates the optical response with example-based lookup tables. FOTS uses a learned optical mapping and a fast marker motion approximation. TacIPC, Taccel, and TacEx move more weight into physically based elastomer/contact simulation. TacSL chooses a fast approximation that scales well for policy learning. DiffTactile makes the physics differentiable so that gradients can be used for identification and control.
Core Papers
Taccel
Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation is a high-performance simulator for robots equipped with vision-based tactile sensors. In the local codebase, Taccel is not an Isaac Sim extension. It is built around NVIDIA Warp and a custom warp_ipc backend. The repo exposes TaccelModel, TactileRobot, VBTS models, and examples such as peg insertion, Tac-Man, and soft-object grasping.
The core physics idea is a combination of:
- ABD (Affine Body Dynamics) for efficient rigid or stiff body representation. A body pose is represented as (x = AX + p), with 3 translation parameters and 9 affine parameters in the matrix (A).
- FEM soft gel for tactile elastomers.
- Neo-Hookean hyperelasticity for rubber-like deformation.
- IPC (Incremental Potential Contact) for robust, penetration-avoiding contact and friction.
- GPU parallelism via Warp for thousands of parallel environments.
Taccel’s important distinction is that it treats tactile simulation as a full robotics simulation problem, not just an image rendering problem. It is designed to simulate robot/object motion, soft gel deformation, tactile RGB, depth, normal maps, and marker/flow signals at scale. For dexterous tactile robotics, it is the most specialized system in this list, but as of this code inspection it does not provide a ready Isaac Sim bridge.
Taxim
Taxim: An Example-based Simulation Model for GelSight Tactile Sensors targets GelSight-style optical tactile RGB simulation. Its key idea is pragmatic calibration rather than full optical modeling. Given deformation/contact geometry, Taxim approximates the elastomer response, computes surface normals, and uses a polynomial lookup table to map local geometry and location to RGB intensity. It also includes marker motion through linear elastic deformation theory and superposition.
Taxim is valuable because it separates the sensor appearance problem from the full contact mechanics problem. It does not try to be a full FEM contact simulator. Instead, it asks: if I know the gel surface geometry, can I synthesize a realistic GelSight image quickly and calibrate it from a modest number of real examples? For many workflows, that is exactly the right abstraction.
FOTS
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills is another sensor-output-level simulator, but with a different modeling style from Taxim. FOTS uses an MLP mapping plus planar shadow generation for optical tactile RGB, and a marker distribution approximation for marker motion. It explicitly targets fast online use and sim-to-real tactile-motor learning.
The key difference from Taxim is that Taxim uses a polynomial lookup table, while FOTS uses a learned optical mapping and a fast approximate model for marker displacement under normal, shear, and twist loads. FOTS is useful when marker motion matters and the goal is efficient tactile-motor policy training rather than the most complete soft-body mechanics.
IPC, GIPC, UIPC, and TacIPC
IPC is the underlying contact formulation: each time step is solved as an optimization problem with inertia, elasticity, contact barrier energy, friction, and constraints. The barrier energy grows as surfaces approach penetration, which makes the simulation intersection-free and inversion-free when solved properly.
GIPC makes IPC more GPU-friendly and efficient by reformulating the barrier Hessian using analytic eigensystems and robust geometric contact measures. UIPC / libuipc packages a unified GPU IPC framework for rigid bodies, soft bodies, cloth, threads, and their couplings. These tools are not “tactile renderers,” but they are the mechanics layer that makes physically plausible soft tactile simulation possible.
TacIPC is especially relevant because it directly applies FEM + IPC to optical tactile elastomer simulation. It sits between the general IPC literature and systems such as Taccel or TacEx. If the question is “why are people using IPC for tactile sensors?”, TacIPC is one of the cleanest bridges.
TacEx
TacEx: GelSight Tactile Simulation in Isaac Sim - Combining Soft-Body and Visuotactile Simulators is the most direct public route for using GelSight-like tactile simulation inside Isaac Sim / Isaac Lab. It is modular: Isaac Sim handles robot scene setup, rendering, PhysX, cameras, and Isaac Lab environments; UIPC/GIPC-style soft-body simulation handles the gel/contact side; Taxim generates tactile RGB; FOTS generates marker motion.
This makes TacEx architecturally attractive for Isaac workflows. If the goal is “I want a tactile sensor inside Isaac Sim right now,” TacEx is closer than Taccel. The tradeoff is that TacEx is an integration framework: its physical fidelity and scalability depend on which physics option is used. GIPC/UIPC improves soft contact quality, but camera/rendering and soft-body memory costs can limit very large parallel RL setups.
TacSL
TacSL: A Library for Visuotactile Sensor Simulation and Learning lives on the Isaac Gym / Isaac simulator side and is optimized for large-scale tactile policy learning. It simulates visuotactile images and contact-force distributions efficiently, and it includes learning algorithms such as asymmetric actor-critic distillation (AACD).
TacSL’s central choice is speed. It uses a simplified soft-contact approximation suitable for many parallel training environments, then exposes RGB/contact-force observations to RL and distillation pipelines. It is less about high-fidelity FEM gel mechanics and more about making tactile learning scalable enough for sim-to-real policies.
TACTO
TACTO is the classic open-source baseline for high-resolution vision-based tactile simulation. It supports sensors such as DIGIT and OmniTact and integrates with PyBullet. Its own README is candid: it is not meant to provide physically accurate contact dynamics such as deformation and friction; instead, it relies on existing physics engines and focuses on rendering tactile readings.
This limitation is also why TACTO remains useful. It is simple, fast, configurable, and good for prototyping tactile perception/control pipelines. In a reading map, TACTO is the “fast rendering baseline” against which later systems like Taxim, FOTS, TacIPC, TacEx, TacSL, and Taccel can be understood.
Differentiable and Low-Cost Branches
Efficient Tactile Simulation with Differentiability for Robotic Manipulation by Xu et al. focuses on dense tactile normal and shear force fields, with analytical gradients that can accelerate policy learning. It is not primarily a GelSight RGB renderer. Its value is in representing tactile feedback as a differentiable force field over contact surfaces.
DiffTactile pushes further into differentiable physics. It combines FEM-based elastomer simulation, multi-material object simulation, penalty contact, and an optical-response module. Its differentiability supports gradient-based parameter identification and tactile-assisted manipulation learning.
Tacchi and Tacchi 2.0 are lower-cost elastomer simulation approaches based around particle/MPM-style modeling with Taichi. Tacchi 2.0 adds a pinhole camera model and marker motion images, supporting pressing, slipping, and rotating contacts. This branch is useful when full FEM/IPC is too heavy but a pure renderer is too weak.
Tactile Gym 2.0 is less a physics core than a benchmark and sim-to-real RL environment. It is valuable when comparing low-cost high-resolution sensors such as TacTip, DIGIT, and DigiTac across tactile tasks.
Comparison Table
| Method | Main role | Simulator ecosystem | Contact / gel model | Tactile output | Best use |
|---|---|---|---|---|---|
| TACTO | Fast tactile renderer baseline | PyBullet + renderer | Delegates physics; not physically accurate contact dynamics | RGB, depth | Prototyping, perception/control baselines |
| Taxim | GelSight optical RGB + marker model | Can plug into other simulators | Approximate deformation + calibrated optical LUT | RGB, marker motion | Realistic GelSight appearance |
| FOTS | Fast optical tactile + marker simulator | Integrated with MuJoCo-like engines | Approximate contact/marker model | RGB, marker displacement | Sim-to-real tactile-motor learning |
| TacIPC | FEM + IPC elastomer simulation | Integrable with existing simulators | FEM + IPC | Deformation, pseudo-image, marker displacement | Robust optical tactile elastomer physics |
| Taccel | High-performance tactile robotics simulator | NVIDIA Warp + custom warp_ipc | ABD + FEM + Neo-Hookean + IPC | RGB, depth, normals, marker/flow | Scalable high-fidelity VBTS robotics |
| TacEx | Isaac Sim tactile extension | Isaac Sim + Isaac Lab | PhysX options + UIPC/GIPC soft body | Taxim RGB, FOTS marker motion | GelSight Mini inside Isaac Sim |
| TacSL | Tactile simulation and learning library | Isaac Gym / Isaac simulator | Fast soft-contact approximation | RGB, normal/shear force fields | Large-scale tactile RL and sim-to-real |
| DiffTactile | Differentiable tactile simulator | Custom differentiable stack | FEM elastomer + penalty contact + multi-material objects | Dense tactile feedback, optical response | Gradient-based identification and control |
| Tacchi / Tacchi 2.0 | Low-cost elastomer simulator | Taichi, MuJoCo/Gazebo integration | Particle/MPM-style elastomer | RGB, marker images | Lightweight optical tactile simulation |
| Tactile Gym 2.0 | Tactile RL benchmark | Gym-style environments | Sensor/task abstraction | Sensor images for TacTip/DIGIT/DigiTac | Comparing sensors and sim-to-real RL |
What This Means for Isaac Sim and Dexterous Hands
For an Isaac Sim + dexterous hand project, I would separate the decision into three cases.
If the priority is immediate Isaac Sim integration, start with TacEx. It already aims to bring VBTS into Isaac Sim / Isaac Lab and explicitly combines Taxim, FOTS, and UIPC/GIPC-style soft-body simulation.
If the priority is fast policy learning, study TacSL. It is more learning-oriented than TacEx and includes the RL/distillation machinery needed for tactile policies.
If the priority is high-fidelity soft tactile contact across many sensors or contact-rich hands, study Taccel and TacIPC/GIPC/UIPC. Taccel is not an Isaac Sim frontend, but its backend design is the most directly aimed at large-scale VBTS robotics. A practical bridge would likely use Isaac Sim for the robot scene and Taccel as a local tactile backend, or use Taccel for physics and Isaac Sim mainly for visualization.
The important engineering point is that “using Taccel in Isaac Sim” is not just an import statement. Someone has to own the synchronization layer:
flowchart TD
A["Isaac Sim / Isaac Lab<br/>robot articulation, object pose, scene"] --> B["State Adapter<br/>link poses, joint states, local contact frames"]
B --> C["Taccel / Warp-IPC<br/>gel FEM, ABD objects, IPC contact"]
C --> D["Tactile Outputs<br/>RGB, depth, normals, marker flow"]
D --> E["Policy / Logger / Isaac Observation Wrapper"]
That adapter is exactly what is missing from the current local Taccel repo.
Takeaways
The tactile simulation literature is easier to navigate once it is split into layers. TACTO, Taxim, and FOTS mostly answer “how do I produce tactile images or marker motion?” TacIPC, IPC, GIPC, and UIPC answer “how do I make deformable contact robust?” TacEx and TacSL answer “how do I put tactile sensing into Isaac-family simulators and learning workflows?” Taccel asks the larger systems question: “can we make high-fidelity vision-based tactile robotics scalable?”
For my own roadmap, I would read the field in this order:
- TACTO, Taxim, and FOTS to understand tactile rendering.
- IPC, TacIPC, GIPC, and UIPC to understand contact mechanics.
- TacEx and TacSL to understand Isaac ecosystem integration.
- Taccel to understand the high-performance backend design.
- DiffTactile, Xu et al., Tacchi, and Tactile Gym 2.0 as alternative routes for differentiability, lightweight simulation, and benchmarks.
References
- Yuyang Li, Wenxin Du, Chang Yu, Puhao Li, Zihang Zhao, Tengyu Liu, Chenfanfu Jiang, Yixin Zhu, and Siyuan Huang. Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation. arXiv:2504.12908, 2025. Paper, Project, Code
- Zilin Si and Wenzhen Yuan. Taxim: An Example-based Simulation Model for GelSight Tactile Sensors. arXiv:2109.04027, 2021. Paper, Code
- Yongqiang Zhao, Kun Qian, Boyi Duan, and Shan Luo. FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills. arXiv:2404.19217, 2024. Paper, Code
- Duc Huy Nguyen, Tim Schneider, Guillaume Duret, Alap Kshirsagar, Boris Belousov, and Jan Peters. TacEx: GelSight Tactile Simulation in Isaac Sim - Combining Soft-Body and Visuotactile Simulators. arXiv:2411.04776, 2024. Paper, Code
- Iretiayo Akinola, Jie Xu, Jan Carius, Dieter Fox, and Yashraj Narang. TacSL: A Library for Visuotactile Sensor Simulation and Learning. arXiv:2408.06506, 2024. Paper, Project
- Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra. TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors. IEEE RA-L / ICRA, 2022. Paper, Code
- Wenxin Du, Wenqiang Xu, Jieji Ren, Zhenjun Yu, and Cewu Lu. TacIPC: Intersection- and Inversion-free FEM-based Elastomer Simulation For Optical Tactile Sensors. arXiv:2311.05843, 2023. Paper
- Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M. Kaufman. Incremental Potential Contact: Intersection- and Inversion-free Large Deformation Dynamics. ACM TOG / SIGGRAPH, 2020. Project
- Kemeng Huang, Floyd M. Chitalu, Huancheng Lin, and Taku Komura. GIPC: Fast and Stable Gauss-Newton Optimization of IPC Barrier Energy. ACM TOG, 2024. Project
- spiriMirror contributors. libuipc: A Modern C++20 Library of Unified Incremental Potential Contact. Code, Docs
- Jie Xu, Sangwoon Kim, Tao Chen, Alberto Rodriguez Garcia, Pulkit Agrawal, Wojciech Matusik, and Shinjiro Sueda. Efficient Tactile Simulation with Differentiability for Robotic Manipulation. CoRL, 2022. Project
- Zilin Si, Gu Zhang, Qingwei Ben, Branden Romero, Zhou Xian, Chao Liu, and Chuang Gan. DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation. ICLR, 2024. Paper, Project
- Zixi Chen, Shixin Zhang, Shan Luo, Fuchun Sun, and Bin Fang. Tacchi: A Pluggable and Low Computational Cost Elastomer Deformation Simulator for Optical Tactile Sensors. IEEE RA-L, 2023. Paper
- Yuhao Sun, Shixin Zhang, Wenzhuang Li, Jie Zhao, Jianhua Shan, Zirong Shen, Zixi Chen, Fuchun Sun, Di Guo, and Bin Fang. Tacchi 2.0: A Low Computational Cost and Comprehensive Dynamic Contact Simulator for Vision-based Tactile Sensors. arXiv:2503.09100, 2025. Paper
- Yijiong Lin, John Lloyd, Alex Church, and Nathan F. Lepora. Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch. arXiv:2207.10763, 2022. Paper, Code
- Yashraj Narang, Balakumar Sundaralingam, Miles Macklin, Arsalan Mousavian, and Dieter Fox. Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections. ICRA, 2021. Paper