[Paper Notes] TactiDex: A Real-World Tactile-Guided Benchmark for Human-Like Dexterous Manipulation
Published:
TactiDex asks a precise question: if a robot reproduces the geometry of a human hand trajectory but touches the object with the wrong fingers, timing, or force, has the skill really been transferred? The paper’s answer is no. It contributes a human hand–object interaction dataset with synchronized whole-hand pressure, kinematics, and object motion, then uses those tactile traces as structured supervision for reinforcement-learning-based transfer.
The most important boundary is easy to miss: TactiSkill is tactile-guided during data processing and policy training, while the reported real-robot deployment does not use online tactile feedback. Human tactile references define desired contact patterns, simulated contact forces shape the reward and the critic, and the learned trajectory is retargeted to physical hands for open-loop execution. The work therefore demonstrates tactile-supervised trajectory generation and sim-to-real transfer; closed-loop tactile correction remains future work.
Paper Info
The paper is “TactiDex: A Real-World Tactile-Guided Benchmark for Human-Like Dexterous Manipulation” by Suting Ni, Hanbing Zhang, Zhenyu Wei, Guo Chen, Chixuan Zhang, Ye Shi, and Jingya Wang from ShanghaiTech University and InstAdapt. It is accepted to ACM Multimedia 2026. These notes refer to arXiv:2607.09190v1, submitted on July 10, 2026.
The contribution has two coupled parts:
- TactiDex: a tactile-rich human hand–object interaction dataset and contact-aware benchmark;
- TactiSkill: a residual RL transfer method with a three-component tactile reward.
Why Kinematic Matching Is Incomplete
Human-to-robot transfer commonly optimizes hand pose, joint motion, fingertip position, wrist trajectory, and object trajectory. These targets describe where the hand and object should move. They leave several physically distinct executions indistinguishable:
- a fingertip may hover a few millimeters above the surface;
- a mesh may penetrate the object and still achieve a low joint-space error;
- the correct object motion may be produced with the wrong fingers;
- average force may look reasonable while short destructive spikes occur;
- a visually similar grasp may distribute load very differently across the hand.
TactiDex treats contact as measured supervision. The transfer objective now includes which fingers should contact, when contact should occur, how force should be distributed, and whether transient forces remain safe.
TactiDex Dataset
The dataset combines high-precision motion capture with a dual-glove system. An inner motion-tracking glove records articulated hand kinematics; an outer tactile glove records spatial pressure across the whole hand. OptiTrack tracks wrists and objects in a shared global coordinate frame, and objects are reconstructed as simulation-ready meshes.
| Property | TactiDex specification |
|---|---|
| Participants | 10 |
| Objects | 49 calibrated everyday objects |
| Interaction sequences | 757 |
| Scale | approximately 5.1M frames |
| Task forms | left hand, right hand, and bimanual |
| Tactile array | 162 piezoresistive sensing elements across fingertips and palm |
| Tactile sampling | 17 Hz |
| Reported force resolution | up to 0.01 N |
| Motion capture | 8 OptiTrack PX13W cameras at 120 Hz |
| Capture volume | (1.2\,\mathrm{m}\times1.8\,\mathrm{m}) |
| Additional labels | wrist/object 6D pose, MANO hand state, task ID, interaction phases, language description |
The object set includes rigid household items, delicate objects, containers, tools, and paired parts such as lids, rods, and receptacles. This supports functional and long-horizon interactions beyond isolated grasps. The authors report a motion-capture mean ray error of 0.2 mm after calibration.
The streams run at different native rates, so synchronization is a central part of the collection system. An eSync module and the shared capture pipeline align 120 Hz pose/kinematic data with lower-rate tactile measurements at the frame level.
Tactile-Constrained Annotation Refinement
Raw motion capture still produces floating fingers, near-contact ambiguity, and mesh penetration. TactiDex refines the fitted MANO motion in two stages.
First, MANO parameters are fitted to captured 3D keypoints with L-BFGS. Shape is shared across a subject’s frames, while pose regularization and second-order temporal smoothness suppress jitter.
Then tactile evidence determines which geometric relationships count as real contact:
- Contact interval detection: geometric proximity is gated by measured pressure, filtering near-contact frames that look plausible in 3D but carry no force.
- Reference-grasp refinement: a stable frame is selected inside each contact interval. Fingers above the pressure threshold receive stronger surface-attraction weights; inactive fingers are down-weighted to avoid invented contacts.
- Temporal propagation: inverse kinematics propagates the refined grasp through the contact interval while preserving free-space motion elsewhere.
- Collision correction: an object Signed Distance Field removes residual hand–object penetration.
This processing step is one of the paper’s strongest ideas. Touch acts as evidence for annotation, improving the physical validity of geometry before policy learning starts.
TactiSkill: Residual RL as a Force Modulator
TactiSkill starts from a frozen kinematic imitation policy (pi_{\mathrm{base}}) and learns a residual policy (pi_{\mathrm{res}}):
[ \pi^*=\pi_{\mathrm{base}}+\pi_{\mathrm{res}}. ]
The action is a residual joint-position target,
[ a_t=\Delta q_t, ]
executed by a low-level PD controller. The residual policy has a more specific job than generic tracking correction: it adjusts joint configuration so that simulated fingertip forces follow the human tactile reference while the hand and object continue to track their target motion.
1. Human sensor to simulated force
The tactile glove outputs raw ADC values; PhysX exposes contact forces in Newtons. A finger-wise calibration map converts the recorded signals into force references compatible with simulation:
[ F^{\mathrm{human}}_t=\mathcal{M}(\mathrm{ADC}_t). ]
The main text describes (mathcal{M}) as finger-wise and nonlinear, while the appendix gives a calibrated linear form,
[ F=k(\mathrm{ADC}{\mathrm{raw}}-\mathrm{ADC}{\mathrm{offset}}). ]
This mapping is a critical reproducibility interface because sensor calibration directly determines the target force scale.
2. Asymmetric actor–critic
Training uses PPO with asymmetric information. The actor receives deployable proprioception and target references, including the human target tactile signal. The critic additionally receives privileged simulation state: exact object dynamics, center of mass, joint velocity, and real-time simulated fingertip contact forces.
For one 12-DoF Inspire hand, the appendix reports:
| Input | Dimension | Contents |
|---|---|---|
| Proprioceptive state | 46 | joints, sine/cosine encoding, wrist pose and velocity |
| Target reference | 330 | object BPS, target tactile distances, future wrist/hand/object trajectories |
| Privileged critic state | 49 | exact object dynamics, 3D fingertip forces and magnitudes, internal physics |
| Actor input | 376 | proprioception + target |
| Critic input | 425 | actor input + privileged state |
| Action | 12 | residual joint-position targets |
Bimanual dimensions are doubled. Actor and critic are MLPs with hidden sizes ([512,256,128]) and ELU activations.
The Three-Component Tactile Reward
The tactile reward is
[ R_{\mathrm{tactile}} =w_g r_{\mathrm{guide}} +w_a r_{\mathrm{align}} +w_s r_{\mathrm{safe}}. ]
Each term blocks a different failure mode.
Contact guidance: make the intended contact happen
For (N_f) fingers and contact threshold ( au), the policy receives credit when the simulated and human signals both indicate active contact:
[ r_{\mathrm{guide}} =\frac{1}{N_f}\sum_{i=1}^{N_f} \mathbf{1}(F_i^{\mathrm{sim}}>\tau) \mathbf{1}(F_i^{\mathrm{human}}>\tau). ]
This term provides a contact trigger and discourages air grasping. The reported training threshold is 0.3 N.
Human-like alignment: match the force profile
Contact alone is insufficient. TactiSkill aligns the force magnitude of each finger through a bounded Tanh distance:
[ r_{\mathrm{align}} =\frac{1}{N_f}\sum_{i=1}^{N_f} \left[ 1-\tanh\left( \frac{|F_i^{\mathrm{sim}}-F_i^{\mathrm{human}}|}{\sigma} \right) \right]. ]
The bounded metric reduces sensitivity to tactile outliers and preserves useful gradients while encouraging the same cross-finger force distribution as the human demonstration.
Contact safety: suppress force spikes
The safety term applies an exponential penalty when simulated force exceeds an allowed limit:
[ r_{\mathrm{safe}} =\exp\left[ -\lambda\sum_{i=1}^{N_f} \max(0,F_i^{\mathrm{sim}}-F_{\mathrm{limit}})^2 \right]. ]
The body text defines a human-relative limit (F_i^{\mathrm{human}}+\delta); the hyperparameter table also lists a 40 N safety force limit. The intent is consistent: average force alignment needs an additional guard against rare large impulses.
The complete objective combines tactile structure with pose imitation, object tracking, task completion, energy, and velocity regularization:
[ R_{\mathrm{total}} =\lambda_{\mathrm{im}}r_{\mathrm{im}} +\lambda_{\mathrm{task}}r_{\mathrm{task}} +\lambda_{\mathrm{reg}}r_{\mathrm{reg}} +\lambda_{\mathrm{tactile}}R_{\mathrm{tactile}}. ]
Evaluation: Geometry, Touch, and Safety
The main evaluation uses 73 representative sequences spanning single-hand and bimanual tasks. The benchmark separates four dimensions:
- Geometry: object translation/rotation error, MPJPE, and fingertip error;
- Tactile fidelity: mean tactile force error (MTFE) and contact F1;
- Safety: PeakSafe@3N and SafeTac@3N;
- Task outcome: kinematic success (SR_{\mathrm{kin}}) and tactile-aware success (SR_{\mathrm{tac}}).
(SR_{\mathrm{kin}}) requires object rotation error at most (30^\circ), translation error at most 3 cm, MPJPE at most 8 cm, and fingertip error at most 6 cm. (SR_{\mathrm{tac}}) adds MTFE at most 3 N and contact F1 of at least 0.3. For bimanual tasks, both hands must satisfy the spatial conditions.
| Method | (SR_{\mathrm{kin}}) | (SR_{\mathrm{tac}}) | OTE-t | Contact F1 | PeakSafe@3N | SafeTac@3N |
|---|---|---|---|---|---|---|
| Kinematic baseline | 72.91% | 39.35% | 1.1947 cm | 0.5569 | 63.66% | 35.84% |
| TactiSkill without contact bonus | 76.87% | 41.93% | 1.2031 cm | 0.5425 | 57.00% | 35.68% |
| TactiSkill without alignment | 76.37% | 42.20% | 1.0828 cm | 0.5134 | 59.88% | 34.41% |
| TactiSkill without safety | 77.05% | 41.24% | 1.0265 cm | 0.5686 | 68.70% | 38.72% |
| TactiSkill full | 81.95% | 64.64% | 0.9577 cm | 0.7384 | 72.80% | 53.57% |
The full method improves tactile-aware success by 25.29 percentage points over the kinematic baseline and also raises kinematic success by 9.04 points. This supports the paper’s physical-regularization argument: contact supervision helps the policy avoid geometrically convenient but physically invalid optima.
The safety ablation is especially informative. Removing the safety term produces the lowest mean tactile error, 0.0921 N, yet SafeTac@3N falls to 38.72%, compared with 53.57% for the full method. A low average can hide rare destructive events; peak and episode-level metrics are necessary for contact-rich control.
Real-Robot Deployment
The physical platform uses two 7-DoF Franka Panda arms and two Inspire dexterous hands. The simulated hand has 12 DoF, while the physical Inspire hand exposes six independent actuators. An offline optimization maps the simulated trajectory to six actuator commands by balancing fingertip-position matching, proximity to a heuristic anchor mapping, and temporal smoothness:
[ \mathcal{L} =W_{\mathrm{pos}}\mathcal{L}{\mathrm{pos}} +W{\mathrm{anchor}}\mathcal{L}{\mathrm{anchor}} +W{\mathrm{smooth}}\mathcal{L}_{\mathrm{smooth}}. ]
The appendix uses weights 50, 20, and 10. Franka arms track relative wrist delta poses with a 100 Hz operational-space controller; optimized hand commands are sent through a serial interface. The paper shows qualitative single-hand and bimanual tasks including phone use, lid handling, cooking, pouring, and cutting.
The deployment evidence should be read carefully:
- the physical hand’s tactile sensors are not fed back into the policy;
- hand execution is open loop after offline retargeting;
- the paper presents representative videos and snapshots without a real-world trial table or hardware success rate;
- successful transfer therefore validates the usefulness of tactile-shaped simulation trajectories, while online contact recovery is not evaluated.
Strengths
The dataset aligns real pressure, precise hand motion, object pose, mesh geometry, language, and task phases in one capture system. Its tactile-constrained annotation refinement is reusable beyond this specific RL method: measured pressure resolves ambiguity that geometry alone cannot.
The method also assigns clear roles to its three reward components. Guidance establishes contact, alignment shapes the distribution, and safety limits extreme behavior. The ablations show that these roles are complementary, and the safety analysis demonstrates why average tactile error is insufficient.
Finally, the benchmark defines tactile-aware success separately from kinematic success. This exposes a substantial evaluation blind spot: the kinematic baseline reaches 72.91% geometric success while only 39.35% of trials satisfy the tactile criterion.
Limitations and Open Questions
The largest limitation is the absence of robot-side closed-loop touch during deployment. Real sensing could detect slip, unexpected contact, pose error, and material changes, but the current system cannot react to them online.
Several other questions remain:
- The dataset contains 757 sequences and 49 objects, a strong multimodal collection with modest behavioral scale compared with modern video or robot corpora.
- The tactile glove samples at 17 Hz, which may miss fast transients and limits direct study of high-frequency slip.
- Human glove pressure and robot contact force differ in sensor geometry, compliance, contact area, and morphology. The sensor-to-simulation calibration is therefore a strong modeling assumption.
- The dataset records 162 spatial sensing elements across fingers and palm, while TactiSkill largely compresses this signal into per-finger normal-force targets. Palm contact and within-finger pressure geometry remain underused.
- The evaluation is predominantly simulation-based. Physical results lack repeated-trial statistics, object perturbations, unseen objects, and force measurements on the real robot.
- The real Inspire hand is underactuated relative to the simulated model, and the final open-loop optimization can absorb part of the apparent transfer gain.
- “Human-like” is operationalized as agreement with a recorded reference under selected force and contact metrics. The experiments do not establish a universal human contact strategy or physical human-likeness on hardware.
- Some implementation descriptions deserve further clarification, including nonlinear versus linear sensor calibration and the dynamic versus fixed safety-force limit.
Takeaways
TactiDex makes a strong case that contact should enter the human-to-robot pipeline in three places:
- annotation: touch distinguishes real contact from geometric proximity;
- learning: target contact timing and force distribution shape physically plausible policies;
- evaluation: kinematic success must be paired with contact fidelity and peak-force safety.
The next step is a closed loop: use the TactiDex prior to initialize contact behavior, then let real robot tactile sensing correct force and slip during execution. That would extend tactile-guided transfer into tactile-reactive dexterous manipulation.