[Paper Notes] TactAlign: Human-to-Robot Policy Transfer via Tactile Alignment
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
TactAlign transfers human tactile demonstrations to a robot whose tactile sensors have a different layout and signal distribution. It first learns separate self-supervised encoders for a wearable OSMO glove and Xela fingertip sensors. It then builds noisy cross-embodiment correspondences from similar hand-object transitions and trains a rectified flow to transport human tactile latents into the robot latent space. A shared action-chunking policy can consequently learn from both human and robot trajectories.
The method’s useful insight is that touch does not need an exact sample-by-sample correspondence to become transferable. Fingertip motion, object motion, and contact state provide weak semantic anchors. Rectified flow turns those anchors into a dense mapping while tolerating imperfect pseudo-pairs.
Across pivoting, insertion, and lid closing, the full method reaches 79% average success, compared with 38% for robot-only training, 21% without tactile input, and 28% without tactile alignment. With 20 human demonstrations and no robot task demonstrations, it also achieves 100% success on light-bulb screwing; both ablations score 0%. These results make a strong case for alignment, while the evaluation remains limited to one glove–robot sensor pairing, fingertip sensing, ten rollouts per object, and tasks with structured pose information during alignment-data preparation.
Paper Info
The paper is “TactAlign: Human-to-Robot Policy Transfer via Tactile Alignment” by Youngsun Wi, Jessica Yin, Elvis Xiang, Akash Sharma, Jitendra Malik, Mustafa Mukadam, Nima Fazeli, and Tess Hellebrekers, with affiliations spanning the University of Michigan, NVIDIA, Amazon Frontier AI & Robotics, UC Berkeley, the University of Washington, and Microsoft Research. This note covers arXiv:2602.13579v1, posted on February 14, 2026. The project page is yswi.github.io/tactalign.
1. Why Cross-Sensor Touch Is Hard
Human demonstrations are attractive because people manipulate objects quickly, dexterously, and with natural tactile feedback. Yet a glove reading cannot be inserted directly into a robot policy. The two embodiments differ in hand geometry, contact placement, sampling, and sensor physics.
TactAlign uses an OSMO glove, which produces a three-axis magnetic signal at each fingertip with spatial resolution $1\times3$. The robot is a Franka Panda with an Allegro Hand and Xela fingertip sensors, whose readings have resolution $30\times3$. Both are magnetic systems, but OSMO uses particle-based magnetic skin while Xela uses discrete magnets. Their scales, spatial structures, drift behavior, and contact responses therefore differ substantially.
The desired map is
[ g:T^h\rightarrow T^r, ]
where $T^h,T^r\in\mathbb R^d$ are the learned human and robot tactile latent spaces. Training has two offline trajectory collections:
[ H={\mathcal T_1^h,\ldots,\mathcal T_N^h}, \qquad R={\mathcal T_1^r,\ldots,\mathcal T_M^r}. ]
Each timestep contains multi-finger tactile readings $F_t$, fingertip poses $P_t$, and wrist pose $w_t$. A subset of human and robot trajectories also has estimated object pose $o_t$. The robot dataset is intentionally smaller because kinesthetic robot data is more costly to collect.
“Unpaired” has a precise scope here. Human and robot trajectories do not need temporal alignment or matching tactile measurements. Pseudo-pairs are still constructed between demonstrations of the same task, object, reset state, and goal state, using estimated object and hand motion. This is weaker supervision than paired data, though it is stronger than arbitrary collections with no shared task structure.
2. Stage One: Modality-Specific Tactile Representations
TactAlign starts with separate human and robot encoder–decoder models. Each encoder receives a 0.1-second tactile window and is trained through self-supervised reconstruction with mean-squared error:
[ \operatorname{Enc}^h(f_i^h)=h_i\in\mathbb R^d, \qquad \operatorname{Enc}^r(f_i^r)=r_i\in\mathbb R^d. ]
The architecture is JEPA-inspired. A learnable length-one query performs cross-attention pooling so sensors with different token counts produce fixed-dimensional latents. Separate reconstruction objectives preserve the structure of each sensing modality; equal latent dimensionality makes a later cross-domain transport map possible.
Both encoders use roughly ten minutes of play data plus the in-domain alignment dataset. At this stage, the model has two compact tactile representations, but $h_i$ and $r_j$ still lack shared semantics. Raw human features can be actively harmful during joint policy training, as the ablation later shows.
3. Stage Two: Pseudo-Pairs from Interaction
The paper uses observable interaction dynamics as the bridge between embodiments. For a human timestep $i$ and robot timestep $j$, define consecutive fingertip–object transitions
[ O_i^h=(p_i^h,o_i^h,p_{i+1}^h,o_{i+1}^h), \qquad O_j^r=(p_j^r,o_j^r,p_{j+1}^r,o_{j+1}^r). ]
After task-level normalization using robot statistics, their distance is
[ \begin{aligned} S(O_i^h,O_j^r) ={}&\lVert p_i^h-p_j^r\rVert+\lVert o_i^h-o_j^r\rVert
&+\lambda\lVert\widehat{\Delta p_i^h}-\widehat{\Delta p_j^r}\rVert +\lambda\lVert\widehat{\Delta o_i^h}-\widehat{\Delta o_j^r}\rVert. \end{aligned} ]
The first two terms compare current hand and object configuration. The latter two compare their transition directions. The implementation uses $\lambda=1$, selects the three nearest robot transitions for each human transition, and rejects matches beyond $\delta=2.0$. Appendix sweeps show stable alignment over nearby values of both hyperparameters.
A binary contact filter removes semantically implausible pairs. Signal norms below $\delta_h=1200$ for OSMO or $\delta_r=30$ for Xela indicate non-contact. Only contact-to-contact and non-contact-to-non-contact matches survive. This small rule matters because contact onset may cause a large tactile change while producing only subtle kinematic motion.
The resulting set
[ P={(h_i^\ast,r_j^\ast)} ]
contains useful yet noisy correspondences. Similar object motion does not uniquely determine force, shear, finger pressure, or contact patch, so the method treats each match as an initial guide instead of exact ground truth.
4. Rectified Flow as the Alignment Map
Rectified flow learns a time-dependent velocity field $v_\theta(x,t)$ that transports the human latent distribution toward the robot distribution. Using the endpoint-consistent human-to-robot convention, a pseudo-pair defines
[ x_t=(1-t)h_i^\ast+t r_j^\ast, \qquad \dot x_t=r_j^\ast-h_i^\ast, ]
and the velocity-matching objective is
[ \min_{v_\theta} \sum_{(h_i^\ast,r_j^\ast)\in P} \int_0^1 \left| (r_j^\ast-h_i^\ast)-v_\theta(x_t,t) \right|^2dt. ]
At inference, an ODE solver starts at a human latent $x_0=h_i$ and integrates the field to obtain a robot-space feature:
[ g_\theta(h_i)=x_1, \qquad \frac{dx_t}{dt}=v_\theta(x_t,t). ]
The v1 paper prints the interpolation and velocity with reversed endpoints while also stating $x_0=h_i$ and human-to-robot inference. The equations above use the direction consistent with that stated inference procedure.
Why use a flow instead of directly regressing $h_i^\ast\mapsto r_j^\ast$? The pseudo-pairs are many-to-many and noisy. A learned velocity field can “rewire” crossing assignments while fitting a lower-cost distributional transport. In the authors’ synthetic illustration, this property recovers coherent source-to-target structure even when individual pair lines are imperfect.
The alignment network is a three-hidden-layer MLP of width 1,024. It is trained with 100 discretized time steps for 200,000 epochs at learning rate $5\times10^{-5}$, taking about ten minutes on one RTX 4090. Inference uses a vanilla Euler solver.
5. Shared Human–Robot Policy Learning
After alignment, TactAlign trains a shared policy adapted from ACT. For human input, the policy sees aligned fingertip features ${\hat h_{t,k}}{k=1}^K$, human fingertip poses, and wrist pose. For robot input, it sees native robot features ${r{t,k}}_{k=1}^K$ with robot proprioception:
[ \pi_\phi({\hat h_{t,k}},P_t^h,w_t^h)=a_t^h, \qquad \pi_\phi({r_{t,k}},P_t^r,w_t^r)=a_t^r. ]
Both embodiments supervise action chunks expressed in the robot base frame. Depending on the task, actions contain fingertip locations and wrist orientation. During policy training, the tactile encoders and alignment module stay frozen; the policy-side modules learn on their common representation.
The action chunk size is 32. Pivoting, insertion, and lid closing run at 10 Hz, executing 4, 2, and 8 predicted actions per replanning step. Light-bulb screwing runs at 30 Hz and executes 12 actions per step for finer control. Predicted task-space actions are converted to joint commands through inverse kinematics.
This design separates representation alignment from control learning. A single flow trained on pivoting and insertion can be reused when training the lid-closing policy, testing whether tactile semantics transfer to a new task class.
6. Data and Experimental Protocol
The alignment stage uses 200 human demonstrations and 100 robot demonstrations from pivoting and insertion. For policy co-training, each of pivoting, insertion, and lid closing receives 140–160 human demonstrations: 100 demonstrations (about 30 minutes) use the object also seen by the robot, and 20 demonstrations (about five minutes) are collected for each additional human-only object. Each task also has 50 robot demonstrations (about 60 minutes) with one training object.
Objects are split into three categories:
- Seen-by-both: present in human and robot policy data.
- Human-only: present only in human policy data.
- Unseen-by-both: held out from all policy training.
Every object is evaluated with ten policy rollouts. Pivoting and insertion were present during alignment training. Lid closing was absent from both alignment and encoder training, so it measures cross-task reuse of the learned flow.
The light-bulb experiment is more demanding: it uses only 20 human demonstrations, no robot demonstrations for the task, four fingertip positions, a fixed wrist, and a Manus glove augmented with OSMO sensing. Occlusion from the lamp shade makes reliable visual or teleoperated control difficult.
7. Main Results
Latent Alignment
After flow alignment, the Earth Mover’s Distance between human and robot tactile distributions falls from 0.091 to 0.020, a 78% reduction. UMAP visualizations show the two distributions nearly overlapping. Feature magnitude also organizes consistently with contact strength even though force labels are never used for encoder or alignment training.
Human–Robot Co-Training
Across all object categories and tasks, average success is:
| Method | Average success |
|---|---|
| Robot only | 38% |
| Human + robot, no tactile input | 21% |
| Human + robot, raw unaligned tactile features | 28% |
| TactAlign | 79% |
The full model reaches 76% on pivoting, 72% on insertion, and 74% on lid closing. Aggregated by object exposure, it obtains 100% on seen-by-both objects, 71% on human-only objects, and 65.5% on unseen-by-both objects.
The ablations reveal more than a generic benefit from extra human trajectories. Removing tactile input costs 59 percentage points on average. Keeping touch but removing alignment costs 51 points. Raw cross-sensor latents frequently damage learning because a shared policy receives incompatible semantics under a shared feature slot.
Human-Only Dexterous Transfer
On light-bulb screwing, TactAlign succeeds in 10/10 rollouts and takes about 61 seconds on average to illuminate the bulb. The no-tactile policy fails to establish stable contact; the no-alignment policy jams and may fully unscrew the bulb. Both baselines score 0/10.
Force Semantics Without Force Supervision
The authors freeze the tactile models, train a force decoder only on robot latents, and evaluate it on aligned human latents. Alignment reduces cross-sensor force-prediction error by about 96.75% averaged across force axes. Performance approaches the robot-to-robot baseline along two axes, while a larger gap remains along the third. This probe supports the claim that the flow carries physically meaningful contact structure instead of merely improving task-specific action prediction.
8. What the Paper Gets Right
First, the method identifies an intermediate representation problem that policy co-training cannot solve automatically. The “without alignment” baseline is especially valuable: additional tactile data can lower performance when embodiment-specific meanings are mixed.
Second, pseudo-pairs use information that is available during ordinary demonstrations. They require estimated hand and object motion, not force labels, exact temporal synchronization, CAD models, or paired sensor readings. Contact filtering adds a strong physical cue with minimal machinery.
Third, the experiments separate three kinds of transfer: new objects seen only in human data, objects unseen in either domain, and a new task absent from alignment training. The human-only light-bulb result then tests the regime where scalable human collection is most valuable.
Finally, the force probe evaluates latent semantics outside the policy loss. Its gains make the representation claim more credible than success rates alone.
9. Limitations and Open Questions
The largest external-validity limitation is sensor diversity. Experiments use one OSMO–Xela pairing, and both systems are magnetic. Vision-based tactile sensors, full-palm arrays, multiple robot hands, or sensing families with very different physical responses may produce a harder transport problem.
The alignment data is unpaired in time, yet its construction depends on shared task/object conditions, estimated object poses, task-wise normalization, and nearest-neighbor transition matching. Scaling to unconstrained demonstrations with unknown objects, shifting goals, occlusion, or inaccurate pose estimates needs additional machinery.
Evaluation uses ten rollouts per object, leaving considerable statistical uncertainty around individual percentages. The tasks and hardware are real, but the study covers three co-training tasks and one human-only task on one arm–hand platform.
The human signal preprocessing takes the absolute value to handle OSMO sign flips across magnetic-skin quadrants. This removes an ambiguity, while potentially discarding directional structure that future hardware or calibration might retain.
TactAlign addresses tactile and proprioceptive discrepancies. It does not align human and robot vision, nor does it resolve all kinematic and action-space embodiment gaps. A multimodal policy that jointly aligns vision, touch, and embodiment remains an open direction.
Takeaway
TactAlign reframes human tactile transfer as weakly anchored distribution transport. Interaction geometry proposes approximate correspondences; contact state removes obvious mistakes; rectified flow converts the remaining noisy pairs into a smooth human-to-robot map; and a shared policy learns on the aligned space.
The practical lesson is concise: human touch becomes useful robot supervision only after its semantics are made compatible with the robot’s sensors. The full method’s 79% average success, its 51-point gain over unaligned tactile co-training, and the 10/10 human-only light-bulb result show how consequential that representation step can be.