[Paper Notes] HACo: Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A finger can stop against an object while its position command keeps moving. The measured joint configuration records the stopped finger; the command also carries the offset that produces contact force. This difference makes action labels a central problem in learning dexterous compliance.
HACo learns motion references produced by compliance-regulated teleoperation. It combines fingertip tactile sensing with joint-torque feedback, then lets the action model query those signals through gated cross-attention. An auxiliary target preserves the difference between the compliant command and the measured hand state. On five real bimanual tasks, the authors report 83% mean success versus 35% for T-Rex, with 20 trials per task. The most useful contribution for reproduction is the explicit definition of what the policy should imitate under contact.
Paper and source version
HACo: Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation is by Naisheng Ye, Yinzhe Zhou, Junkai Zhao, Yuhang Lu, Checheng Yu, Zhenjie Yang, Pengwei Wang, and Hongyang Li, from the University of Hong Kong, Beijing Academy of Artificial Intelligence, and Johns Hopkins University. These notes follow the nine-page arXiv:2609.36596v1 PDF, submitted September 29, 2026. The arXiv record identifies it as a preprint; the project page provides demonstrations. Experimental numbers below are author-reported.
1. Three hand configurations, three meanings
Consider sliding a second playing card into a hand that already holds one. Excessive normal force raises friction and blocks insertion; insufficient force loses the existing grasp. Visually similar finger poses can correspond to very different loads.
Section III distinguishes the following quantities:
| Quantity | Meaning | Role in learning |
|---|---|---|
| $q_t^{\mathrm{nom}}$ | Nominal command from human-motion retargeting | Preserves motion intent, but can encode excessive loading |
| $q_t^{\mathrm{obs}}$ | Measured hand configuration | Records realized motion, including motion blocked by contact |
| $q_t^{\mathrm{cmp}}$ | Reference after compliance regulation | Executable hand-action target |
Learning the next observed configuration alone omits the command offset that can maintain a grasp. Learning the nominal command can reproduce the operator’s excessive pressure. HACo records the regulated reference and learns it directly. Its action is $a_t^{\mathrm{cmp}}=[x_t^{\mathrm{cmp}},q_t^{\mathrm{cmp}}]$, combining the arm end-effector reference and hand joint reference. The outputs go to the existing low-level controllers.
This gives “active compliance” a concrete meaning here: physical feedback changes the generated motion reference as contact evolves. The policy does not output a prescribed force trajectory or learned stiffness matrix.
2. Constructing compliant demonstrations
The demonstration system has a low-level position/impedance controller and an outer regulator that adjusts the operator’s nominal references before execution. The arm and hand use different corrections (PDF, Section III-A).
For the arm, only the part of the estimated end-effector wrench outside an admissible range drives the Cartesian admittance response:
\[M_a\delta\ddot{x}_t=-D_a\delta\dot{x}_t-K_a\delta x_t-r_w(\hat w_t), \qquad x_t^{\mathrm{cmp}}=x_t^{\mathrm{nom}}+\delta x_t.\]Here $r_w$ extracts the excess wrench; $M_a,D_a,K_a$ specify virtual mass, damping, and stiffness. When the wrench returns inside the admissible range, the forcing term vanishes and the displacement relaxes under those dynamics.
For each active fingertip, the measured force $f_{i,t}$ is clipped in magnitude, preserving its direction, to obtain $\bar f_{i,t}$. A virtual gain converts the requested unloading into a fingertip displacement:
\[d_{i,t}=K_i^{-1}(\bar f_{i,t}-f_{i,t}).\]The hand then solves for a small, smooth joint correction:
\[\begin{aligned} \delta q_t^\star=\arg\min_{\delta q}\;& \sum_i w_{i,t}\|J_{i,t}\delta q-d_{i,t}\|_2^2\\ &+\lambda_r\|\delta q\|_2^2 +\lambda_s\|\delta q-\delta q_{t-1}^\star\|_2^2,\\ q_t^{\mathrm{cmp}}&=q_t^{\mathrm{nom}}+\delta q_t^\star. \end{aligned}\]$J_{i,t}$ maps joint displacement to fingertip displacement, and $w_{i,t}$ smoothly activates a contact’s objective. Joint-position and update limits also constrain the optimization. The two regularizers discourage large corrections and abrupt changes between time steps. Because fingers are kinematically coupled, satisfying one contact can move other fingertips.
The paper explicitly treats $K_i$ as a virtual gain, without identifying the object’s physical stiffness. This regulator supplies the training targets. Its thresholds and gains therefore shape the behavior available for imitation; recording extra tactile channels alone would not reconstruct that supervision.
3. Compliant intent is an auxiliary training target
HACo defines the time-aligned difference
\[\Delta q_t^{\mathrm{ci}}=q_t^{\mathrm{cmp}}-q_t^{\mathrm{obs}}.\]This is distinct from the teleoperation correction $\delta q_t^\star=q_t^{\mathrm{cmp}}-q_t^{\mathrm{nom}}$. The correction measures how regulation changed the human command; compliant intent measures where the executable command lies relative to the actual hand.
Under sustained contact, the authors interpret $\Delta q_t^{\mathrm{ci}}$ as carrying force-regulating information. That interpretation has a condition: tracking lag, friction, and actuator dynamics can also contribute to command-state error. It should not be treated as a calibrated force measurement.
Training concatenates compliant-action and intent trajectories, normalized with separate dataset statistics:
\[Y_1=[A^{\mathrm{cmp}};\Delta q^{\mathrm{ci}}],\qquad Y_0\sim\mathcal N(0,I),\] \[Y_s=(1-s)Y_0+sY_1,\qquad v^\star=Y_1-Y_0, \qquad s\sim\mathcal U[0,1].\]Conditional flow matching trains a velocity predictor $v_\theta(Y_s,s,O_t)$ with action and intent losses:
\[\mathcal L= \mathbb E\|P_{\mathrm{act}}(v_\theta-v^\star)\|_2^2 +0.5\,\mathbb E\|P_{\mathrm{ci}}(v_\theta-v^\star)\|_2^2.\]The projections select the two channel groups; losses are averaged over valid time steps and channels. Only the predicted compliant action is executed. Adding the auxiliary intent to that action would alter the reference that the model was trained to produce.
HACo predicts 40-step chunks. Real-Time Chunking uses a sampled clean prefix of at most 12 steps during training, with losses on the remaining postfix. At inference, a 10-step prefix committed from the preceding chunk conditions the next chunk, whose continuation uses the latest observation. These are step counts; the paper does not establish a control frequency from which to convert them into milliseconds.
flowchart TD
A[Human motion retargeting] --> B[Arm admittance and fingertip force regulation]
B --> C[Compliant arm and hand references]
C --> D[Action supervision]
C --> E[Subtract synchronized measured hand state]
E --> F[Auxiliary compliant-intent supervision]
G[Fingertip tactile and joint-torque histories] --> H[Kinematics-aligned haptic tokens]
H --> I[Gated cross-attention in action expert]
J[Images, language and robot state] --> I
D -. Training .-> I
F -. Training .-> I
I --> K[Predicted compliant action chunk]
K --> L[Low-level arm and hand controllers]
L --> G
Reading diagram of the paper’s training and execution paths. The auxiliary intent branch supplies supervision; the executed output is the compliant action.
4. Linking touch to the joints that carry its load
The platform provides ten fingertip tactile streams and 44 hand-joint torque measurements. Each finger’s tactile encoder receives nine steps of wrench history plus the current $240\times240$ deformation map. Its torque encoder processes the nine-step history of joints on that finger’s kinematic chain.
The two features are fused within each finger first. Hand and digit identity embeddings are added, then a Transformer exchanges information across the ten finger tokens. Validity masks handle unavailable measurements. This preserves the correspondence between a localized fingertip contact and the loading of its own joints before modeling coordinated grasping.
Joint torque broadens the available evidence, including loads from palm or phalange contacts outside fingertip coverage. It does not uniquely locate those contacts. The tactile stream supplies local contact structure; torque supplies a spatially ambiguous signature of transmitted load.
The Compliance Grounding Module (CGM) lets each adapted action layer query the resulting haptic memory $P_t$:
\[C_t^l=\operatorname{Attn}\bigl(\operatorname{AdaLN}(Z^l,s),P_t,P_t;m_t\bigr), \qquad \widetilde Z^l=Z^l+\tanh(\alpha_l)C_t^l.\]The residual gate $\alpha_l$ starts at zero, so this added branch initially preserves the pretrained computation. Fine-tuning learns how much haptic information to inject. Here $s$ is the flow time and $m_t$ masks unavailable haptic tokens (PDF, Sections III-C–D).
5. What the experiments establish
Experiments use two UR5 arms with Sharpa hands, one ZED Mini egocentric camera, and two wrist-mounted RealSense D405 cameras. MANUS Metagloves Pro and VIVE Trackers capture the demonstrations. The authors collect 100 demonstrations per task and train for 30,000 steps on four H100 GPUs, with global batch size 48 and seed 42. HACo starts from GR00T N1.7; its AdamW learning rate is $2\times10^{-5}$, with weight decay $10^{-5}$ and 5% warmup.
The following condenses Table I. Each task-method pair has 20 physical trials.
| Task | Main interaction demand | T-Rex | HACo |
|---|---|---|---|
| Insert poker cards | Friction across several contacts | 4/20 | 18/20 |
| Open book | Tangential force to separate pages | 6/20 | 17/20 |
| Draw on balloon | Contact on a fragile curved surface | 2/20 | 14/20 |
| Unscrew cap | Coordinated rotational torque | 12/20 | 19/20 |
| Squeeze toothpaste | Compression and dispensing | 11/20 | 15/20 |
| Macro mean | 35% | 83% |
GR00T, GR00T + Tactile, and ViTacFormer average 15%, 22%, and 4%, respectively. The 83% versus 35% comparison is a 48-percentage-point gain. These are complete policy comparisons: pretraining, architecture, and action supervision differ across methods, so the total gain cannot be assigned to one module.
The ablations are more informative about the mechanism:
| HACo variant | Mean success |
|---|---|
| No haptic feedback | 27% |
| Torque feedback only | 45% |
| Fingertip tactile feedback only | 68% |
| Both modalities, without coupled encoding | 70% |
| Nominal action, without intent supervision | 59% |
| Compliant action, without intent supervision | 73% |
| CGM without its gate | 66% |
| Full HACo | 83% |
Changing the action target raises success from 59% to 73% in the no-intent setting; adding intent supervision then reaches 83%. These are sequential comparisons, not independent additive effects. The authors also report 19% lower mean fingertip force over contact-active samples for full HACo relative to the nominal-action variant. That aggregate supports reduced loading, but does not give peak-force bounds.
Direct visual-haptic fusion reaches 41%, and appending haptic features to the action sequence reaches 47%. Simply making haptic data available therefore falls short of the full design in this experiment. Removing wrist cameras gives 76%; tactile feedback helps with occluded contact while visual placement still matters.
6. Limits and a reproduction priority
The evidence covers five tasks on one bimanual platform. Each task’s success rate moves by five percentage points when one of its 20 outcomes changes. Training uses one reported seed, and the tables provide no confidence intervals. The PDF does not establish transfer across hands, a stability guarantee under arbitrary contacts, or a maximum safe contact force.
The project’s failure examples help locate the remaining problems: poor thumb placement fails to separate pages; missed marker contact destabilizes a balloon; excessive pickup pressure lifts two cards together; nozzle misalignment dispenses toothpaste beside the brush. Success requires contact timing and visual alignment as well as load regulation.
The authors identify force-feedback teleoperation, learning beyond retargetable demonstrations, and dense whole-hand tactile coverage as open directions. Their current operator interface captures motion while the regulator adjusts loads, which limits the demonstration of deliberate human force strategies.
For a reproduction, I would first audit synchronized logs of $q^{\mathrm{nom}}$, $q^{\mathrm{cmp}}$, and $q^{\mathrm{obs}}$, then compare nominal and compliant labels with the same policy. That priority follows the 59% to 73% ablation. If those labels are already reliable, the next check is whether each fingertip’s tactile features are paired with the correct joint chain. For deployment, I would add peak-force and contact-transition measurements: a successful rollout can still contain a brief load spike that the reported mean force hides.