[Paper Notes] FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity
Published:
TL;DR
FACT transfers human dexterous experience through a model that first predicts visual motion and a coarse action, then jointly refines the action and its future force consequences. Its EgoForce dataset adds calibrated human tactile measurements to video and hand motion.
The distinction to watch is the role of force: observed touch supplies current context, while future force is a generated target that interacts with action during denoising. On seven binary real-robot tasks, FACT averages 41.4% success versus T-Rex’s 8.6%. The full ten-task suite also includes musical-note scores and a pipette adjustment metric, which should remain separate.
Paper and reading version
FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity — Zhi Cao, Yining Hong, Jiaolong Yang, Yejin Choi, Li Fei-Fei, Jiajun Wu, Huang Huang. These notes follow arXiv:2610.19155v1, an arXiv preprint, checked on October 25, 2026. PDF · Project
These notes focus on Sections III–V, Figure 3, and Tables I–III. The model is evaluated after skill-specific fine-tuning, not as one unchanged zero-shot policy across the ten tasks.
EgoForce adds contact measurements to human demonstrations
The collection platform combines a head camera, two wrist cameras, tactile gloves, and hand motion capture. A calibration system applies approximately uniform pneumatic pressure from 0 to 50 psi. For each sensing element, pressure and area determine a reference force, $F_i=P_iA_i$, which is paired with the raw A/D reading to fit a calibration mapping.
EgoForce contains approximately 2,000 hours, 40,000 episodes, 1,000 tasks, 120 scenes, and 1,000 demonstrators. Fine manipulation occupies more than two thirds of the corpus. Pretraining mixes these human demonstrations with 100 hours of T-Rex robot teleoperation data, sampled proportionally to hours.
Calibration does not remove every human–robot difference. It does provide a physical supervision signal that visual motion alone omits: similar hand poses can correspond to different contact loads.
The coarse planner predicts a shared visual future
FACT’s coarse planner is a 5B video DiT initialized from Wan2.2-TI2V. Task text and anchor-image features enter through cross-attention. Clean video context, robot state, and past tracks condition the generation of future video, visual tracks, and an action chunk.
For a clean target $c$, noise $\epsilon$, and noise level $\tau$,
\[x_\tau=(1-\tau)c+\tau\epsilon,\qquad v^*=\epsilon-c.\]Per-token noise conditioning lets clean context coexist with noisy prediction targets. The model uses a block-causal attention structure; track tokens retain a dedicated geometric pathway. The coarse loss combines video and action flow losses with track flow, position, and smoothness terms:
\[\mathcal L_{\rm coarse}=\mathcal L_{\rm dyn}+\mathcal L_{\rm act} +20\left(\mathcal L_{\rm trk}^{\rm flow} +5\mathcal L_{\rm trk}^{\rm pos} +2\mathcal L_{\rm trk}^{\rm smooth}\right).\]At inference, six Euler steps take the generated modalities to $\tau=0.4$. Video and tracks are read out; the action is still partially denoised. Hidden states from observation and action tokens become fixed coarse memory for the next stage. This shared visual-motion prediction is the proposed route for transferring task structure across different hands.
Future force and action are refined together
The refiner receives the partially denoised action, observed tactile history, and noisy future-force tokens. Bidirectional self-attention allows action and future force to influence each other. Cross-attention reads the cached coarse memory without rerunning the large planner.
The two target streams use different schedules:
\[\tau_a:0.4\rightarrow0,\qquad \tau_F:1\rightarrow0.\]Four Euler steps update the remaining action suffix and unobserved force slots. Executed actions and observed force remain fixed. A new tactile refresh supplies fresh contact evidence and triggers another refinement of the unexecuted suffix.
The policy outputs a 16-step action chunk at 20 Hz, with 62 dimensions for the two arms and hands. Future tactile prediction has shape $4\times10\times6$, matching four 5 Hz refresh slots and ten fingertips. The refiner loss supervises action and force velocities, with force weight 0.5, plus a VQ term. The overall objective adds this loss to the coarse objective, and the components train jointly.
“Frozen coarse memory” refers to caching within a sampling pass. The DiT backbone itself is fine-tuned during training. Confusing these two meanings of frozen would lead to a different implementation.
Ten tasks require three kinds of result
The YAM station uses two Sharpa Wave hands with fingertip six-axis sensing, one ego camera, and two wrist cameras. Each downstream task has ten robot and ninety human demonstrations for fine-tuning. Binary tasks use ten evaluation rollouts, with randomized object placement.
| Binary task | T-Rex | Full FACT |
|---|---|---|
| Lab pipetting | 0% | 40% |
| Open test-tube cap | 20% | 80% |
| Open microcentrifuge flip cap | 20% | 50% |
| Tie shoelaces | 0% | 20% |
| Shuffle cards | 10% | 30% |
| Multi-object grasp | 0% | 40% |
| Use screwdriver | 10% | 30% |
| Mean of seven binary tasks | 8.6% | 41.4% |
The maximum 60-point gain over T-Rex occurs on test-tube cap opening. It is not an average gain across the suite.
Harp and xylophone scores count cleanly sounded notes out of ten: FACT reaches 6.7 and 5.4, versus T-Rex’s 2.1 and 3.2. Pipette-volume adjustment reports whether the robot engages the dial, followed by error only on engaged trials. FACT engages in 5/10 trials, with 6.4 µL mean absolute error; T-Rex engages in 2/10, with 14.0 µL error. Reporting the error without engagement frequency would hide half of FACT’s failures.
What the ablations support
Removing future-force prediction reduces the seven-task binary mean from 41.4% to 14.3%; removing track prediction gives 35.7%. For harp playing, eliminating force prediction drops the clean-note score from 6.7 to zero. The force-prediction pathway matters within this architecture, though the ablation changes the learned model as a whole.
Human force supervision during pretraining is also consequential. Removing it while retaining human data drops test-tube opening from 80% to zero and pipetting from 40% to 20%. Multi-object grasp remains at 40%, showing that the effect varies by task.
On three selected tasks, replacing part of a fixed 100-trajectory fine-tuning set with additional robot data produces fairly small observed changes. This supports the usefulness of human data in that regime. Ten evaluation trials and a few mixtures cannot establish a universal human-to-robot data exchange rate.
My takeaway is to test future-force prediction as an action-coupled target, and to preserve engagement or contact-establishment metrics alongside final precision. For manipulation like turning a pipette dial, reaching stable contact is already a large part of the problem.