[Paper Notes] LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer
Published:
This post supports English / 中文 switching through the language toggle in the top navigation.
TL;DR
LAP changes the supervision used to adapt a pretrained vision-language model (VLM) for robot control. A deterministic parser summarizes each continuous end-effector action chunk with ordinary text such as move left 5 cm; move up 2 cm; rotate clockwise 15 degrees. The VLM learns to predict this structured language-action, keeping motor supervision close to its language pretraining distribution and giving motions shared semantics across robot bodies.
LAP-3B combines this language-supervised VLM with a lightweight flow-matching action expert. During training, the VLM predicts language-action tokens and the expert predicts the original continuous action chunk. The expert cannot attend to the language-action tokens, and its gradients are stopped at the VLM boundary. During deployment, no language-action sentence is generated or executed: the expert directly maps the current images, task instruction, and proprioceptive state to continuous actions at 25 Hz.
This distinction is the conceptual center of the paper. Language-actions shape the representation learned during pretraining; they are not an intermediate command language in the deployed control loop. Across three previously unseen single-arm embodiments, LAP-3B reports more than 50% average zero-shot success, approximately 2× the strongest controlled baseline. The paper also reports faster LIBERO adaptation, up to 2.5× fewer demonstrations for comparable real-robot progress, better compatibility with motion-prediction VQA, and favorable scaling from 4B to 27B parameters.
Paper Info
“LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer” is by Lihan Zha, Asher J. Hancock, Mingtong Zhang, Tenny Yin, Yixuan Huang, Dhruv Shah, Allen Z. Ren, and Anirudha Majumdar, with affiliations at Princeton University and Physical Intelligence. This note covers arXiv:2602.10556v2, revised February 15, 2026. The work is listed in the official RSS 2026 program. The project page provides real-robot videos, and the repository releases code and checkpoints.
1. The Bottleneck Is Also an Action-Representation Problem
VLA models inherit visual and semantic priors from internet-scale VLM pretraining. Those priors support object recognition, language following, and scene generalization. Motor adaptation introduces a new output distribution: continuous values, discretized coordinates, learned action codes, or unused vocabulary tokens. These symbols have weak connections to the meanings already organized inside the VLM.
This creates a difficult optimization balance. The backbone must acquire precise control while retaining the transferable visual-semantic structure that made VLM initialization valuable. Large heterogeneous robot datasets help, yet they cover only a small portion of the possible embodiment space. A model that binds its features to the action codes and sensor conventions of the training robots may fail after a gripper, wrist camera, kinematic chain, or control interface changes.
A shared Cartesian action space alone does not solve the problem. Two robots can both expose end-effector deltas while producing different images, proprioceptive patterns, reachable motions, and low-level dynamics. LAP therefore changes the target presented to the VLM: each motor chunk is paired with a semantically meaningful description of its net spatial effect.
2. Language-Actions Are Generated by Rules, Not by Another Model
For an observation
[ o_t={I_t^1,\ldots,I_t^n,s_t}, ]
task instruction (l), and continuous action chunk (a_{t:t+H}), LAP constructs a text sequence (\hat a^{\mathrm{lang}}_{t:t+H}). This conversion uses robot trajectory fields already present in the dataset. It requires no LLM, learned tokenizer, or human annotation.
The parser first computes the cumulative translation and rotation between (t) and (t+H). Translation follows a fixed convention: (+x) is forward, (+y) is left, and (+z) is up. Roll and pitch use tilt; yaw uses rotate, with clockwise and counterclockwise marking the sign. Magnitudes are discretized into integer centimeters and degrees. Zero-valued clauses are omitted.
The basic template is
[ \hat a_t^{\mathrm{lang}} = \text{“}\langle\text{verb}\rangle\; \langle\text{direction}\rangle\; \langle\text{magnitude}\rangle\; \langle\text{unit}\rangle\text{”}. ]
For example, a chunk with (\Delta x=5) cm, (\Delta y=2) cm, and (\Delta\text{yaw}=-15^\circ) becomes
move forward 5 cm; move left 2 cm; rotate clockwise 15 degrees
The full clause order is fixed:
move <direction> <k> cm;
tilt <direction> <k> degrees;
rotate <clockwise/counterclockwise> <k> degrees;
open/close gripper
LAP describes the net effect of a chunk, not every waypoint inside it. Many detailed trajectories can share the same language-action. The raw chunk remains available as the supervision for the continuous action expert, so the model still learns intermediate motion, timing, and gripper commands.
Half of the language-actions are expressed in the robot base frame and half in the end-effector frame during training. The prompt names the chosen frame. This randomization encourages the backbone to use the visual observations and frame semantics instead of attaching one direction word to one embodiment-specific visual pattern.
3. Two Training Targets with an Insulated Interface
The VLM learns the language-action with a standard autoregressive cross-entropy objective:
[ \mathcal L_{\mathrm{CE}} =- \mathbb E_{(\hat a^{\mathrm{lang}},o_t,l)\sim\mathcal D} \sum_i \log p_\theta!\left( \hat a^{\mathrm{lang}}{t,i} \mid o_t,l,\hat a^{\mathrm{lang}}{t,<i} \right). ]
The action expert receives the same observation and task context and learns the continuous chunk through flow matching. Given a target action (a), Gaussian noise (z), and (\tau\sim\mathcal U(0,1)), the paper constructs
[ x_\tau=(1-\tau)z+\tau a, \qquad u=a-z, ]
and optimizes
[ \mathcal L_{\mathrm{flow}} = \mathbb E \left[ \left|v_\phi(x_\tau,\tau;o,l)-(a-z)\right|_2^2 \right]. ]
The joint objective is
[ \mathcal L=\mathcal L_{\mathrm{flow}}+\lambda\mathcal L_{\mathrm{CE}}, ]
with (\lambda=0.8) in pretraining and (0.4) in fine-tuning.
flowchart TD
O["Multi-view RGB images"] --> V["PaliGemma-3B VLM"]
T["Task instruction"] --> P["Structured prompt"]
S["Discretized end-effector state"] --> P
F["Named reference frame"] --> P
P --> V
V --> L["Language-action token prediction"]
L --> CE["Cross-entropy loss<br/>training only"]
V -->|"cross-attention features"| A["Flow-matching action expert"]
R["Raw continuous action chunk"] --> FM["Flow-matching loss"]
A --> FM
A --> C["Continuous action chunk<br/>training and inference"]
L -. "attention blocked" .-> A
A -. "gradient stopped at VLM" .-> V
The masking and gradient directions matter. Language-action tokens may attend to the image, prompt, and state prefix. Action-expert tokens also attend to the prefix, while attention from the expert to language-action tokens is blocked. Gradients from the expert stop before entering the VLM. The VLM is therefore shaped by language supervision; the expert learns to turn the resulting prefix representation into detailed actions.
This controlled design also strengthens the comparison with (\pi_{0.5})-replicated. The architecture and data mixture are held fixed, while VLM supervision changes from FAST tokens to language-actions.
4. What Language Enters at Inference Time?
The deployment input contains the high-level task instruction, such as Put the marker into the cup. The paper uses the structured prompt
Task: <instruction>, predict the robot's action in the <base frame or end-effector frame>;
State: s1 s2 ... sD;
Answer:
An example from the appendix is
Task: Put the marker into the cup, predict the robot's action in the base frame;
State: 20 121 34 144 112 45 235 44 21 255;
Answer:
The state tokens encode Cartesian end-effector position, a continuous 6D rotation representation, and binary gripper state after discretization. Camera images enter through the visual encoder.
At inference, the flow expert starts from noise and integrates its learned ODE to produce a continuous action chunk. The deployed path does not autoregressively generate text such as move left 5 cm, and no parser converts such a sentence into a geometric goal. The robot executes continuous delta-pose commands through its low-level control stack, obtains updated images and state, and queries the policy again.
Language-actions therefore teach an internal spatial vocabulary during pretraining. They summarize what a demonstrated chunk accomplished. The continuous expert learns how that accomplishment unfolds at motor resolution. Closed-loop observation updates handle accumulated error; the text summary itself provides no exact endpoint guarantee or path constraint.
5. Training Recipe and Data Composition
LAP-3B initializes its VLM from PaliGemma-3B and follows the Mixture-of-Transformers action-expert design used by (\pi_{0.5}). Images are resized to (224\times224), with at most two images per sample. States and delta end-effector actions are normalized with global 1st and 99th percentiles. The action horizon is 16.
The main configuration uses a global batch size of 2,048, learning rate (10^{-4}), 5,000 warmup steps, EMA beginning at step 5,000 with decay 0.999, Adam ((\beta_1,\beta_2)=(0.9,0.95)), gradient clipping at 1.0, and weight decay (10^{-4}). The main text notes that a 15,000-step checkpoint—about 0.65 epoch and roughly ten wall-clock hours on 64 TPU v6e chips—already works on real robots. The appendix reports roughly 50 TPU v6e-64 hours for each final hero run and more than 4,000 TPU v6e-64 hours across approximately 200 pretraining experiments.
The training mixture is dominated by DROID:
| Source | Fraction of training samples |
|---|---|
| DROID | 85.26% |
| Fractal | 5.86% |
| Bridge | 3.39% |
| MolmoAct | 1.73% |
| Twelve other OXE datasets combined | 3.76% |
Idle segments below a motion threshold and trajectories without task instructions are removed. This composition matters when interpreting zero-shot transfer: the model sees broad multi-robot data, yet one source provides most samples.
6. Zero-Shot Cross-Embodiment Results
The real-robot study uses four platforms. DROID is represented in pretraining. Custom Franka, YAM, and Kinova are held out as unseen embodiments. They differ in degrees of freedom, grippers, cameras, and control interfaces. The task families cover pick-and-place, sorting, tissue pulling, towel placement, and pouring, all using 6-DoF manipulation.
Each embodiment is evaluated on two tasks with 20 trials per task. Success is binary and requires full task completion. Across the three unseen robots and six tasks, LAP-3B reports over 50% average zero-shot success, around 2× the strongest baseline. In the authors’ setup, the evaluated public OpenVLA, MolmoAct, X-VLA, (\pi_{0.5})-Base, and (\pi_{0.5})-DROID checkpoints reach zero success on the unseen platforms. Controlled (\pi_0)- and (\pi_{0.5})-replicated models produce more meaningful motion but lack the spatial precision required for reliable completion.
The comparison on the seen DROID setup is also useful. LAP-3B exceeds the controlled replicated variants by roughly 15 percentage points and performs comparably to the DROID-specific (\pi_{0.5}) checkpoint without a separate DROID fine-tuning stage.
These numbers establish non-trivial transfer under a meaningful embodiment shift. They do not establish universal hardware compatibility. All evaluated platforms are single-arm manipulators, and the interface still assumes compatible observations, calibrated proprioception, normalized state/action fields, and a usable end-effector or joint controller.
7. Adaptation, VQA Co-Training, and Scaling
On LIBERO, LAP-3B reaches 78% success after one fine-tuning epoch and 96.8% within six epochs. The full benchmark average is 96.8 for LAP-3B and 97.2 with VQA co-training. Several methods obtain similar or higher saturated scores, including X-VLA at 98.1, so LAP’s main LIBERO claim concerns adaptation speed and initialization quality.
The real-robot adaptation study adds two harder tasks: hanging tape on a rack with YAM and folding a towel before placing it in a basket with Custom Franka. LAP-3B reaches about 50% staged task progress on YAM with 20 demonstrations, which the paper reports as roughly 2.5× fewer demonstrations than the baselines for comparable performance. On Franka, it remains stronger across the tested data regimes.
Language-actions also make a motion-prediction VQA objective natural. Given two frames, the VLM answers a question such as “What movement did the robot make from the first image to the second in the robot base frame?” using the same language-action format. Co-training improves spatial generalization and downstream adaptation in the reported experiments.
Finally, the authors replace PaliGemma with Gemma 3 backbones at 4B, 12B, and 27B scales. LAP’s token and action validation losses improve monotonically with capacity, while the FAST-supervised replicated baseline saturates or degrades. This is evidence that semantically aligned motor supervision can use additional model capacity more effectively, although the scaling study reports validation losses instead of full real-robot evaluations at every size.
8. Why Might the Representation Transfer?
The paper provides two main diagnostics. A t-SNE visualization shows stronger overlap between seen- and unseen-embodiment representations for LAP-3B. On held-out unseen-robot data, its best action prediction error is 0.151, compared with 0.168 for (\pi_{0.5})-replicated and 0.189 for (\pi_0)-replicated. The corresponding flow-expert validation losses are 0.049, 0.051, and 0.052.
These observations support a plausible mechanism: language supervision preserves and reorganizes the VLM’s visual-semantic features around shared spatial effects, producing a better context representation for the action expert. The controlled action-representation comparison is particularly valuable because architecture and data are matched.
The evidence remains correlational. t-SNE is a lossy visualization, lower offline error does not fully explain closed-loop success, and several ingredients work together: language-actions, frame randomization, knowledge insulation, prompt-state tokenization, heterogeneous data, and the action expert. More targeted ablations could isolate their individual contributions.
9. Strengths and Limitations
The central idea is simple and scalable. Dataset actions create their own text labels through deterministic parsing, so the recipe adds no annotation pipeline or learned tokenizer. The training/inference separation also gives LAP practical control frequency: rich language supervision trains the representation, while continuous flow matching handles real-time action generation.
The controlled baselines address an important confound by matching architecture, data mixture, and major optimization settings. More than 1,300 real-robot trials, explicit confidence intervals, three unseen embodiments, and released code/checkpoints make the empirical case stronger than a small qualitative demonstration.
Several boundaries remain:
- The zero-shot study covers single-arm manipulators. Bimanual systems, dexterous hands, mobile manipulation, and embodiments with radically different action semantics remain open.
- Every zero-shot task uses 20 trials, leaving wide uncertainty around per-task success rates.
- A language-action records net displacement and omits the internal path. High-frequency reactive control, extreme precision, and contact-rich deformable manipulation may need hierarchical or multi-scale descriptions.
- Cross-embodiment deployment still requires compatible state/action adapters, coordinate conventions, normalization, camera inputs, and low-level controllers.
- DROID contributes 85.26% of the training mixture, so broader balance across embodiments would make the generalization claim more robust.
- The authors’ representation diagnostics explain part of the result; independent replication and wider hardware tests are still needed.
Takeaways
LAP makes three ideas concrete:
- Action representation can determine how much VLM knowledge survives motor adaptation. Ordinary directional language supplies semantics that arbitrary action codes lack.
- A training representation does not need to become a deployment interface. Language-actions supervise the VLM; the flow expert directly generates continuous actions at test time.
- Cross-embodiment transfer benefits from describing effects shared across bodies. “Move left five centimeters” identifies a common spatial consequence even when cameras, grippers, kinematics, and joint commands differ.
The broader research lesson is that robot foundation models need more than larger heterogeneous datasets. The supervision interface decides which regularities can be shared. LAP shows that a small representational change—expressing low-level motion in the VLM’s native language space—can materially change zero-shot embodiment transfer.