[Paper Notes] Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models
Published:
TL;DR
AFCE trains a VLA to predict a temporal Effect that encodes demonstrated commands together with their observed physical consequences. A decoder then converts that Effect into executable actions. Future robot states and images provide training supervision; deployment uses current images, language, and robot state.
The result is strongest on coordinated manipulation. On DexJoCo, overall success reaches 50.97%, compared with 39.15% for the evaluated π₀.₅ checkpoint. The single-arm average falls slightly, and Pinch Tongs drops sharply. The useful question is therefore how this intermediate target helps coordination while preserving enough detail for precise control.
Paper and reading version
These notes follow arXiv:2610.13212v2, revised October 12, 2026; the initial submission was October 9. Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models is by Min Lin, Bokai Zhou, Jianqi Lin, Kaiyuan Yang, Yicheng Chang, Junjie Yu, Xiaofan Liu, Jianhua Han, Rui Li, Shen Zhao, and Xiaodan Liang, from Sun Yat-sen University and Yinwang Intelligent Technology Co., Ltd. It is an arXiv preprint. Sources: PDF and project page. Method details below correspond to §3 and Appendix A; results to §4 and Appendices B–C.
What exactly is the policy predicting?
A motor command and the motion it produces are different signals. Contact can prevent a finger from reaching its commanded position; an object can move differently depending on the grasp. AFCE puts both commanded and realized motion into the target used to train the policy.
For a demonstration window starting at time $t$, the tokenizer constructs
\[E_t^*=T_\phi(A_t,\Delta S_t,V_t)\in\mathbb{R}^{H\times D_E}, \qquad H=30,\quad D_E=256.\]$A_t$ is the demonstrated action chunk, $\Delta S_t$ the realized robot response, and $V_t$ sparse visual-transition evidence. There is one temporally ordered Effect token per action step. The asterisk marks a target extracted from a demonstration. Despite the word “tokenizer,” this is a continuous representation consumed by flow matching; it should not be read as a set of discrete semantic labels or a single desired object pose.
At deployment, with $o_t=(I_t,\ell,s_t)$, the computation is
\[\hat Z_t\sim p_\theta(Z\mid o_t),\qquad \hat A_t=D_\psi\bigl(N^{-1}(\hat Z_t),s_t\bigr).\]Here $N$ is a fixed normalization of the Effect space. The policy predicts the normalized Effect, and the decoder combines its denormalized value with the current robot state. It executes a prefix of the resulting chunk, observes again, and replans.
flowchart TD
A[Demonstration actions + realized response + visual transitions] --> B[Stage I: Effect tokenizer]
B --> C[Freeze tokenizer and cache Effect targets]
C -. training supervision .-> D[Stage II: flow-matching Effect policy]
O[Current images + language + robot state] --> D
D --> E[Predicted Effect]
E --> F[State-conditioned action decoder]
S[Current robot state] --> F
F --> G[Execute action prefix and observe again]
Stage I: fuse the evidence, then preserve executable detail
The action branch encodes position, rotation, and finger commands with separate group projections. The response branch encodes palm translation, relative orientation, fingertip displacement, and finger-joint changes using observed states and forward kinematics. It retains both arms for bimanual demonstrations.
The visual branch uses frozen DINOv3 features sampled at sparse times. Its localized transition tokens describe feature change, image-plane displacement, location, and temporal extent. They can include robot motion as well as object motion. A visual change spanning several control steps is associated with that interval, so the tokenizer does not treat it as a new independent observation at every step.
A learned query for each action step fuses the relevant evidence through cross-attention. A temporal Transformer then mixes information across the window. This encoder can use noncausal attention because it only constructs offline training targets.
The representation must support reconstruction:
\[\mathcal L_I=\mathcal L_A+\lambda_S(u)\mathcal L_S +\lambda_V(u)\mathcal L_V+\lambda_R\mathcal L_{\mathrm{rot}}, \qquad \mathcal L_A=\frac{2\mathcal H_p+\mathcal H_r+\mathcal H_f}{3}.\]The action decoder receives $E_t^*$ and $s_t$; auxiliary heads reconstruct robot responses and visual transitions. Each $\mathcal H$ is a normalized, group-averaged Huber error. Position has weight two, while the denominator remains three. An additional SO(3) chordal loss constrains decoded orientation. The auxiliary visual head receives spatial and temporal query metadata, without receiving the feature change it must reconstruct.
After training, the tokenizer is frozen. Channel-wise mean and standard deviation are estimated on the training demonstrations, shared across temporal positions, and fixed for Stage II:
\[Z_t^*=N(E_t^*)=\frac{E_t^*-\mu_E}{\max(\sigma_E,\sigma_{\min})}.\]This freezes the target definition while the policy learns to predict it. The future-evidence encoders and auxiliary reconstruction heads are absent from deployment.
Stage II: Effect accuracy must survive action decoding
AFCE adapts π₀.₅ by replacing its action input/output projections with Effect-width projections. Images, instruction, and robot state still condition the flow-matching backbone. With Gaussian noise $\epsilon$ and flow time $\tau$, training uses
\[Z_{t,\tau}=(1-\tau)Z_t^*+\tau\epsilon,\qquad v^*=\epsilon-Z_t^*,\] \[\mathcal L_{\mathrm{FM}}= \mathbb E\left[\frac{\|v_\theta(Z_{t,\tau},\tau;o_t)-v^*\|_F^2}{HD_E}\right].\]Flow time $\tau$ is separate from environment time $t$. Sampling integrates from noise toward $\tau=0$.
A small latent error can still produce a large action error. The authors therefore estimate the clean endpoint directly,
\[\hat Z_{t,0}^{(\tau)}=Z_{t,\tau}-\tau v_\theta(Z_{t,\tau},\tau;o_t),\]and decode it to supervise actions without backpropagating through a full sampling trajectory. The combined objective is
\[\mathcal L_{II}=\mathcal L_{\mathrm{FM}}+ r(u)\mathbb E_\tau\!\left[g(\tau) \left(\lambda_{\mathrm{GT}}\mathcal L_{\mathrm{GT}} +\lambda_{\Delta f}\mathcal L_{\Delta f}\right)\right].\]$\mathcal L_{\mathrm{GT}}$ compares decoded and demonstrated actions using equally weighted position, rotation, and finger groups. It does not reuse Stage I’s position multiplier or SO(3) term. $\mathcal L_{\Delta f}$ supervises changes in finger commands between adjacent steps, scaled by each joint’s demonstrated difference magnitude. The ramp $r(u)$ warms up these action losses; $g(\tau)$ selects the flow-time interval for action supervision and accounts for its probability mass.
The Stage-I decoder is initialized into Stage II and updated jointly at one quarter of the policy learning rate. Both action losses also send gradients into the policy. Freezing the tokenizer therefore does not mean freezing the decoder.
Appendix D reports 30,000 tokenizer updates and 60,000 policy updates, both with global batch size 32. Inference uses ten flow denoising steps for a 30-step Effect chunk. Single- and dual-arm configurations have separate native-coordinate adapters and decoding interfaces, while sharing the Effect model. Shared parameters alone do not establish an embodiment-invariant representation.
Where the improvement comes from
Training uses 100 demonstrations for each of 11 DexJoCo tasks: 1,100 episodes and 523,763 retained frames at 30 FPS. The simulated Panda–Allegro setup has six single-arm and five bimanual tasks. Native actions have 22 dimensions per arm. Evaluation randomizes object placement and table height under rand-obj.
Each method uses one trained checkpoint evaluated on three environment seeds, with 50 episodes per task per seed: 1,650 rollouts per method. The deviations below measure variation across evaluation seeds, not independent training runs.
| Method | Single-arm success | Bimanual success | Overall success |
|---|---|---|---|
| UVT | 53.44 ± 2.22% | 14.53 ± 3.06% | 35.76 ± 0.56% |
| π₀.₅ | 63.67 ± 1.86% | 9.73 ± 2.41% | 39.15 ± 2.11% |
| AFCE | 62.33 ± 1.45% | 37.33 ± 2.66% | 50.97 ± 1.65% |
AFCE leads on all five bimanual tasks. Microwave rises from 32.00% to 81.33%, and Photograph from 8.00% to 59.33%, relative to π₀.₅. The 27.60-point bimanual improvement contributes about 12.55 points to the overall average; the single-arm deficit offsets about 0.73 points. This accounts for the 11.82-point net improvement.
Pinch Tongs falls from 43.33% to 5.33%. Unlock iPad and Assembly remain difficult even with improvements, at 7.33% and 9.33%. These results constrain any claim that predicting Effects generally solves dexterous control.
The ablations complicate the explanation
Table 3 varies the evidence used to construct the target, always retaining demonstrated actions:
| Effect inputs | Single-arm | Bimanual |
|---|---|---|
| Actions only | 57.22% | 30.93% |
| Actions + visual transitions | 48.67% | 25.47% |
| Actions + robot response | 55.67% | 21.60% |
| Actions + response + visual transitions | 62.33% | 37.33% |
The complete combination has the best group means, yet adding either future-evidence source alone reduces performance relative to actions only. The action-only variant also already exceeds the direct-action baseline on bimanual tasks. My reading is that the complete target construction helps within this architecture, while the total gain over π₀.₅ cannot be assigned entirely to future supervision. Representation, prediction, and decoding all change the learning problem.
Reconstruction and execution need separate checks. In the dimensionality study, $D_E=256$ gives 25 successful GT-Effect replays out of 110 selected training demonstrations, versus 18/110 at $D_E=64$. This is a representation-and-decoder diagnostic without policy-side Effect prediction; it is not directly comparable to the main closed-loop score.
Appendix C further reports position readout error of 8.30 mm for demonstration-derived Effects with the adapted decoder, versus 36.91 mm for predicted Effects with the same decoder on matched demonstration windows. Those windows belong to the training data. A visually overlapping t-SNE projection cannot erase this gap or establish held-out generalization.
What I would carry into a follow-up experiment
The design worth testing is the separation between offline target construction and online control: future evidence can shape what a policy learns without requiring future observations at deployment. Its costs are synchronized action/state/image data, offline feature construction, and a representation that must retain fine control detail while remaining predictable.
For a follow-up on dexterous hands, I would retain the action-only latent baseline and separately measure reconstruction, GT-Effect replay, and predicted-Effect rollout. I would also report precision-sensitive tasks individually and repeat training across seeds. If an Effect representation improves coordination but keeps losing finger-level accuracy, that tradeoff should guide changes to its temporal resolution, decoder, or auxiliary supervision. The Pinch Tongs regression is the concrete test any such change needs to pass.