[Paper Notes] DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation
Published:
DynamicVLA studies a failure mode that is easy to hide in static manipulation: by the time a VLA finishes thinking, the object may have already moved. Static pick-and-place tolerates slow action chunking because the scene changes little during inference. Dynamic object manipulation changes the constraint. A rolling can, a moving ball, or a deflected object requires fast perception, temporal anticipation, and action execution that stays synchronized with the latest observation.
My read: the paper’s center of gravity is temporal control, not model scaling. DynamicVLA uses a compact 0.4B model, overlaps inference and execution with Continuous Inference, and uses Latent-aware Action Streaming to discard stale actions. The benchmark contribution, Dynamic Object Manipulation (DOM), is equally central because it creates the data and evaluation surface that static robot datasets lack.
Paper Info
The paper is “DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation” by Haozhe Xie, Beichen Wen, Jiarui Zheng, Zhaoxi Chen, Fangzhou Hong, Haiwen Diao, and Ziwei Liu from S-Lab, Nanyang Technological University. It is available as arXiv:2601.22153. The project page is haozhexie.com/project/dynamic-vla, and the official implementation is released at hzxie/DynamicVLA.
The Problem
Most VLA systems use action chunking: the model observes the scene, predicts a short action sequence, then the robot executes that sequence. This amortizes inference cost, but it creates two timing problems under moving objects.
First, there is a perception-execution gap. The model starts inference from observation (O_t), but the resulting action chunk becomes available only after latency (m). The first actions in the chunk were meant for the old state, while the world has already advanced to (O_{t+m}). Second, there is inter-chunk waiting. Many systems trigger the next inference only after finishing the current action chunk, which leaves pauses or stale behavior between chunks.
DynamicVLA frames dynamic manipulation as a synchronization problem:
[ A_t = M(O_t, L_t, P_t), \quad A_t = {a_t,\ldots,a_{t+n}} ]
Here (O_t) is a temporal window of visual observations, (L_t) is the language instruction, and (P_t) is proprioception. The latent object state (s_t) keeps evolving while the model reasons. The control system therefore needs a model that is fast enough and an execution policy that knows which actions are already outdated.
Model Design
DynamicVLA has a compact VLA backbone and a diffusion-style action expert. The language backbone is SmolLM2-360M, truncated to the first 16 transformer layers. For vision, the paper uses FastViT, a convolutional vision encoder that compresses spatial structure efficiently and avoids the token growth of transformer-based visual encoders under multi-frame inputs.
The model input includes:
- visual observations from a sparse temporal window, defaulting to (O_t={o_{t-2},o_t});
- language instruction tokens;
- robot proprioception, projected as a state token.
The output is an action chunk:
- horizon (n=20);
- each action is a 32D vector representing end-effector pose and gripper state, with padding for unused dimensions.
The action expert is a conditional Flow Matching Transformer. Given VLM features (f_t), noisy action chunk (A_t^\tau), and diffusion timestep (\tau), it learns the denoising vector field:
[ \ell_\tau(\theta)=\mathbb{E}{p(A_t|f_t),q(A_t^\tau|A_t)} \left[\left|E\theta(A_t^\tau,O_t)-u(A_t^\tau|A_t)\right|\right] ]
This architecture is designed around latency. The model is small enough for high-frequency inference, while the action expert still produces temporally structured action chunks.
Continuous Inference
Continuous Inference changes when the next inference starts. In a serial chunking pipeline, the system predicts (A_t), executes the full chunk, then predicts the next chunk. That creates waiting at chunk boundaries.
DynamicVLA instead starts a new inference as soon as the previous inference finishes. While the robot executes actions from (A_t), the model already predicts (A_{t+m}). If the action horizon (n) is longer than inference latency (m), a fresh chunk arrives before the current chunk is exhausted. Execution no longer blocks on inference completion.
This is a simple systems idea, but it matters because dynamic manipulation is dominated by timing. A policy that waits cleanly in static scenes can lose the object in dynamic scenes.
Latent-Aware Action Streaming
Continuous Inference creates overlapping action chunks. At a given execution timestep, actions from the older chunk (A_t) and the newer chunk (A_{t+m}) may both exist. DynamicVLA resolves this with Latent-aware Action Streaming (LAAS).
LAAS applies two rules:
- discard actions in (A_t) whose intended timesteps are earlier than (t+m), since they correspond to already outdated observations;
- when old and new chunks overlap, prioritize the newer chunk (A_{t+m}).
This turns action chunking from a fixed commitment into a streaming control interface. The model still benefits from chunk prediction, but execution remains tied to the most recent observation available after inference delay.
DOM Benchmark
The paper also introduces Dynamic Object Manipulation (DOM), a benchmark and data pipeline for moving-object manipulation. It contains 200K synthetic episodes, 2.8K scenes, 206 objects, and 2K real-world episodes collected without teleoperation.
The data collection pipeline is important. In simulation, Isaac Sim provides object 6D pose and velocity. A state-machine controller produces four stages: approach object, grasp and lift, approach target and place, then reset. In the real world, teleoperation is too slow for fast moving objects, so the authors build a real-world “simulator”: synchronized third-person RGB-D cameras estimate object pose and velocity, then feed the same state-machine controller.
DOM evaluates three groups of abilities:
- Interaction: closed-loop reactivity, dynamic adaptation, long-horizon sequencing.
- Perception: visual understanding, spatial reasoning, motion perception.
- Generalization: visual generalization, motion generalization, disturbance robustness.
This benchmark design is useful because it separates dynamic manipulation into the axes that usually get collapsed into a single success rate.
Results
In simulation, DynamicVLA is evaluated against Diffusion Policy, OpenVLA-OFT, (\pi_0), (\pi_{0.5}), SmolVLA, GR00T-N1.5, VLA-Adapter-Pro, and VLASH. Each method is fine-tuned on DOM. DynamicVLA reaches 47.06% average success across the nine DOM dimensions, while the strongest baseline in the table is 13.61%. It also completes tasks faster, with 8.53 s average time versus around 10 s for most baselines.
The interaction gains are especially large. In simulation, DynamicVLA reports 60.5% closed-loop reactivity, 38.5% dynamic adaptation, and 40.5% long-horizon sequencing. Real-world interaction tasks show the same pattern: on tasks such as placing a moving coffee can, conical bottle, pickleball, or collecting repeated balls, DynamicVLA is far more reliable than (\pi_{0.5}), SmolVLA, and VLASH.
Perception remains harder. DynamicVLA improves visual understanding, spatial reasoning, and motion perception, but motion-conditioned targets are still difficult. The real-world perception section reports 51.9% average success for DynamicVLA, while the best baselines remain near 11.7%.
For generalization, DynamicVLA improves transfer to unseen appearances and unseen motion patterns. Disturbance robustness remains the weakest axis, which makes sense because random impacts and surface variation create state changes that are difficult to model from rigid-body pose streams alone.
Ablations
The ablations make the paper’s engineering choices clearer:
- Full DynamicVLA reaches 47.06% success on DOM.
- Removing both Continuous Inference and LAAS drops success to 30.27%.
- Keeping Continuous Inference but removing LAAS gives 39.72%, showing that overlapping inference is useful but stale action handling still matters.
- Replacing FastViT with a transformer-style visual encoder drops success to 28.89%.
- A 135M backbone is too small at 26.67%, while a 1.7B backbone is too slow at 24.33%. The 360M setting gives the best latency-capacity balance.
The appendix adds a useful detail about temporal context. A single current frame reaches 38.22%, while the sparse pair ({o_{t-2},o_t}) reaches 47.06%. Adding more frames gives little extra benefit. The takeaway is precise: dynamic manipulation needs motion cues, but extra visual history can quickly become redundant if it raises compute without improving timing.
Strengths and Limitations
DynamicVLA is strong because it treats latency as part of the model design. The architecture, inference schedule, action streaming rule, and benchmark all target the same failure mode. The paper also avoids relying on human teleoperation for dynamic object data, which is important because human reaction time becomes a data-collection bottleneck.
The main limitation is scope. DOM focuses on short- to medium-horizon rigid-body dynamics. The real-world setup estimates object 6D pose and velocity, then uses that interface to collect demonstrations and evaluate policies. This is a strong start, but deformable objects, liquids, articulated objects, and long-horizon moving-object tasks need richer state representations and planning. The authors also note that the real-time architecture trades off multimodal understanding against responsiveness; larger models can reason better, but slow inference breaks dynamic control.
Takeaway
DynamicVLA’s practical message is that dynamic robot manipulation is not solved by adding a temporal input window alone. The execution semantics matter. A VLA must decide when to infer, how to overlap inference with execution, and which predicted actions are still temporally valid. CI and LAAS make that control loop explicit.
For future VLA systems, I would keep three ideas from this paper: use compact backbones when the task is timing-sensitive; treat action chunks as streamable predictions, with execution allowed to refresh them; and evaluate moving-object manipulation along interaction, perception, and generalization axes so static success rate does not hide timing failures.