Real-Time Chunking for VLAs: RTC, Training-Time RTC, and VLASH
Published:
Large vision-language-action models have a timing problem. The robot controller wants a new command every 20-30 ms, while a large VLA may need tens or hundreds of milliseconds to produce the next action chunk. If the robot waits, it stalls. If it keeps moving and naively swaps to the next chunk, the new action can be inconsistent with the trajectory already being executed.
These three papers are best read as a sequence of increasingly precise answers to that mismatch:
- RTC makes chunked policies real-time by running inference asynchronously and inpainting the next chunk so it remains compatible with the previous chunk.
- Training-Time RTC keeps the same runtime interface but moves the prefix-conditioning problem into training, removing the expensive inference-time guidance step.
- VLASH reframes the asynchronous failure mode as prediction-execution state mismatch and conditions the VLA on the robot state that will hold when the new chunk actually starts executing.
My main takeaway: real-time action chunking is not one trick. It has at least three layers. The system layer hides latency with asynchronous execution. The action layer preserves continuity across chunk boundaries. The state layer aligns the model input with the future state at execution time.
Paper Info
The first paper is “Real-Time Execution of Action Chunking Flow Policies” by Kevin Black, Manuel Y. Galliker, and Sergey Levine. It introduces real-time chunking (RTC) and is available as arXiv:2506.07339, with the Physical Intelligence project page Real-Time Action Chunking with Large Models and code at Physical-Intelligence/real-time-chunking-kinetix.
The second paper is “Training-Time Action Conditioning for Efficient Real-Time Chunking” by Kevin Black, Allen Z. Ren, Michael Equi, and Sergey Levine, available as arXiv:2512.05964. It is a direct follow-up to RTC.
The third paper is “VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference” by Jiaming Tang, Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, and Song Han, available as arXiv:2512.01031, with project/code at mit-han-lab/vlash.
The Shared Problem
An action-chunking policy predicts a sequence
\[ A_t = [a_t, a_{t+1}, \ldots, a_{t+H-1}], \]
where \(H\) is the prediction horizon. During rollout, the system usually executes only the first \(s \le H\) actions before asking the model for another chunk. This \(s\) is the execution horizon.
If the controller period is \(\Delta t\) and model inference takes \(\delta\), the delay in controller steps is roughly
\[ d = \left\lfloor \frac{\delta}{\Delta t} \right\rfloor. \]
For modern VLAs, \(d\) is often larger than zero. A synchronous system waits for the model after each chunk, producing visible pauses. A naive asynchronous system starts the next inference while the old chunk is still executing, but the new chunk arrives after \(d\) steps. At that moment the robot may jump from an action planned by the old chunk to an action planned by the new chunk under a different observation and a different sampled mode.
The technical question is therefore:
How can the robot keep executing at controller frequency while letting a slow model think in the background, without creating out-of-distribution jumps at chunk boundaries?
Original RTC: Inference-Time Inpainting
RTC answers this by turning asynchronous action generation into an inpainting problem.
Suppose the robot is executing an old chunk and starts inference for the next chunk. While the model is running, the next \(d\) actions are already committed: the robot will execute them from the old chunk before the new chunk is ready. RTC uses these known actions as a prefix constraint for the new chunk. The new chunk should agree with the already-committed prefix and then smoothly continue into newly generated actions.
For flow-matching policies, a chunk is generated by integrating a learned velocity field from noise:
\[ A_t^{\tau + 1/n} = A_t^\tau + \frac{1}{n} v_\pi(A_t^\tau, o_t, \tau). \]
RTC modifies the velocity field during sampling. At each denoising step, it forms an estimate of the final denoised chunk:
\[ \hat{A}t^1 = A_t^\tau + (1-\tau)v\pi(A_t^\tau, o_t, \tau). \]
Then it compares this estimate to the previous chunk on overlapping timesteps. The error is weighted by a mask \(W\). The first \(d\) actions get weight 1 because they are guaranteed to execute. The final \(s\) actions have no overlap with the previous chunk and get weight 0. The intermediate overlap receives exponentially decaying weights, giving nearby future actions stronger continuity pressure than far-future actions.
The correction is computed with a vector-Jacobian product through the denoiser. In implementation terms, the official Kinetix code does:
x_1, vjp_fun, v_t = jax.vjp(denoiser, x_t, has_aux=True)
weights = get_prefix_weights(inference_delay, prefix_attention_horizon, ...)
error = (prev_action_chunk - x_1) * weights[:, None]
pinv_correction = vjp_fun(error)[0]
v_t = v_t + guidance_weight * pinv_correction
That is the core of RTC: the base flow policy proposes the action chunk, while pseudoinverse-style guidance bends the denoising trajectory toward compatibility with the old chunk.
The system part matters just as much as the math. RTC uses a background inference loop. The controller calls GetAction every control step and consumes the current chunk. Meanwhile, the background thread waits until enough actions have been executed, snapshots the newest observation, estimates the next delay from recent latency measurements, runs guided inference, then swaps in the new chunk as soon as it is ready. The execution horizon can adapt as \(s=\max(d, s_{\min})\), so the system remains conservative under changing latency.
In simulation, RTC is evaluated on 12 dynamic Kinetix tasks with force-based control, where holding position is not a meaningful fallback. It outperforms naive asynchronous execution, temporal ensembling, hard masking, and bidirectional decoding under increasing delay. On real bimanual manipulation with \(\pi_{0.5}\), it improves throughput and remains robust even with injected latency, while temporal ensembling can trigger high oscillations.
The limitation is also clear. RTC adds a VJP/backpropagation correction at every denoising step, so the method increases inference cost exactly inside a real-time system. It also applies most naturally to diffusion- or flow-based policies. Conceptually, RTC fixes action compatibility, while the observation and robot state used by the model are still captured at inference start.
Training-Time RTC: Learn the Prefix Constraint
Training-Time RTC keeps the asynchronous RTC interface but changes where the continuity constraint is learned.
The key observation is simple: during training, we already have ground-truth action chunks. We can simulate inference delay by taking the first \(d\) actions of a chunk as a known prefix, then asking the model to denoise only the postfix:
\[ p(A_{t+d:H} \mid o_t, A_{t:t+d}). \]
This turns RTC from an inference-time optimization procedure into a conditional generation problem learned during flow-matching training.
Implementation requires three small changes:
- Allow the flow-matching time \(\tau\) to differ across action tokens.
- Feed the prefix actions as clean, non-noisy tokens by setting their \(\tau\) to 1.
- Mask the loss so the model is trained only on the postfix tokens.
The training loss starts from the usual flow-matching interpolation:
\[ A_t^\tau = \tau A_t + (1-\tau)\epsilon,\quad \epsilon \sim \mathcal{N}(0,I). \]
For the prefix tokens, Training-Time RTC sets \(\tau=1\), making them equal to the clean ground-truth actions. For the postfix tokens, it uses normal noisy flow-matching inputs. The model sees both prefix and postfix, but the loss is computed only on the postfix. At sampling time, the same trick is used: clamp the known prefix, set its time to 1, and denoise the rest without any VJP guidance.
This is why the paper calls the method a drop-in replacement for inference-time RTC. The robot runtime can still pass in an action prefix and delay \(d\), but the action generator now handles the prefix natively. In the official Kinetix code, this appears as simulated_delay: during loss computation, a random delay is sampled, prefix tokens are assigned time 1, and the loss mask ignores those prefix positions.
The trade-off is flexibility. Original RTC can softly condition on all overlapping actions with a real-valued mask. Training-Time RTC uses a hard prefix corresponding to the sampled delay. It also requires choosing a training delay distribution that matches the expected deployment latency. The paper samples delays during training, including uniformly over delays 0 to 10 for the real-world \(\pi_{0.6}\) experiments.
The empirical message is useful. On Kinetix, Training-Time RTC outperforms inference-time RTC when the delay is 2 or higher, with the gap widening at larger delays. The authors interpret this as a failure mode of the inference-time linearization: when the prefix is long, the VJP-based correction has to work harder to produce a consistent postfix. On real tasks such as box building and espresso making, Training-Time RTC keeps similar task performance and speed to inference-time RTC while removing the extra runtime cost.
My read: Training-Time RTC is the version I would prefer when fine-tuning is available. Original RTC is attractive because it can wrap a pre-trained diffusion/flow policy immediately. Training-Time RTC is more deployment-friendly once you can afford a small fine-tuning stage.
VLASH: Align the State, Not Just the Chunk
VLASH points to a different failure mode. In asynchronous inference, the model predicts a chunk at time \(t\), but that chunk is executed starting at time \(t+\Delta\). The prediction interval is
\[ I_t^{\mathrm{pred}} = [t, t+K), \]
while the execution interval is
\[ I_t^{\mathrm{exec}} = [t+\Delta, t+\Delta+K). \]
RTC makes the new chunk compatible with the old chunk, but the model may still be conditioning on the robot state at inference start. By the time the new chunk begins executing, the robot has already moved. VLASH calls this prediction-execution misalignment.
The core idea is future-state awareness. During the inference delay, the robot will execute known actions from the previous chunk. Therefore, the robot state at the beginning of the next execution interval can be estimated by rolling the current state forward through those known actions:
\[ \hat{s}{t+\Delta} = \mathrm{rollforward}(s_t, a_t, a{t+1}, \ldots, a_{t+\Delta-1}). \]
VLASH then conditions the VLA on the current visual observation \(o_t\) and the rolled-forward robot state \(\hat{s}_{t+\Delta}\). For delta-action robots, this can be as simple as accumulating the known delta actions into the proprioceptive state. The visual observation is still the latest available image at inference start, but the robot-state input is shifted to the state where the new actions will actually take effect.
The paper adds an important training detail: existing VLAs may under-use robot state, sometimes performing better with visual-only fine-tuning. Feeding a future state at test time will not help if the model learned to ignore state. VLASH therefore uses temporal-offset augmentation during fine-tuning. For a trajectory \((o_t, s_t, a_t)\), it samples an offset \(\delta\), keeps the visual observation fixed as \(o_t\), and trains the model with state \(s_{t+\delta}\) and target action chunk \(a_{t+\delta:t+\delta+H-1}\).
This creates multiple training examples with the same image but different robot states and different action targets. The model is forced to use the state input to disambiguate what action chunk is appropriate. To make this efficient, VLASH packs one shared observation token sequence with multiple offset branches and uses a block-sparse attention pattern: each offset branch can attend to the shared observation and itself, while branches do not attend to each other. This reuses expensive image/language encoding across offsets.
VLASH also includes action quantization as a practical speed knob. It groups \(q\) fine-grained delta actions into one macro-action:
\[ \hat{a}i = a{iq} + a_{iq+1} + \cdots + a_{(i+1)q-1}. \]
This reduces the number of execution steps and gives a speed-accuracy trade-off. It is separate from future-state awareness, but it matters once asynchronous inference hides most model latency and physical execution becomes the bottleneck.
In Kinetix, VLASH remains close to the synchronous upper bound and is much more robust than naive asynchronous execution under delay. At delay 4, the paper reports 81.7% success for VLASH compared with 51.2% for naive async. In LIBERO, VLASH maintains comparable accuracy to synchronous inference while improving completion time. In real-world experiments, \(\pi_{0.5}\) with VLASH reaches comparable or better score percentages than synchronous inference on manipulation tasks, and action quantization with \(q=2\) gives up to 2.03x speedup while preserving accuracy in the reported setting. The paper also emphasizes reaction latency: asynchronous inference can reduce maximum reaction latency by up to 17.4x compared with synchronous execution because the robot no longer waits for the current chunk to finish before starting model inference.
VLASH’s limitation is that it rolls forward robot state, not the full future world. If the scene changes externally during inference, the visual input is still from the inference-start observation. The method also relies on a VLA interface where proprioceptive state is available and meaningful, and it benefits from fine-tuning with temporal offsets. Still, it gives a clean answer to a problem RTC leaves partly open: what state should the VLA condition on when asynchronous execution shifts the action interval into the future?
How the Three Methods Fit Together
These methods are compatible at the conceptual level because they target different pieces of the real-time stack.
| Method | Main object being corrected | When the correction happens | What it needs |
|---|---|---|---|
| Original RTC | Action chunk continuity | Inference time | Diffusion/flow sampler plus VJP guidance |
| Training-Time RTC | Prefix-conditioned action generation | Training and sampling | Fine-tuning with simulated delay |
| VLASH | Prediction-execution state alignment | Fine-tuning and deployment input construction | Robot-state rollforward and offset augmentation |
Original RTC is the most plug-and-play for a pre-trained flow or diffusion VLA. It can be added at inference time and immediately improves asynchronous execution, though it pays extra compute for guidance.
Training-Time RTC is the clean deployment version of the same idea. It teaches the model the prefix-conditioning interface directly, so deployment avoids the costly guidance loop.
VLASH shifts attention from chunk boundary smoothness to the input side of the policy. It says that a chunk generated from stale robot state is misaligned even if the action sequence is smooth. Its fix is to feed the state where the robot will be when the chunk begins.
Practical Takeaways
If I were implementing a real-time VLA stack, I would separate the design into three layers.
First, use asynchronous inference by default. The controller should keep consuming actions while model inference runs in another thread or process. The runtime needs a shared action queue, a measured inference delay, and conservative handling of latency spikes.
Second, ensure action continuity. If using a pre-trained diffusion/flow policy with no fine-tuning budget, original RTC is the natural first implementation. If fine-tuning is possible, Training-Time RTC is likely the better long-term choice because it removes the VJP overhead from deployment.
Third, align the state. If the model conditions on proprioception, the state should correspond to the execution-time state used by the next chunk. VLASH’s rollforward idea is especially relevant for long-latency systems, remote inference, and fast dynamic tasks.
The three-paper progression also clarifies where future work can go. A strong real-time VLA should probably combine training-time prefix conditioning with future-state-aware inputs. The former keeps chunks continuous; the latter makes the new chunk start from the right robot state. For dynamic scenes, the remaining hard problem is future environment estimation: the robot can roll forward its own state with known actions, but it cannot know exactly how objects, humans, or contacts will evolve during inference.