[Paper Notes] RoboTTT: Context Scaling for Robot Policies
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
Most robot foundation models see one observation or a very short history. RoboTTT makes long visuomotor context a trainable part of the policy: it adds Test-Time Training (TTT) layers to a VLA action head, where small fast weights are updated by gradient descent as a rollout unfolds. The history is compressed into these weights, so the policy can use thousands of timesteps without growing attention over the whole past at every step.
The training recipe scales to 8K timesteps, roughly five minutes at the paper’s 30 Hz control rate. In real-robot experiments, RoboTTT reaches a 79% average completion score across three bimanual assembly tasks, compared with 42% for the single-step GR00T N1.7 baseline. It also performs one-shot imitation from a human video, improves from its own failures, and completes 2 of 10 five-minute Gear Bot assemblies while every baseline records 0 successful runs. These results come from a specific YAM setup, task data, and post-training protocol; they do not establish general long-horizon competence across robots.
Paper and source version
Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng, Fengyuan Hu, Yunhao Ge, Jimmy Wu, Tianyuan Dai, Scott Reed, Li Fei-Fei, Yuke Zhu, and Linxi “Jim” Fan, NVIDIA, Stanford University, and The University of Texas at Austin. These notes follow arXiv:2607.15275v1, first submitted July 16, 2026. See the paper PDF and NVIDIA project page. The source is an arXiv preprint; no accepted venue is assumed. Numbers below are reported by the authors and have not been independently reproduced here.
1. The problem is how to use history, not just how to store it
A robot assembling a multi-part object may need to remember which component was installed, infer what an occluded object looked like before contact, or use its own failed action to choose a recovery. A short-context policy has little evidence for these decisions. Appending many past frames to a Transformer gives the model more evidence, but cached attention makes inference cost grow with the history length, and a fixed history can also introduce spurious correlations.
RoboTTT treats the policy state as a small neural network whose parameters change during the rollout. The base model’s slow weights are trained offline and remain fixed at deployment. The fast weights start from a learned initialization $W_0$ and are updated after each timestep by a test-time learning rule. They store a task-relevant summary in parameter space:
\[W_t \leftarrow W_{t-1}-\eta\nabla_W \mathcal L_{\mathrm{FW}}\big(f_{W_{t-1}}(K_t),V_t\big), \qquad O_t=f_{W_t}(Q_t).\]Here $K_t$ and $V_t$ are the key and value projections of the current token, $Q_t$ is the query, and $f_W$ is a small linear model or MLP. The update writes information into $W_t$; the apply step reads it through $f_{W_t}(Q_t)$. At inference, the policy does this same update-and-apply operation, so contextual learning is part of execution.
The paper’s central claim is conditional: long context becomes useful when the update rule learns what to retain and how to retrieve it. A memory vector or a long list of frames can hold information without making that information useful for action selection.
2. Where TTT sits in the robot policy
RoboTTT is instantiated on GR00T N1.7. Its vision-language backbone produces per-timestep visual-language tokens, and a Diffusion Transformer (DiT) action head predicts an $H$-step action chunk. Attention processes the current timestep. TTT layers are inserted after the self- and cross-attention blocks and process tokens across time.
At timestep $t$, the DiT receives register tokens $R_t$, proprioception $q_t$, and noised action tokens $\tilde A_t$ together with the VLM output $\Phi_t$. The per-timestep features are concatenated along the temporal dimension and passed to the TTT layers. The model uses 16 learned register tokens to carry VLM information across time, avoiding the cost of sending the full VLM token set through the fast model.
A learned gate protects the pretrained policy at initialization:
\[O=\tanh(\alpha)\odot O_{\mathrm{TTT}}+O_{\mathrm{attn}},\]where $\alpha$ starts near zero, at 0.001. The TTT contribution can grow when it helps the task, while the original attention pathway remains available. Each of the 16 DiT layers receives a two-layer MLP fast model in the reported implementation.
The complete temporal computation is therefore:
flowchart LR
A["Current image + proprioception + instruction"] --> B["VLM backbone"]
B --> C["Per-timestep tokens"]
C --> D["DiT attention within timestep"]
D --> E["TTT fast MLP updates across timesteps"]
E --> F["Gated action-head output"]
F --> G["Action chunk"]
G --> H["Rollout updates fast weights"]
H --> E
The fast weights reset to $W_0$ at the beginning of a rollout and then propagate forward. Inference cost per timestep stays fixed with respect to the amount of history already absorbed; the model does not re-attend to every previous frame.
3. Training long sequences without storing every activation
RoboTTT combines two training choices.
Sequence action forcing applies flow matching independently to each action chunk. The noised target is
\[\tilde A_t=\tau_t A_t+(1-\tau_t)\epsilon_t, \qquad \epsilon_t\sim\mathcal N(0,I),\]and the sequence loss is
\[\mathcal L_{\mathrm{fm}}(\xi;W_0) =\frac{1}{T}\sum_{t=1}^{T} \mathbb E_{\tau_t,\epsilon_t} \left[\left\|v_\theta(\Phi_t,\tilde A_t,q_t;W_{t-1}) -(A_t-\epsilon_t)\right\|^2\right].\]Every timestep samples its own noise level. Sharing one noise level across a complete sequence can make all chunks uniformly easy or uniformly difficult, which the authors find destabilizes training.
Truncated backpropagation through time (TBPTT) divides a long sequence into short segments. Gradients stop at a segment boundary, while the fast weights themselves carry over to the next segment. GPU memory is determined by segment length instead of the full context length. The initial state $W_0$ still receives gradients through the first segment, so both the initialization and the update dynamics are learned.
The reported pretraining gradually increases the context length to the target, such as 8K timesteps, for 30K steps on 16 NVIDIA GB200 GPUs. Each downstream task is then post-trained at 1K context for 20K steps. This makes the headline context length a training resource as well as a model feature.
4. Context can be used as supervision without an action target
RoboTTT masks the flow-matching loss on selected parts of a sequence. Those tokens still update the fast weights, but the model is not asked to imitate their actions. This separates context from action targets and enables two experiments.
One-shot imitation from a human video
For Circuit, the same language instruction, “assemble circuit,” is used across configurations. A human video shows the target configuration while the robot remains idle; the following robot trajectory contains the action target. During training, the human video updates fast weights, the video loss is masked, and the robot actions are predicted conditional on the updated state.
At test time, one human video of an unseen configuration gives the policy the missing task information. RoboTTT completes 6 of 10 such trials with a 65% task-completion score. GDN, a recurrent baseline without test-time gradient updates, completes 0 of 10 and scores 33%.
DAgger Distillation for on-the-fly recovery
A DAgger rollout interleaves robot actions with human corrections. Standard training treats the human corrections as targets and often discards the preceding robot mistakes. RoboTTT assigns the two parts different roles:
- executed robot actions and human corrections both update the fast weights as context;
- the imitation loss is applied only to human corrections.
The model can therefore learn a failure-to-correction mapping. At deployment, its own wrong actions become context for the next fast-weight update, and it can attempt a recovery without a human takeover.
On a pool of 100 DAgger trajectories, standard DAgger improves the relevant policies by 9% on average, while DAgger Distillation improves the sequence models by 33% on average: 36% for RoboTTT and 29% for GDN. The robot’s suboptimal actions are valuable as context even though they are not imitation targets.
5. Experiments and what the numbers measure
The evaluation uses a YAM bimanual setup with four RGB cameras: top, bottom, left wrist, and right wrist. The three assembly tasks are:
- Pup Go Car: toy vehicle assembly, about two minutes per episode;
- Circuit: one-minute circuit assembly with 80 possible configurations;
- Gear Bot: ten-stage assembly, about five minutes per episode.
The Circuit training set uses 20 configurations and tests on the remaining 60. Policies are evaluated for 20 trials per task, except Gear Bot with 10 trials. The score is a rubric-based completion percentage, while a full success requires finishing the complete task.
| Method | Pup Go Car | Circuit | Gear Bot |
|---|---|---|---|
| RoboTTT | 9/20 | 13/20 | 2/10 |
| GR00T N1.7 | 3/20 | 3/20 | 0/10 |
| GR00T N1.7 Hist. | 0/20 | 8/20 | 0/10 |
| GDN | 3/20 | 8/20 | 0/10 |
Across task-completion scores, RoboTTT averages 79%, compared with 42% for single-step GR00T N1.7 and 56% for GDN. The paper reports an 87% relative improvement over the single-step baseline and a 41% improvement over GDN. Gear Bot is the sharpest stress test: only RoboTTT fully completes any five-minute runs, and it succeeds in 2 of 10.
Scaling the pretraining context
RoboTTT and GDN are pretrained at 128, 256, 512, 1K, 2K, 4K, and 8K timesteps, then evaluated with the same downstream protocol. RoboTTT’s closed-loop score rises steadily to 71.5% at 8K, versus 43.9% when the same model is pretrained at 1K and 45.6% for the best short-context baseline. The 8K model is 63% higher than its 1K counterpart under the paper’s comparison. GDN shows no comparable scaling trend.
The authors attribute this difference to meta-learning in the TTT update: longer training sequences shape both $W_0$ and the update dynamics over more steps. This interpretation is plausible within the experiment, though it does not prove that every long-context robotics architecture will scale in the same way.
Perturbation robustness
A human removes a component after the robot installs it. RoboTTT recovers the roof in 15/20 trials and a tire in 18/20. The best short-context baselines recover the roof in at most 10/20, while GDN also reaches 18/20 on the tire condition. The result supports within-episode conditioning; it does not show that TTT is always superior to recurrent state updates.
Ablations
Removing sequence action forcing substantially hurts closed-loop progress. Replacing the nonlinear MLP fast model with a linear layer still beats GR00T N1.7, but is 27% worse than the MLP fast model. Adding action tokens to a state-token-only version gives a 23% relative improvement, and learned register tokens give a further 18% relative improvement. These comparisons are useful because the extra tokens alone do not help the GR00T baseline; their value appears when paired with TTT temporal modeling.
6. Strengths, limits, and a practical reading
The paper’s strongest result is a clean link between a mechanism and a capability: fast weights provide a bounded recurrent state, while gradient updates make that state task-adaptive. The context can contain a human demonstration, a failure history, or observations from an occluded assembly, and the same policy interface consumes it.
There are three material limits. First, longer context raises training cost, even though inference cost per timestep remains constant after history compression. Second, the TTT loss is a generic prediction objective; a robotics-specific inner objective may improve adaptation. Third, the reported policy still misses deployment failures, so reinforcement learning aimed directly at task success remains an open direction.
I would read RoboTTT as evidence for context scaling as a robotics design axis, not as a claim that memory automatically solves long-horizon manipulation. The useful recipe is specific: train the update rule on full histories, let failures act as context, mask losses when a sequence should teach adaptation, and evaluate with enough horizon that short-context shortcuts break. For a new robot, the first reproduction question is whether the same fast-weight update can learn a stable failure-to-recovery mapping under that robot’s contact dynamics.