[Paper Notes] Scaling Online Exploration for Robot Agents
Published:
TL;DR
RAX turns online robot experience into executable perception and control programs. A frozen coding model explores through robot APIs, records failures, revises its code, and saves skills that have passed unattended execution. Learning updates the workspace while model parameters stay fixed.
The important result is that additional exploration improves performance on new scenes. The headline 37.4-point average gain combines five benchmark comparisons with different scoring conventions and selected task subsets. Some evaluations still retain an agent in the loop, and some provide demonstration information. Those details matter when interpreting “autonomous” and “without demonstrations.”
Paper and reading version
Scaling Online Exploration for Robot Agents — Runhan Huang, Xiangcheng Zhang, Haoyu Chen, Yitong Li, Yen-Jen Wang, Bohan Lyu, Haotian Ye, Chuye Hong, Yixuan Wang, Chloe Su, Haoyu Xiong, Yangzhen Wu, Enze Xie, Travis Manderson, Stefano Ermon, Pieter Abbeel, Antonio Torralba, Dawn Song, Weirui Ye, Yilun Du. These notes follow arXiv:2610.22496v1, an arXiv preprint, checked on October 25, 2026. PDF · Project
The mechanism below follows Sections 3–4; evaluation details follow Section 5 and Appendices C, F, and H.
Experience becomes a program the robot can run
RAX starts with a task, a robot-specific tool interface, and a workspace containing general instructions. The agent observes RGB-D images, camera calibration, and available force or state signals. It uses control APIs, inverse kinematics, and robot geometry to attempt the task. A failure becomes evidence for a change to a perception routine, a motion plan, or an execution check.
For example, the air-fryer program fits a geometric model to a point cloud, estimates the drawer axis and handle, checks reachability, and pulls in short increments. The model contains a fixed body and a sliding drawer. Measurements from the first successful attempt become parameters; later layouts require fresh perception. This explains how the generated program can adapt without asking the language model to reason about every action.
A useful schematic of the learning process is
\[P_{k+1}=\operatorname{Revise}_{\mathrm{LLM}}(P_k,\mathcal E_k,\mathcal M_k), \qquad a_t=P_k(o_{\leq t}),\]where $P_k$ is the current program, $\mathcal E_k$ is execution evidence, and $\mathcal M_k$ is workspace memory. This is my notation for the workflow, not a training objective from the paper. The LLM’s weights stay fixed.
Programs contain preconditions, stage transitions, outcome checks, and exception handling. If an observation falls outside the program’s reliable operating conditions, it can stop and return control to the agent. A skill enters persistent memory after successful unattended execution, along with its applicability conditions and run evidence. Later tasks can compose those skills.
Where the learned code ends and the robot stack begins
The frontend contains the agent and generated programs. The backend owns hardware drivers, sensing, low-level controllers, and enforced limits. Both interactive tool calls and autonomous programs use this same boundary. An agent-generated collision check supplements the backend; it does not replace its joint limits or operator arming requirement.
The available control abstraction varies by platform. HumanoidArena uses the SONIC whole-body tracker; LIBERO uses its operational-space controller. Some settings offer planning and IK helpers. RAX discovers task-specific logic on top of these interfaces. Reproducing the system therefore requires documenting the robot backend as carefully as the prompt.
Compilation also changes inference cost. In the selected air-fryer session, reported inference time falls from 72 minutes for agent-mediated execution to 18 seconds for its program, a 241× reduction. This is a comparison of inference time for a successful execution, not a claim that the physical robot moves 241 times faster.
The benchmark averages need their protocols
| Setting | RAX | Reference | What the number measures |
|---|---|---|---|
| RoboDojo, seven tasks | 78.3% | VPP2 24.9% | Native score, including partial credit; largest budget per task |
| BEHAVIOR-1K, six tasks | 46.5% | Leading challenge entry 8.2% | Fraction of goal conditions satisfied on ten public test instances |
| HumanoidArena, seven tasks | 89.8% | WB-WAM 81.9% | Success on official unseen test episodes |
| DexJoCo, five bimanual tasks | 94.3% | ME-Dex 48.8% | Success with 32 exploration episodes |
| LIBERO-Pro, position perturbation | 84.4% | VPP2 42.3% | Comparison with the strongest learned model in the summary |
Appendix F averages the five margins to obtain 37.4 percentage points. Its DexJoCo row uses only the five bimanual tasks. Across all eleven DexJoCo tasks at 32 episodes, the reported score is 96.7%, against 65.3% for ME-Dex. On LIBERO-Pro, the coding-agent baseline ASPIRE reaches 63.2%; the 42.1-point headline margin uses VPP2 instead. These are different comparisons.
RoboDojo evaluates the resulting program without its agent on 30–50 unseen scenes per task. BEHAVIOR-1K mostly evaluates an agent-in-the-loop program, with a fresh agent and frozen workspace; exploration also receives detailed review-model feedback. Its 46.5% Q-score cannot be read as 46.5% complete-task success.
Demonstration access also varies. RoboDojo supplies a goal image; BEHAVIOR-1K supplies visual demonstration keyframes without actions or states. HumanoidArena gives the exploring agent three teleoperated demonstrations. The general framework avoids fitting a policy to a teleoperation corpus, but a blanket claim that every experiment has no demonstration input would conflict with the protocol.
What scales, and what still costs work
On RoboDojo, charger insertion increases from 26% with four exploration episodes to 94% with eight; the long-horizon Kong task reaches 96% at 32 episodes. Hang Mugs remains around 33–39% across its tested budgets. More exploration helps many tasks, while particular perception or control difficulties can persist.
Physical experiments cover hidden-object memory, articulated objects, wiping, precise insertion, and long-horizon mobile manipulation. They provide detailed sessions and case studies, not one uniform real-robot success-rate benchmark. Operators still reset scenes. Appendix H reports more than US$100,000 in API credits across all simulation and physical experiments; that is a study-wide total, not a cost per learned skill.
My takeaway is to evaluate the stored program as the main product of exploration. For a follow-up, I would measure unseen-scene success, agent intervention frequency, exploration cost, and reuse across tasks separately. A fast replay that needs frequent intervention and a reliable unattended skill have different practical value, even if both eventually finish the task.