[Paper Notes] GPT-6 Astra Is a Native Visuo-Tactile Policy for Contact-Rich Manipulation
Published:
TL;DR
This paper feeds tactile RGB images directly into GPT-6 Astra alongside scene images, robot state, and demonstrations. The model outputs end-effector and gripper actions, observes the result, and chooses the next action sequence. No tactile-specific policy training is added by the study.
Across nineteen simulated tasks, visual-plus-tactile input reaches 93.8% on UniVTAC and 81.8% on OpenViTac. Vision-only scores are already 76.3% and 80.6%. The main finding is that image-based touch can contribute to direct multimodal control, with strongly task-dependent benefits and substantial response latency.
Paper and reading version
GPT-6 Astra Is a Native Visuo-Tactile Policy for Contact-Rich Manipulation — Chen Yang, Linzhe Shi, Hang Zhang, Mu Xu, Jiansheng Fan, Chen Wang. These notes follow arXiv:2610.26846v1, an arXiv preprint, checked on October 25, 2026. PDF
The interface and protocol are described in Sections 3–4; Tables 1–4 and the trajectory analyses support the results discussed below. “Native” describes using the model’s existing visual interface; it does not mean that the experiment has no demonstrations or robot-control adapter.
Touch arrives as an image with an action history
At decision $t$, Astra receives a task instruction $g$, robot state $x_t$, scene views $I_t$, and, in the tactile condition, two fingertip contact images $T_t$. Its policy can be written as the paper’s Equation 1:
\[A_t\sim\pi(\cdot\mid g,\mathcal D,\mathcal M_t,x_t,I_t,T_t), \qquad a=(\Delta p,\Delta r,u).\]The output contains translation and rotation increments and a gripper target. The adapter executes the sequence and returns an action receipt plus a new observation. An executed command is therefore separate from evidence that the object actually moved as intended.
Demonstration blocks contain an initial observation and state, recorded actions, an execution receipt, and the resulting observation. They remain available throughout the episode. Interaction memory keeps earlier anchors and the most recent feedback, with explicit markers for omitted intervals. This supports comparing a new contact pattern with both a demonstrated transition and the robot’s own preceding attempt.
The model uses xhigh reasoning effort. Twelve tasks use one demonstration with twelve selected segments; six classification or selection tasks use two demonstrations with five segments each; Insert Block v2 uses three shape-specific demonstrations, each with five segments. UniVTAC uses released training recordings, while OpenViTac demonstrations come from its scripted experts.
The cleanest comparison is vision versus vision plus touch
Both conditions share demonstrations, segment starts, recorded actions, and sampled test scenes. The vision-only setting removes tactile images from demonstrations and online observations. Robot state and task text remain available in both.
The study uses a stratified 10% sample with seed 42: 80 UniVTAC scenes and 130 OpenViTac scenes per condition. Most tasks have ten episodes; Insert Block v2 has thirty across three shapes. Means give equal weight to tasks, not to individual episodes.
| Suite | Published strongest specialist mean | Astra, vision | Astra, vision + touch |
|---|---|---|---|
| UniVTAC, eight tasks | SimpleTouch 70.9% | 76.3% | 93.8% |
| OpenViTac, eleven tasks | OpenVTLA 68.7% | 80.6% | 81.8% |
The specialist results are taken from their benchmark reports. They were not freshly rerun on Astra’s sampled scenes. One task also changes the success criterion: Grasp Chip accepts intact transfer and stable supported placement without requiring the original release-and-withdrawal sequence. These qualifications limit a strict head-to-head interpretation.
The observed tactile benefit is 17.5 points on UniVTAC but only 1.2 points on OpenViTac. Across nineteen task scores, touch improves six, leaves nine unchanged, and reduces four. Pull-out Key increases from 30% to 90%, and Insert Hole from 60% to 100%. Weight Classify drops from 50% to 40%, and Empty-Can Selection from 70% to 50%.
Contact cues can guide a correction and still be ambiguous
In a successful hole-insertion trace, opposing slopes in fingertip contact bands and visible shaft deflection lead to a change in orientation while maintaining entrance contact. The resulting observation then guides another advance. This makes the proposed mechanism concrete: the contact image changes the next control decision.
Other traces expose the limits. Weak visible shear is interpreted as low weight even when the held object is heavy. Contact geometry and grip can change how much load appears in a tactile image. In a failed block insertion, the model recognizes misalignment but repeated correction commands do not produce the required motion. In Empty-Can Selection, prolonged probing leaves too little of the 400-action budget to finish transfer.
Selected trajectories demonstrate possible behaviors, not their frequency. Most task-condition scores use only ten episodes, so one outcome changes the score by ten percentage points.
Vision can also reveal texture or shape cues correlated with the physical property being classified. OpenViTac supplies fixed and wrist views for every task, while six UniVTAC tasks supply only the fixed camera. This may help explain the different marginal value of touch, but camera access was not independently randomized to test that explanation.
The next contact correction waits for another response
For the evaluation phase, the paper logs CNY 7,505.05 in provider-estimated costs across both conditions, including billed errors, retries, and reruns, while excluding demonstration preparation and supplementary experiments.
Median complete API response time is 15.4 seconds for vision and 14.0 seconds for vision plus touch; the respective 90th percentiles are 46.3 and 44.0 seconds. These are full response latencies, including provider and gateway waiting, with simulation execution excluded. A response can contain several control actions, but the next observation-dependent correction still waits for another response.
I read this as evidence for a useful sensory interface and a demanding benchmark baseline. Physical-robot validation remains open. A stronger follow-up would remove visual property shortcuts, retain matched evaluation scenes and criteria, and test whether a faster local controller can use the model’s contact interpretation without waiting seconds between corrections.