[Paper Notes] PointVLA: Injecting the 3D World into Vision-Language-Action Models
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
Current VLA models are pre-trained on massive 2D vision-language data, but their reliance on RGB images limits spatial reasoning for real-world manipulation. Retraining from scratch with 3D data is prohibitively expensive. PointVLA proposes a lightweight solution: freeze the pre-trained VLA, inject point cloud features into its action expert via a modular block, and use a skip-block analysis to find which blocks in the action expert are least critical — injecting 3D features only there to minimize disruption. The result is a model that gains 3D spatial understanding (height adaptability, real-vs-photo discrimination, few-shot multi-tasking) while preserving the full benefit of large-scale 2D pre-training, with only 5 additional lightweight injection blocks to train.
Paper Info
The paper is “PointVLA: Injecting the 3D World into Vision-Language-Action Models,” by Chengmeng Li, Yichen Zhu, Junjie Wen, Yan Peng, Yaxin Peng, and Feifei Feng from Midea Group, Shanghai University, and East China Normal University. Accepted at IEEE RA-L 2025. The project page is at pointvla.github.io and the paper at arXiv:2503.07511.
1. Problem and Motivation
VLA models like OpenVLA, \(\pi_0\), and DexVLA have shown impressive capabilities by leveraging pre-trained vision-language models as backbones, then training action experts to translate visual-linguistic understanding into robot actions. Their strength comes from billions of parameters pre-trained on internet-scale 2D data.
But they only see in 2D. This creates real failure modes: a VLA cannot distinguish a photograph of an object from the real thing (both look identical in RGB from the right angle), cannot adapt when an object is placed at a different height than in training, and generally lacks the depth perception needed for precise 3D manipulation.
The naive fix — retrain the whole foundation model with 3D data — is impractical. 3D robotic datasets are orders of magnitude smaller than 2D vision-language corpora. Retraining would also discard the valuable 2D representations. An alternative approach like 3DVLA processes 3D tokens through the LLM backbone, but current VLMs exhibit limited 3D comprehension when fine-tuned on small 3D datasets due to the domain gap between 2D pixels and 3D structures.
PointVLA takes a different path: treat 3D point clouds as a complementary conditioning signal rather than a primary input modality, injecting them into the action expert rather than the vision-language backbone.
2. Method
PointVLA builds on DexVLA (Qwen2-VL as the 2B-parameter VLM backbone + ScaleDP as the 1B-parameter diffusion policy action expert). The key insight is to keep the VLM entirely intact and inject 3D information only into the action expert, where spatial reasoning most directly affects motor behavior.
2.1 Point Cloud Encoder
Rather than using a pre-trained 3D visual encoder (which the authors found hinders generalization to new environments, consistent with findings in DP3 and iDP3), PointVLA adopts a simplified hierarchical convolutional architecture. Upper layers extract low-level features, lower layers learn high-level scene representations, with max pooling between layers to reduce point cloud density. Feature embeddings from each convolutional block are concatenated into a unified multi-level 3D representation.
2.2 Point Cloud Injector
The injector has three components:
- Channel alignment: Transform the 128-dimensional point cloud embedding to match the action expert’s 1280-dimensional channel size
- Action embedding bottleneck: Compress the potentially large action embedding (from chunk-based prediction) to align with the point cloud embedding
- Block-wise injection: For each selected block in the action expert, an MLP adapter processes the point cloud embedding, followed by a zero-initialized linear layer that adds the 3D features to the block’s output
The zero initialization is important — it means the injected features start as zero, so the model initially behaves identically to the vanilla VLA. The 3D signal is gradually learned during fine-tuning without disrupting existing representations.
2.3 Skip-Block Analysis: Where to Inject
Not all blocks in the action expert are equally important. Injecting 3D features everywhere would be both expensive and disruptive. The authors perform a systematic skip-block analysis on DexVLA’s 32-block action expert using a shirt-folding task:
Single-block skipping: The first 11 blocks are critical — skipping any one causes significant performance drops. From block 11 onward, skipping a single block is acceptable, indicating these later blocks contribute less after training.
Multi-block skipping: Starting from block 11, up to 5 consecutive blocks can be skipped before the model fails. Skipping 6 or more causes instant performance collapse.
Based on this analysis, PointVLA injects 3D features into 5 blocks (blocks 12, 13, 16, and two others in the less-critical zone). All modules in the vanilla action expert are frozen except the final layers that fit the embodiment’s output. Only the 5 injection blocks are trained — making the approach highly parameter-efficient.
3. Experiments
3.1 Setup
Two real-world bimanual platforms:
- Bimanual UR5e: Two UR5e arms with Robotiq grippers, three cameras (two wrist RealSense D435i + one top), 14-dim action space, 15Hz. RealSense L515 for point clouds.
- Bimanual AgileX: Two 6-DoF AgileX arms with wrist cameras and base camera, 14-dim action space, 30Hz. Same L515 for point clouds.
Baselines: OpenVLA, Diffusion Policy (DP), 3D Diffusion Policy (DP3), ScaleDP-1B, Octo, and DexVLA. Since PointVLA is built on DexVLA, DexVLA serves as the direct ablation — same model without point cloud injection.
3.2 Few-Shot Multi-Tasking (AgileX)
Four tasks with only 20 demonstrations each (80 total): ChargePhone, WipePlate, PlaceBread, TransportFruit. These test both independent and coordinated bimanual movements.
PointVLA outperforms all baselines. Notably, Diffusion Policy fails on most tasks — with only 20 demos per task, the action representation space becomes entangled. Even scaling up (ScaleDP-1B) doesn’t help much. DexVLA shows strong few-shot capability but PointVLA consistently improves on it, demonstrating that point cloud integration enables more sample-efficient learning.
3.3 Long-Horizon Packing (UR5e)
A challenging conveyor belt task: pick up two laundry detergent bottles from a moving belt and pack them into a box, then seal it (5 sequential subtasks). The assembly line is in motion, the embodiment differs from pre-training data, and the task is long-horizon.
Results (average completion length out of 5 subtasks):
- OpenVLA: 0.36, DP: 0.36, ScaleDP-1B: 0.72
- DexVLA: 1.72
- PointVLA: 2.36
PointVLA surpasses DexVLA by 0.64 in average completion length — a substantial margin on a task where all other baselines essentially fail after the first 1–2 steps.
3.4 Real-vs-Photo Discrimination
A striking experiment: replace a real laundry detergent bottle on the conveyor belt with its photograph displayed on a screen. From the egocentric top camera, the photo closely resembles the real object. All 2D-based models (OpenVLA, DP, ScaleDP, DexVLA) attempt to grasp the non-existent object, with DexVLA entering a repetitive grasping loop. PointVLA is the only model that correctly recognizes no real object exists on the belt, achieving 3/3 success while all baselines score 0/3.
This is perhaps the most compelling demonstration of why 3D understanding matters for safety — a purely 2D model can be trivially “deceived” by a printed image.
3.5 Height Adaptability
Training data uses a 3mm foam layer under the bread; at test time, this is replaced with a 52mm layer. All 2D baselines (OpenVLA, DP, ScaleDP, DexVLA) fail — they push down to the trained height and miss the object. PointVLA succeeds 5/5 by perceiving the actual 3D height and adjusting accordingly.
3.6 Simulation (RoboTwin)
On the RoboTwin benchmark (14-DoF mobile bimanual platform, 16 diverse tasks), PointVLA achieves the highest average success rate across all tasks with both 20 and 50 demonstrations. Interestingly, for pure 3D methods like DP3, adding RGB input can actually hurt performance, while PointVLA’s approach of conditionally integrating 3D as a complement to 2D avoids this problem.
4. Strengths and Limitations
Strengths. The paper’s core contribution is conceptually clean: rather than choosing between 2D pre-training and 3D understanding, PointVLA gets both by treating point clouds as a complementary signal injected into carefully selected locations. The skip-block analysis is a principled and reusable technique — it provides a general recipe for identifying where to inject new modalities into pre-trained models. The real-vs-photo and height adaptability experiments are not just ablations; they expose genuine safety-critical failure modes of 2D-only VLAs.
Limitations. The paper is built entirely on top of DexVLA, so the generality of the approach to other VLA architectures (e.g., \(\pi_0\), OpenVLA’s architecture) remains untested. The point cloud encoder is deliberately simple (hierarchical convolution), which the authors acknowledge could be improved. The skip-block analysis is conducted on a single task (shirt folding) — whether the same blocks are “less critical” across different tasks and embodiments is an open question. Finally, the real-world experiments, while compelling, use relatively small evaluation sets (3–5 rollouts for some tasks).
5. Takeaways
PointVLA solves a practical problem cleanly. The insight that 3D features should be injected into the action expert (not the VLM backbone) is well-motivated: the VLM’s job is semantic understanding, which 2D pre-training already handles well; it’s the action expert that needs spatial precision. The skip-block analysis provides a principled way to do this injection without disrupting pre-trained representations.
The real-vs-photo experiment is the kind of result that sticks with you. It’s a simple setup, but it exposes a fundamental limitation of 2D-only models that no amount of scale will fix — you cannot distinguish real from fake without depth. As robots move into less controlled environments, this kind of 3D grounding will shift from “nice to have” to essential.
For practitioners, the takeaway is that you don’t need to retrain your VLA from scratch to add 3D. A lightweight injection module with careful placement can get you most of the benefit at a fraction of the cost.