[Paper Notes] Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects via 2D Point Trackers
Published:
This post supports English / 中文 switching via the site language toggle in the top navigation.
TL;DR
A hand covers a bottle, moves it, and reveals it in a different orientation. Recovering its pose requires finding correspondences to observations made before the occlusion. Point2Pose uses a long-range 2D point tracker for that association, lifts the tracked points into 3D with depth, and registers them against an object map built during the video. It tracks multiple unknown rigid objects and reconstructs their surfaces without a supplied CAD model or a preliminary scan.
Compared with FoundationPose, the main gain is reduced object preparation and recovery during long tracking sequences. FoundationPose remains more accurate on the paper’s continuously visible benchmarks when given CAD models. Point2Pose’s v2 abstract explicitly describes a tradeoff between single-object accuracy and broader tracking capabilities.
Paper info
Tzu-Yuan Lin, Ho Jae Lee, Kevin Doherty, Yonghyeon Lee, and Sangbae Kim authored Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects via 2D Point Trackers. The listed affiliations are MIT and Boston Dynamics; Doherty’s contribution was made in personal time, independently of his affiliated organization.
These notes cover the 20-page arXiv:2604.10415v2, revised August 25, 2026, following the April 12 first submission. The official repository lists ECCV 2026. The PDF contains the method and evaluation discussed below; the project page hosts the project materials.
1. What changes from FoundationPose
Both methods consume RGB-D observations. Point2Pose’s phrase monocular RGB-D means one RGB-D camera stream; measured depth remains an input.
| Question | Point2Pose, as evaluated in the paper | FoundationPose |
|---|---|---|
| What object information is needed? | A few image points to identify each target; geometry is accumulated online | A CAD model, or reference images used to build a neural object representation |
| How is pose obtained? | Long-range point correspondence, 3D registration, and TSDF refinement | Render candidate poses, refine them with a learned network, and score them |
| How are multiple objects handled? | Aggregate their point queries in one tracker pass; maintain separate object maps | Apply the single-object pipeline to each target, with additional association and scheduling |
| What happens after tracking loss? | Recover correspondences to the existing map when points become visible again | Global pose estimation is available, but detecting loss and invoking it are system-level decisions |
| When is geometry built? | During tracking, using object-centric TSDF fusion | Supplied beforehand, or reconstructed from reference views before downstream estimation |
FoundationPose also has a model-free mode. Its reference views supply information about the target, and its published evaluation uses reference images with pose annotations. The pose networks generalize to new objects without fine-tuning, while the neural representation is fitted to the target’s reference observations. Point2Pose starts from the current sequence and builds its own geometric reference. The distinction concerns available object information and when it is acquired. FoundationPose, Sections 3.2 and 4.1
Point2Pose likewise uses pretrained components: SAM2 for segmentation, SuperPoint for selecting points, and a causal BootsTAP point tracker in the paper’s implementation. It needs no new pose-network training for each object. “No training” in a demo description should be understood in that deployment sense.
2. From a clicked point to a 6D pose
User clicks prompt SAM2 to obtain an object mask. Within that mask, the system samples points that balance detector confidence and spatial coverage. A cluster of easy points on one small patch gives poor geometric constraints, even if every point is tracked accurately.
For a tracked pixel $u_n=(u_n^x,u_n^y)$ with valid depth, back-projection gives a current 3D observation:
\[\tilde p_n=D_t(u_n)K^{-1} \begin{bmatrix}u_n^x\\u_n^y\\1\end{bmatrix},\]where $K$ is the camera intrinsic matrix. This is the pinhole back-projection underlying the paper’s lifting step. Each query retains an identity across time, so the current observation corresponds to a stored object-map point $p_n$. With $T$ mapping the object frame into the camera frame, the registration problem is
\[T^*=\underset{T\in\mathrm{SE}(3)}{\arg\min} \sum_n\left\|\tilde p_n-Tp_n\right\|^2.\]Equation (2) has an SVD solution when the correspondences are reliable. In practice, the hard part is deciding which correspondences to trust.
As the object rotates, new surfaces need new points. Appendix A samples additional points when the object rotates more than $10^\circ$ relative to previously sampled frames, or fewer than 25 tracked points remain visible. New points first enter a pending state. Three consecutive checks of pose stability, track quality, and mask consistency, followed by a 3D consistency check, protect the map from noisy additions. An erroneous point admitted to the map can corrupt many later frames.
3. Why one RANSAC solution can be wrong
The paper gives a useful mug example. Points on a largely symmetric body may appear stationary as the mug rotates, while a few points on the handle reveal the true rotation. The incorrect motion can have more supporting tracks than the correct one. A single consensus estimate can therefore select the wrong rotation.
Point2Pose runs sequential RANSAC with SVD: estimate a candidate, remove its consensus set, then seek another candidate among the remaining correspondences. This preserves alternative motions, including a smaller group of informative tracks.
The online TSDF then checks the candidates against dense depth. For each pose, observed surface points are transformed into the object frame; a small absolute TSDF value indicates agreement with the reconstructed surface. After selection, the method refines the pose with a Huber loss on TSDF residuals. A compact expression of this refinement objective is
\[\min_{T\in\mathrm{SE}(3)} \sum_{p\in\mathcal P_{\mathrm{cur}}} \rho_H\!\left(\Phi(T^{-1}p)^2\right).\]Here $\Phi$ is the TSDF and $\mathcal P_{\mathrm{cur}}$ is the masked depth point cloud. The paper’s Eq. (3) implements this with pose increments and Levenberg-Marquardt optimization. Sparse tracks propose plausible motions; dense geometry decides which motion fits the observed surface. Point2Pose, Section 3.3
At new keyframes, a factor graph jointly adjusts keyframe poses and map points. Its objective combines a first-frame prior, relative-pose constraints, and bearing/range observation residuals. TSDF fusion uses the estimated trajectory to accumulate object surfaces, and the resulting geometry supports later pose estimates. This coupling also means that sustained pose errors can damage reconstruction.
4. Recovery depends on remembering point identity
All objects’ query points are processed together, while maps and pose estimates remain object-specific. When an object reappears, recovered tracks can reconnect the current frame to its existing map. The estimate does not have to rely solely on the last visible frame’s pose.
The strongest direct evidence is a small ablation. P2P-SH(10) discards points after ten consecutive invisible frames. Across two real sequences containing five occlusion events, the full method recovers 5/5 within 30 frames after reappearance; the shortened-history version recovers 0/5. This supports retaining long-range identity, although five events do not establish a general recovery rate. The quantified recovery window also gives a more useful engineering interpretation than assuming that every disappearance is resolved in the first returning frame. Section 4.5
There is no visual measurement of a fully hidden object’s changing pose. The demonstrated capability is recovery after reappearance, conditional on usable point tracks, depth, and geometry.
5. Accuracy changes with the tracking scenario
The following values come from Point2Pose v2, Tables 2 and 3. They are ADD-S / ADD AUC (%) over thresholds from 0 to 0.1 m, with higher values better. FoundationPose receives CAD meshes; Point2Pose receives no object CAD model. These are the authors’ evaluations, not measurements reproduced for this post.
| Benchmark | Point2Pose ADD-S / ADD | FoundationPose ADD-S / ADD |
|---|---|---|
| YCBInEOAT: single-object manipulation | 92.67 / 85.11 | 96.00 / 92.48 |
| YCBMultiTrack-Synthetic: largely continuous visibility | 88.67 / 77.94 | 98.39 / 97.58 |
| YCBMultiTrack-Real: full occlusion and re-entry | 89.43 / 74.17 | 42.49 / 35.23 |
YCBInEOAT contains nine evaluated sequences with five objects. The new real-world benchmark contains eleven sequences with one to three objects, recorded with a RealSense D435i and OptiTrack ground truth. For multi-object evaluation, FoundationPose and BundleSDF are run separately for each object; Point2Pose tracks the objects together. Evaluation setup and results
The reversal between synthetic and real sequences matters. With a CAD mesh and sustained visibility, FoundationPose is more accurate. Frequent complete occlusion exposes the tested tracker’s difficulty recovering, and Point2Pose gains from its persistent correspondence map. This comparison does not establish that a FoundationPose system with explicit loss detection, re-detection, and global re-initialization would have the same failures.
ADD-S uses closest-point matching and tolerates geometric symmetries; ADD is more sensitive to orientation errors. On the weakly textured HO3D AP12 sequence, Point2Pose reports 84.33 ADD-S AUC but 46.64 ADD AUC. Coarse geometric alignment can look acceptable while orientation remains unreliable. Table 1 also gives Point2Pose a reconstruction Chamfer distance of 1.02 cm, versus 0.58 cm for BundleSDF, where lower is better.
Table 4 supports the geometric checks: removing multiple hypotheses reduces ADD AUC from 82.76 to 65.52; removing SDF refinement gives 77.37, and removing graph optimization gives 78.80. These are comparisons within that ablation table. Its full-method score differs from Table 1’s aggregate, so the values should not be mixed as though they were one identical run.
6. What this means for manipulation
Coordinate conventions are easy to overlook. Point2Pose initializes each object frame to coincide with the camera frame at the start, $T^C_{O,0}=I$. That frame stays attached to the object. The estimated motion and reconstructed shape are consistent in this chosen frame, but a grasp point specified in a CAD coordinate system needs an additional fixed alignment. The paper also assumes a camera fixed to the world in its formulation. A moving-camera deployment needs explicit handling of camera motion when the controller requires world-frame trajectories.
Runtime is another practical constraint. Point2Pose reports 2–10 Hz, depending on tracker resolution and point count, with point tracking as the main bottleneck. FoundationPose reports about 32 Hz for tracking and 1.3 s for pose estimation on an RTX 3090 with an Intel i9-10980XE. These are separate implementations and workloads, so they do not give a controlled speed ratio. They do distinguish frequent local updates from more expensive global recovery. Point2Pose, Section 4.4; FoundationPose, Section 4.5
I would start with FoundationPose for a known part whose CAD frame defines the insertion axis or grasp contacts, provided the system includes a recovery policy. I would test Point2Pose first for unfamiliar rigid objects that must be tracked together through repeated occlusions. Weak texture, inaccurate masks, unreliable depth, and growing point counts remain reasons to test on the actual camera and objects before connecting either estimator to a controller.
The next comparison I would run is FoundationPose with explicit re-initialization against Point2Pose, using the same masks, depth, objects, and compute budget. Recovery delay, identity switches, orientation error after recovery, and downstream grasp success would show whether Point2Pose’s tracking gains solve the manipulation problem at hand.
As of September 11, 2026, the official repository also advertises model-based tracking and Gaussian Splatting reconstruction added in August. Those extensions broaden the software beyond the CAD-free TSDF method evaluated in this paper; the tables above should not be assigned to those newer branches.